[syslog-ng] [PATCH]: Experimental SMTP destination

[Alexander Clouter]

Gergely Nagy <algernon at balabit.hu> wrote:
>>
>> So isn't the built-in MTA a solution in search of a problem? 
> 
> Nope, it isn't. I have routers that do not have a reliable MTA, and I
> don't intend to install one on them for the sole purpose of sending
> syslog alerts.
> 
Then make them reliable (for me it was 'aptitude install nullmailer', 
not complicated).  How do you plan to send cron job alerts to the 
sysadmin if there is no local MTA?  All UNIX hosts really should have a 
way to send email to their sysadmin, especially if they are a crucial 
part of the organisation's infrastructure.

> There might be N other ways to accomplish the same thing, a built-in
> smtp destination was the easiest for me, so I coded it up. All the other
> soultions I could think of would have taken me more time and effort.
> 
N ways that already exist, and have worked for ~30 years.

> I had an itch, I scratched it, I'm happy.
> 
Well, that's a different and perfectly okay objective.  The issue is 
that sending mail from a UNIX host is an already solved problem.

>> Also, I see some problems with the built-in SMTP destination. 
>> Misconfiguration now is a problem for syslog-ng itself - instead of 
>> the external MTA.
> 
> Yes, and that is good so. I will notice such misconfigurations far
> sooner.
>
A non-zero mail queue or rejections lurking in /var/log/mail are trivial 
to notice.
 
>> I'm pretty sure it won't be too hard to configure syslog-ng so, that
>> it's pretty easy to troubleshoot the MTA that it sends it's messages to.
> 
> The thing is, I don't want a full-blown MTA on the same host that
> syslog-ng runs on. So at best, I can have a program that forwards alerts
> to an external host.
>
Nullmailer is *not* a full-blown MTA but it does reliably deliver mail 
to a smarthost.  It's a 500kB (100kB binary) install with *all* the man 
pages and Debian documentation.

Also, the flexibility you could call upon with 'mailx' (for example, grep 
a log file and append the +-20 lines to the email) is far more than what 
could be put into syslog-ng.

> - While it is easy to troubleshoot the forwarder, the MTA is running on
> another host. That alone makes it slightly more complicated to
> troubleshoot it, as there's one more variable in the picture.
>
I think you are seeing a problem where one does not exist.
 
> - I will either need a reasonably feature-full forwarder that can do TLS
> and authentication, so I can use the SMTP server of my choice, or I need
> an intermediate server that can accept mails from the syslog host, and
> forward it (using TLS & auth) to the real smtp server.
> 
Nullmailer does auth whilst nbsmtp can do tls/auth/queue.

> Having an built-in smtp destination, that will have tls & auth real
> soon, does not have these issues.
> 
These are problems that do not exist.  Can you tell me what problems you 
have seen (not theorised) there are by just forking sendmail 
(wrapper)/nbsmtp/whatever?

>> But the main point is, that I don't think that an internal SMTP
>> destination will do any good for misconfiguration. I.e. misconfiguration
>> is a problem in itself, and no matter how many destinations you add to
>> syslog-ng, you won't be able to prevent misconfiguration.
> 
> Correct. But as far as I'm concerned, I find it easier to debug one
> program, on one host, than go through N hops to find which one is
> misbehaving.
> 
It's far more difficult to ram the SMTP RFC's into a syslog daemon.

> With the program() destination, in order for my alerts to leave my
> system, it would go through several hops.
> 
> Assuming a sendmail that can do TLS and auth (best case):
> 
>        syslog-ng -> sendmail -> external SMTP host
>
'best' = 'typical'

> With a bit of luck, the sendmail program will do proper logging, so I
> can troubleshoot it easily. Hopefully, it will also do queuing, either
> in memory, or on disk, so that when the SMTP host is unreachable, my
> alerts will not get lost.
>
'luck' = 'typical'

'queue' is down to personal preference, but as I keep hammering, 
nullmailer and nbsmtp both do queuing and have done so for *years*.  I 
am guessing syslog-ng will not queue a 10kB email for me (as it would be 
nice to call on mailx's '~!').

> There's too many ifs in there, and I'm really not interested in finding
> a small enough forwarder that can do all that. I know busybox can't, nor
> can nullmailer. Dragonfly Mail Agent (dma) might be a good candidate,
> but I'd need to review it. Esmtp is relay-only, and does not queue.
> 
nullmailer can and does queue, I use it on my laptop to get mail off to 
a smarthost.  It also logs.

> With a built-in smtp destination, the setup would look like this:
> 
>        syslog-ng -> external SMTP host
> 
> Boom. The smtp driver will detect if there's a network outage, and queue
> the mail, and retry later. It does this using syslog-ng's built-in
> queuing mechanism. I don't need to go an extra mile to make sure that
> any message sent from syslog-ng will be retried at least until the queue
> fills up.
>
nullmailer....nullmailer....nullmailer. :-/

If we say, "yes there is a void in lightweight MTA's that can do 
queuing, auth and TLS" the fix is to dedicate your efforts to produce 
one.  Then *everything* can use it...not just syslog-ng.  Putting a SMTP 
transport in syslog-ng helps no one, but the syslog-ng users.

>> A problem that was not yet mentioned is, that having an SMTP destination
>> now makes syslog-ng dependent on yet another library, libesmtp.
> 
> It's optional, though. By that logic, we shouldn't have sql()
> destinations, either, since one can just use program() with a perl
> script. We wouldn't need tls, either, since one can set up a tunnel. We
> wouldn't need tcp() and the rest, either, since we can use netcat...
>
I am forced to use program to open 'psql' as syslog-ng cannot do simple 
"SELECT function('foobar', 'cheese', 'toys');" and also a Perl script to 
process squid log files too[1] as syslog-ng cannot do xyz; tasks which 
are syslog-ng specific...  These work *extremely* well.

Comparing smtp() to sql() is not quite like for like.  You get 
additional functionality when accessing it natively such as prepared 
statements and such.  Natively speaking SMTP does not buy you anything 
that I am aware of?

Of course I guess it could be argued no one is forcing me to use 
smtp(), which would be a fair point. :)

>> All in all, it sounds like the perfect solution for a certain
>> widely-used proprietary operating system, that has no proper built-in
>> SMTP possibilities. And because the SMTP destination is configurable
>> with ENABLE_SMTP, Linux builds can easily avoid the new dependency ;)
> 
> If you're talking about the OS I think you're talking about, let me
> assure you, I wouldn't touch it with a ten feet pole if I can avoid it
> (not to mention, that libestmp is most probably not available for said
> platform).
> 
Ironically libesmtp is larger than nullmailer on my amd64 Debian laptop.

I think the reference you are missing here is that with Windows it is 
*very* difficult (unsurprisingly) to apply the original UNIX philosophy 
of stringing together many small simple tools that do their singular 
task very well to perform the job you need.  In Windows, if the 
application you are using does not support what you need it to do...you 
are generally out of luck.

What do you do when given the task of parsing a CSV file (overlook 
escaping) to summing up the numbers in the fourth column and producing 
the average?  Under UNIX would you reach for OpenOffice or awk?  Under 
Windows you typically have no choice but to do the task in Excel.

This is what I'm trying to communicate over, possibly poorly.  I really 
am just very curious what the actual *real* problem is, and where all 
this supposed unreliability in calling sendmail comes from?

On a passing note, what would be handy is some v6 netmask() 
action...much obliged if you find and scratch that itch :)

Cheers

[1] http://www.digriz.org.uk/syslog-ng-integration

-- 
Alexander Clouter
.sigmonster says: In the stairway of life, you'd best take the elevator.