On Mon, 2011-02-14 at 14:37 +0000, Alexander Clouter wrote:
Gergely Nagy <algernon@balabit.hu> wrote:
So isn't the built-in MTA a solution in search of a problem?
Nope, it isn't. I have routers that do not have a reliable MTA, and I don't intend to install one on them for the sole purpose of sending syslog alerts.
Then make them reliable (for me it was 'aptitude install nullmailer', not complicated).
nullmailer fails at supporting tls, so even if I install it, I still need an intermediate relay, or another tool to provide TLS for me. At which point, I can just add smtp() to syslog-ng, teach it TLS & auth (both of which is supported by libesmtp, I only need a few lines of glue code in the destination driver) and I'm better off that way. Less extra space, less stuff to configure, less points of potential failure.
How do you plan to send cron job alerts to the sysadmin if there is no local MTA? All UNIX hosts really should have a way to send email to their sysadmin, especially if they are a crucial part of the organisation's infrastructure.
Those alerts are logged to syslog, and my smtp destination can send the emails. I don't need a local MTA, nor do I desire one.
There might be N other ways to accomplish the same thing, a built-in smtp destination was the easiest for me, so I coded it up. All the other soultions I could think of would have taken me more time and effort.
N ways that already exist, and have worked for ~30 years.
N ways already exist, neither of which is suitable for my use-case. I might be the first guy in 30 years, in which case I feel honored.
I had an itch, I scratched it, I'm happy.
Well, that's a different and perfectly okay objective. The issue is that sending mail from a UNIX host is an already solved problem.
Provided that said host has the tools for it, yes. None of the tools available in debian were found to fit my needs. Unless combined, which IS the UNIX way, but is something I want to avoid in this case.
I'm pretty sure it won't be too hard to configure syslog-ng so, that it's pretty easy to troubleshoot the MTA that it sends it's messages to.
The thing is, I don't want a full-blown MTA on the same host that syslog-ng runs on. So at best, I can have a program that forwards alerts to an external host.
Nullmailer is *not* a full-blown MTA but it does reliably deliver mail to a smarthost. It's a 500kB (100kB binary) install with *all* the man pages and Debian documentation.
It does not support TLS, though, as far as I saw. Thus, I'd still need an intermediate relay, or another program to provide TLS for me.
Having an built-in smtp destination, that will have tls & auth real soon, does not have these issues.
These are problems that do not exist. Can you tell me what problems you have seen (not theorised) there are by just forking sendmail (wrapper)/nbsmtp/whatever?
Problem #1: I need a suitable sendmail on the host. Problem #2: Depending on the MTA, I will likely need something else to provide TLS support for it. Problem #3: External tools do not provide anything over my smtp driver, at least not in my use-case, while my smtp driver solves my immediate issue of not having to install external tools to get the job done. Yes, the problem can be solved with program() in most cases. My case is not the common case, though, and I do not see harm in catering to a few corner-cases when it does not disrupt anything else.
But the main point is, that I don't think that an internal SMTP destination will do any good for misconfiguration. I.e. misconfiguration is a problem in itself, and no matter how many destinations you add to syslog-ng, you won't be able to prevent misconfiguration.
Correct. But as far as I'm concerned, I find it easier to debug one program, on one host, than go through N hops to find which one is misbehaving.
It's far more difficult to ram the SMTP RFC's into a syslog daemon.
Which is why I didn't do that, but rely on libesmtp, which does the bulk of the job for me. I don't need all of the SMTP RFC, only a subset, which libesmtp implements. If I ever need more than that, I will either extend libesmtp, or find a way to use program(). Right now, program() would require more work from me, than the smtp destination did.
With the program() destination, in order for my alerts to leave my system, it would go through several hops.
Assuming a sendmail that can do TLS and auth (best case):
syslog-ng -> sendmail -> external SMTP host
'best' = 'typical'
Yes. But I see no harm in making corner-cases easy too. Typical installs can still use program() if so they wish, and they need the extra features (if any).
With a bit of luck, the sendmail program will do proper logging, so I can troubleshoot it easily. Hopefully, it will also do queuing, either in memory, or on disk, so that when the SMTP host is unreachable, my alerts will not get lost.
'luck' = 'typical'
'queue' is down to personal preference, but as I keep hammering, nullmailer and nbsmtp both do queuing and have done so for *years*. I am guessing syslog-ng will not queue a 10kB email for me (as it would be nice to call on mailx's '~!').
nullmailer doesn't have TLS, so I'd have to install yet another tool to fix that, or use an intermediate relay. I'd rather not do either. I tried to check what nbsmtp can do, but it's not available on neither Lucid, nor Squeeze, and the homepage returns a default Debian Apache page. As for queuing: syslog-ng will queue any log messages that it couldn't send via the smtp driver (the message, not the email), and will reconstruct the mail at a later point.
There's too many ifs in there, and I'm really not interested in finding a small enough forwarder that can do all that. I know busybox can't, nor can nullmailer. Dragonfly Mail Agent (dma) might be a good candidate, but I'd need to review it. Esmtp is relay-only, and does not queue.
nullmailer can and does queue, I use it on my laptop to get mail off to a smarthost. It also logs.
But, as I said, it doesn't do TLS. All of the listed solutions do part of what I want, but neither does it all. Thus, if I go with either, I will have to install something else aswell, which just complicates things. A built-in smtp() driver is far easier for me. It can do everything I want, in one place.
With a built-in smtp destination, the setup would look like this:
syslog-ng -> external SMTP host
Boom. The smtp driver will detect if there's a network outage, and queue the mail, and retry later. It does this using syslog-ng's built-in queuing mechanism. I don't need to go an extra mile to make sure that any message sent from syslog-ng will be retried at least until the queue fills up.
nullmailer....nullmailer....nullmailer. :-/
TLS... TLS... TLS.
If we say, "yes there is a void in lightweight MTA's that can do queuing, auth and TLS" the fix is to dedicate your efforts to produce one. Then *everything* can use it...not just syslog-ng. Putting a SMTP transport in syslog-ng helps no one, but the syslog-ng users.
I'm fine with helping syslog-ng users only. Producing a suitable MTA that can do everything I want would require far more effort than producing the smtp destination took.
I am forced to use program to open 'psql' as syslog-ng cannot do simple "SELECT function('foobar', 'cheese', 'toys');" and also a Perl script to process squid log files too[1] as syslog-ng cannot do xyz; tasks which are syslog-ng specific... These work *extremely* well.
So sql() has next to no use, let's remove it, it's useless. (I'd actually agree with that - sql-specific drivers, which exploit the features specific to said sql server would be of much more value, in my opinion)
Comparing smtp() to sql() is not quite like for like. You get additional functionality when accessing it natively such as prepared statements and such. Natively speaking SMTP does not buy you anything that I am aware of?
I already listed why I prefer a native driver over using program(), but let me reiterate: * I want TLS, auth and queuing, preferably in one place. None of the existing lightweight solutions I know of provide that in themselves. Yes, nullmailer comes close, but it does lack TLS, which forces me to setup or install yet another tool. * program() was designed with a long-running process in mind, and even though syslog-ng will restart a program that exists, that's not the intended usage. Yes, one can write a wrapper that doesn't exist, but then I could just use the smtp destination. It also restarts the program, if it exits, but personally, I find that an abuse, and if I'd use program(), I'd do that with a long-running wrapper instead. And then it's easier to use smtp()... * A built-in driver suited my immediate needs best, it's optional, noone's forced to use it if they're happy with something else.
All in all, it sounds like the perfect solution for a certain widely-used proprietary operating system, that has no proper built-in SMTP possibilities. And because the SMTP destination is configurable with ENABLE_SMTP, Linux builds can easily avoid the new dependency ;)
If you're talking about the OS I think you're talking about, let me assure you, I wouldn't touch it with a ten feet pole if I can avoid it (not to mention, that libestmp is most probably not available for said platform).
Ironically libesmtp is larger than nullmailer on my amd64 Debian laptop.
That's interesting. libesmtp.so is ~72k on my amd64 lucid, while nullmailer (/usr/lib/nullmailer/* + /usr/bin/nullmailer-inject + /usr/sbin/nullmailer*) is ~172k. I'm not sure how much of that I'd need for simple forwarding, but I assume I need at least nullmailer-queue and nullmailer-send, which are 60k. And indeed, 60k is smaller than 72k, but libesmtp also does TLS, which nullmailer does not.
I think the reference you are missing here is that with Windows it is *very* difficult (unsurprisingly) to apply the original UNIX philosophy of stringing together many small simple tools that do their singular task very well to perform the job you need. In Windows, if the application you are using does not support what you need it to do...you are generally out of luck.
I suppose you're right. I do not know Windows well enough to make an informed opinion (and I'd prefer to keep it that way).
What do you do when given the task of parsing a CSV file (overlook escaping) to summing up the numbers in the fourth column and producing the average? Under UNIX would you reach for OpenOffice or awk? Under Windows you typically have no choice but to do the task in Excel.
I'd go with awk, yes. But if I had an application that would need this data, and I had a library to parse CSV files, I probably wouldn't call out to awk, but use the parser library instead. It does depend on the use case, though. If I'm writing a shell script, then awk it is. If I'm writing a GUI application, I'd rather use a library instead of calling out to awk.
This is what I'm trying to communicate over, possibly poorly. I really am just very curious what the actual *real* problem is, and where all this supposed unreliability in calling sendmail comes from?
The actual problem is, that I need TLS + Auth. I could do that with program(), if I install an MTA that can do at least auth and queuing, and a wrapper for TLS. However, that's something I want to avoid for various reasons, ranging from limiting the installed and running programs to a minimum to finding it easier to debug & maintain when it's built into syslog-ng itself. If the smtp driver fails, I'll see that immediately from the logs, and can start debugging right away. If an external program fails, I will see the fail in the logs, then will have to start figuring out WHY it failed (which is something I already know from the smtp driver's logs), and can only start debugging then. Using an external program also means I will have to learn - sooner or later - how to efficiently debug it. I already know how to handle syslog-ng. It's a corner case, indeed, highly motivated by personal preferences. I do not think that this makes it a bad itch to scratch, though. It's merely an itch that few people have.
On a passing note, what would be handy is some v6 netmask() action...much obliged if you find and scratch that itch :)
Hnnngnhn. And I was so happy that I scratched this itch, you find me another one! Argh. (Added to my TODO list :P). -- |8]