[PATCH]: Experimental SMTP destination
Hi! I've had a little free time, and started toying with the idea of implementing an SMTP destination for syslog-ng. So far, it's progressing nicely, but there are a few shortcomings, as I will explain later. For anyone interested, it can be found on the 'work/modules/afsmtp' branch of my syslog-ng-3.3 tree: git://git.balabit.hu/algernon/syslog-ng-3.3.git Or, one can browse the branch on the web: http://git.balabit.hu/?p=algernon/syslog-ng-3.3.git;a=shortlog;h=work/module... One will need the libesmtp library installed for the module to work. It should be available in most distributions, but if one doesn't have it, it can be downloaded from http://www.stafford.uklinux.net/libesmtp/ (although, as of this writing, the site seems to be down). The usage is something like this: destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); }; One can specify any number of recipients, and the mail will be sent to all of them. Custom headers can also be specified, but at this time, a selected few headers do not work as one would expect, due to libesmtp handling them specially. These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines). Specifying mail_from or subject multiple times will override any previous values. Adding the same custom header multiple times will only add the first (though, plans include changing that, so that one can include the same header multiple times, with different values - if I can find a use-case for that). There are also a few other minor issues here and there, but nothing major, as far as I could check. Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet. -- |8]
Gergely Nagy <algernon@balabit.hu> wrote:
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
The sysadmin in me does say a pipe to 'sendmail' is probably more straight forward, flexible and reliable: ---- destination d_mail { program("sendmail -f noreply@example.com admin@example.com admin2@example.com" template("From: syslog-ng alert service <noreply@example.com>\nSubject: [ALERT] $LEVEL condition in $PROGRAM!\nX-Program: $PROGRAM\nX-Facility: $FACILITY\n\nHi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n.\n")); } ---- The closing '\n.\n' does the EOF.
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
A task that should be left surely to the local MSA (mail submission agent)? Cheers -- Alexander Clouter .sigmonster says: Drilling for oil is boring.
On Sat, 2011-02-12 at 19:06 +0000, Alexander Clouter wrote:
Gergely Nagy <algernon@balabit.hu> wrote:
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
The sysadmin in me does say a pipe to 'sendmail' is probably more straight forward, flexible and reliable: ---- destination d_mail { program("sendmail -f noreply@example.com admin@example.com admin2@example.com" template("From: syslog-ng alert service <noreply@example.com>\nSubject: [ALERT] $LEVEL condition in $PROGRAM!\nX-Program: $PROGRAM\nX-Facility: $FACILITY\n\nHi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n.\n")); } ----
The closing '\n.\n' does the EOF.
Yes, that's a possibility, indeed, but that assumes one has a sendmail command installed.
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
A task that should be left surely to the local MSA (mail submission agent)?
Yep, that is an option indeed, and has advantages over a built-in driver. However, it is an option that I do not like all that much. Having a built-in smtp destination driver has a few advantages over using a program destination: * You don't need an external application (my #1 reason, apart from "because I could"). * No need to fork off to launch the external program (though, this isn't a big issue, since email alerts will probably be rather rare, so the fork & launch overhad is negligible) * Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place * Safer: If $MSG happens to be multi-line, and one manages to craft a message with an embedded "\r\n.\r\n", we're in trouble. Similar things could be done to the headers aswell. Of course, that can be guarded against, but then the program destination becomes considerably different, and one would need a wrapper program. Or escaping template functions (which would be useful, if we don't have any yet..) That, and having the option to do it without an external program was one of the driving forces behind the code (I really, really don't like calling external programs, if I can avoid it). -- |8]
Gergely Nagy <algernon@balabit.hu> wrote:
[snipped sendmail approach]
Yes, that's a possibility, indeed, but that assumes one has a sendmail command installed.
For those crazy enough to run syslog-ng on a non-sendmail environment, there are alternatives: http://glob.com.au/sendmail/
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
It does not answer "where did my email alert go?" Did syslog eat it? Did the smarthost toast it? Was it lost further upstream?
* Safer: If $MSG happens to be multi-line, and one manages to craft a message with an embedded "\r\n.\r\n", we're in trouble. Similar things could be done to the headers aswell. Of course, that can be guarded against, but then the program destination becomes considerably different, and one would need a wrapper program. Or escaping template functions (which would be useful, if we don't have any yet..)
No need to guard against it, add support into syslog-ng to send EOF at the end of each message and you use that as your magic marker instead.
That, and having the option to do it without an external program was one of the driving forces behind the code (I really, really don't like calling external programs, if I can avoid it).
Probably time to stop using UNIX :P Cheers -- Alexander Clouter .sigmonster says: "Nuclear war would really set back cable." -- Ted Turner
On Sat, 2011-02-12 at 22:56 +0000, Alexander Clouter wrote:
Gergely Nagy <algernon@balabit.hu> wrote:
[snipped sendmail approach]
Yes, that's a possibility, indeed, but that assumes one has a sendmail command installed.
For those crazy enough to run syslog-ng on a non-sendmail environment, there are alternatives:
Sadly, that does not run on my router.
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
It does not answer "where did my email alert go?" Did syslog eat it? Did the smarthost toast it? Was it lost further upstream?
Of course it does not make me able to troubleshoot issues beyond the box itself, but it does make it easier to see whether the message left the system at all, as I only have to look at syslog-ng's debug output, and don't have to hunt down wherever sendmail logged to.
* Safer: If $MSG happens to be multi-line, and one manages to craft a message with an embedded "\r\n.\r\n", we're in trouble. Similar things could be done to the headers aswell. Of course, that can be guarded against, but then the program destination becomes considerably different, and one would need a wrapper program. Or escaping template functions (which would be useful, if we don't have any yet..)
No need to guard against it, add support into syslog-ng to send EOF at the end of each message and you use that as your magic marker instead.
Well, consider a log message sent by a malicious client: "blahblah\r\n.\r\nmail from:<>\r\nrcpt to:<somewhere@example.com>\r\ndata\r\nblahblah\r\n.\r\n" As far as I remember, the syslog protocol (the new one) allows embeded newlines, so such messages should be accounted for, one way or the other.
That, and having the option to do it without an external program was one of the driving forces behind the code (I really, really don't like calling external programs, if I can avoid it).
Probably time to stop using UNIX :P
I only use unix to boot into emacs. I guess it shows. ;) -- |8]
Gergely Nagy <algernon@balabit.hu> wrote:
[snipped sendmail approach]
Yes, that's a possibility, indeed, but that assumes one has a sendmail command installed.
For those crazy enough to run syslog-ng on a non-sendmail environment, there are alternatives:
Sadly, that does not run on my router.
My 4MB NAND router has a sendmail binary and nullmailer is also readily available. I am refering to the wrapper that does mail submission into the local queue. http://www.busybox.net/downloads/BusyBox.html#sendmail I would not suggest for a moment someone runs the actual full sendmail mail routing server. :)
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
It does not answer "where did my email alert go?" Did syslog eat it? Did the smarthost toast it? Was it lost further upstream?
Of course it does not make me able to troubleshoot issues beyond the box itself, but it does make it easier to see whether the message left the system at all, as I only have to look at syslog-ng's debug output, and don't have to hunt down wherever sendmail logged to.
It logs where all your existing mail server logs go to. Just sayin' Cheers -- Alexander Clouter .sigmonster says: Shipping not included.
On Sun, 2011-02-13 at 09:34 +0000, Alexander Clouter wrote:
Gergely Nagy <algernon@balabit.hu> wrote:
[snipped sendmail approach]
Yes, that's a possibility, indeed, but that assumes one has a sendmail command installed.
For those crazy enough to run syslog-ng on a non-sendmail environment, there are alternatives:
Sadly, that does not run on my router.
My 4MB NAND router has a sendmail binary and nullmailer is also readily available. I am refering to the wrapper that does mail submission into the local queue.
Yep, I know about that, but if I want mail delivered, I wouldn't rely on it. If there's a network outage, or the destination server is not accessible for whatever reason, busybox's sendmail will not attempt to re-deliver the message. My driver will notice such outages, and keep the message in its queue, thus, preventing message loss. This also guards against /usr/bin/sendmail failing for whatever reasons (misconfiguration, upgrade in progress, break-in, and so on). Nullmailer or similar small solutions can help with that, of course, but why would I want to install nullmailer, when syslog-ng can do the mailing without external help? -- |8]
On Feb 12, 2011, at 16:08, "Alexander Clouter" <alex@digriz.org.uk> wrote:
Gergely Nagy <algernon@balabit.hu> wrote:
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
It does not answer "where did my email alert go?" Did syslog eat it? Did the smarthost toast it? Was it lost further upstream?
How is the any different than any other destination? SMS, MongoDB, SQL, etc. all have the same problem if it isn't where it is expected. Even files. Cheers, Bill
Bill Anderson <Bill.Anderson@bodybuilding.com> wrote:
On Feb 12, 2011, at 16:08, "Alexander Clouter" <alex@digriz.org.uk> wrote:
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
It does not answer "where did my email alert go?" Did syslog eat it? Did the smarthost toast it? Was it lost further upstream?
How is the any different than any other destination? SMS, MongoDB, SQL, etc. all have the same problem if it isn't where it is expected. Even files.
I phrased this badly. My point is that you have to look in the mail logs anyway, any syslog-ng logs is not going to tell you anything a call to the local sendmail binary does not. Cheers -- Alexander Clouter .sigmonster says: Being schizophrenic is better than living alone.
On Sun, 2011-02-13 at 09:27 +0000, Alexander Clouter wrote:
Bill Anderson <Bill.Anderson@bodybuilding.com> wrote:
On Feb 12, 2011, at 16:08, "Alexander Clouter" <alex@digriz.org.uk> wrote:
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
It does not answer "where did my email alert go?" Did syslog eat it? Did the smarthost toast it? Was it lost further upstream?
How is the any different than any other destination? SMS, MongoDB, SQL, etc. all have the same problem if it isn't where it is expected. Even files.
I phrased this badly. My point is that you have to look in the mail logs anyway, any syslog-ng logs is not going to tell you anything a call to the local sendmail binary does not.
That's not entirely accurate. If the problem is persistent, and can be reproduced with syslog-ng running in debug mode, my driver will echo back most of the SMTP session, so one can easily see whether the problem is between syslog-ng or the SMTP server used. Calling sendmail may or may not reveal that information: if you have a local mail setup that works, sendmail will accept the message. Then when the local MTA tries to deliver the message, it will encounter the same error then, and one will have to dig into the logs, which varies from MTA to MTA. With the program destination, one will need to assume that if there's network outage or any other error, the MTA will queue the mail and retry later. While this is true in most cases, it's not always the case. Thus, when one has an MTA that doesn't do this, program() can easily lead to lost messages. Using a built-in driver that uses syslog-ng's built in queuing will not have such problems. -- |8]
Hi, Op 12-02-11 20:34, Gergely Nagy schreef:
However, it is an option that I do not like all that much. Having a built-in smtp destination driver has a few advantages over using a program destination:
* You don't need an external application (my #1 reason, apart from "because I could"). * No need to fork off to launch the external program (though, this isn't a big issue, since email alerts will probably be rather rare, so the fork & launch overhad is negligible)
So isn't the built-in MTA a solution in search of a problem? Also, I see some problems with the built-in SMTP destination. Misconfiguration now is a problem for syslog-ng itself - instead of the external MTA. This, of course, has two sides: when the misconfiguration consists of a log/mailing loop, using an external MTA could be a serious problem when tens of MTA instances are started in parallel; but an internal MTA will bring the system to it's knees by immediately filling memory, disk and the network with repeated messages - without any external help ;-) However, at the moment your smtp destination does not seem to have a "throttle" configuration setting, while program() does have one.
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
I'm pretty sure it won't be too hard to configure syslog-ng so, that it's pretty easy to troubleshoot the MTA that it sends it's messages to. But the main point is, that I don't think that an internal SMTP destination will do any good for misconfiguration. I.e. misconfiguration is a problem in itself, and no matter how many destinations you add to syslog-ng, you won't be able to prevent misconfiguration.
That, and having the option to do it without an external program was one of the driving forces behind the code (I really, really don't like calling external programs, if I can avoid it).
Being a Linux user, I don't think calling external programs is a problem per se. Yes, it has overhead, but in this case, the overhead seems perfectly acceptible. A problem that was not yet mentioned is, that having an SMTP destination now makes syslog-ng dependent on yet another library, libesmtp. All in all, it sounds like the perfect solution for a certain widely-used proprietary operating system, that has no proper built-in SMTP possibilities. And because the SMTP destination is configurable with ENABLE_SMTP, Linux builds can easily avoid the new dependency ;) Best regards, Valentijn
On Mon, 2011-02-14 at 09:46 +0100, Valentijn Sessink wrote:
Op 12-02-11 20:34, Gergely Nagy schreef:
However, it is an option that I do not like all that much. Having a built-in smtp destination driver has a few advantages over using a program destination:
* You don't need an external application (my #1 reason, apart from "because I could"). * No need to fork off to launch the external program (though, this isn't a big issue, since email alerts will probably be rather rare, so the fork & launch overhad is negligible)
So isn't the built-in MTA a solution in search of a problem?
Nope, it isn't. I have routers that do not have a reliable MTA, and I don't intend to install one on them for the sole purpose of sending syslog alerts. There might be N other ways to accomplish the same thing, a built-in smtp destination was the easiest for me, so I coded it up. All the other soultions I could think of would have taken me more time and effort. I had an itch, I scratched it, I'm happy.
Also, I see some problems with the built-in SMTP destination. Misconfiguration now is a problem for syslog-ng itself - instead of the external MTA.
Yes, and that is good so. I will notice such misconfigurations far sooner.
This, of course, has two sides: when the misconfiguration consists of a log/mailing loop, using an external MTA could be a serious problem when tens of MTA instances are started in parallel; but an internal MTA will bring the system to it's knees by immediately filling memory, disk and the network with repeated messages - without any external help ;-)
Erm, no. The smtp destination has a configurable queue, if it fills up, it will start to drop messages. That way, it will not consume all memory. As for filling the disk with repeated messages, that's what suppress() is for in the file destination. Yes, you can bring the system to its knees, but one can do that without the help of the smtp destination too. The same 'problem' applies to program() too, by the way.
However, at the moment your smtp destination does not seem to have a "throttle" configuration setting, while program() does have one.
Oops, good catch! Although, that's just a bug in the grammar file (and about 4 lines to fix, which I will do tonight after work).
* Tighter integration with syslog-ng allows for easier troubleshooting: one only has to look at one place
I'm pretty sure it won't be too hard to configure syslog-ng so, that it's pretty easy to troubleshoot the MTA that it sends it's messages to.
The thing is, I don't want a full-blown MTA on the same host that syslog-ng runs on. So at best, I can have a program that forwards alerts to an external host. Now this has two consequences: - While it is easy to troubleshoot the forwarder, the MTA is running on another host. That alone makes it slightly more complicated to troubleshoot it, as there's one more variable in the picture. - I will either need a reasonably feature-full forwarder that can do TLS and authentication, so I can use the SMTP server of my choice, or I need an intermediate server that can accept mails from the syslog host, and forward it (using TLS & auth) to the real smtp server. Having an built-in smtp destination, that will have tls & auth real soon, does not have these issues.
But the main point is, that I don't think that an internal SMTP destination will do any good for misconfiguration. I.e. misconfiguration is a problem in itself, and no matter how many destinations you add to syslog-ng, you won't be able to prevent misconfiguration.
Correct. But as far as I'm concerned, I find it easier to debug one program, on one host, than go through N hops to find which one is misbehaving. With the program() destination, in order for my alerts to leave my system, it would go through several hops. Assuming a sendmail that can do TLS and auth (best case): syslog-ng -> sendmail -> external SMTP host With a bit of luck, the sendmail program will do proper logging, so I can troubleshoot it easily. Hopefully, it will also do queuing, either in memory, or on disk, so that when the SMTP host is unreachable, my alerts will not get lost. There's too many ifs in there, and I'm really not interested in finding a small enough forwarder that can do all that. I know busybox can't, nor can nullmailer. Dragonfly Mail Agent (dma) might be a good candidate, but I'd need to review it. Esmtp is relay-only, and does not queue. That's about all of the lighter MTA solutiosn available in Debian. Thus, it looks like I need an intermediate step: syslog-ng -> sendmail -> internal SMTP relay -> external SMTP host The problem with this, is that things can go wrong at yet another place, and if the internal relay is not accessible for one reason or the other (eg, it's restarting due to an upgrade), since the sendmail program does not queue, and does not retry, there's a reasonable chance of alerts disappearing. With a built-in smtp destination, the setup would look like this: syslog-ng -> external SMTP host Boom. The smtp driver will detect if there's a network outage, and queue the mail, and retry later. It does this using syslog-ng's built-in queuing mechanism. I don't need to go an extra mile to make sure that any message sent from syslog-ng will be retried at least until the queue fills up.
That, and having the option to do it without an external program was one of the driving forces behind the code (I really, really don't like calling external programs, if I can avoid it).
Being a Linux user, I don't think calling external programs is a problem per se. Yes, it has overhead, but in this case, the overhead seems perfectly acceptible.
As I said, the overhead is negligible indeed. It's not the overhead I'm worried about in this case.
A problem that was not yet mentioned is, that having an SMTP destination now makes syslog-ng dependent on yet another library, libesmtp.
It's optional, though. By that logic, we shouldn't have sql() destinations, either, since one can just use program() with a perl script. We wouldn't need tls, either, since one can set up a tunnel. We wouldn't need tcp() and the rest, either, since we can use netcat...
All in all, it sounds like the perfect solution for a certain widely-used proprietary operating system, that has no proper built-in SMTP possibilities. And because the SMTP destination is configurable with ENABLE_SMTP, Linux builds can easily avoid the new dependency ;)
If you're talking about the OS I think you're talking about, let me assure you, I wouldn't touch it with a ten feet pole if I can avoid it (not to mention, that libestmp is most probably not available for said platform). -- |8]
Gergely Nagy <algernon@balabit.hu> wrote:
So isn't the built-in MTA a solution in search of a problem?
Nope, it isn't. I have routers that do not have a reliable MTA, and I don't intend to install one on them for the sole purpose of sending syslog alerts.
Then make them reliable (for me it was 'aptitude install nullmailer', not complicated). How do you plan to send cron job alerts to the sysadmin if there is no local MTA? All UNIX hosts really should have a way to send email to their sysadmin, especially if they are a crucial part of the organisation's infrastructure.
There might be N other ways to accomplish the same thing, a built-in smtp destination was the easiest for me, so I coded it up. All the other soultions I could think of would have taken me more time and effort.
N ways that already exist, and have worked for ~30 years.
I had an itch, I scratched it, I'm happy.
Well, that's a different and perfectly okay objective. The issue is that sending mail from a UNIX host is an already solved problem.
Also, I see some problems with the built-in SMTP destination. Misconfiguration now is a problem for syslog-ng itself - instead of the external MTA.
Yes, and that is good so. I will notice such misconfigurations far sooner.
A non-zero mail queue or rejections lurking in /var/log/mail are trivial to notice.
I'm pretty sure it won't be too hard to configure syslog-ng so, that it's pretty easy to troubleshoot the MTA that it sends it's messages to.
The thing is, I don't want a full-blown MTA on the same host that syslog-ng runs on. So at best, I can have a program that forwards alerts to an external host.
Nullmailer is *not* a full-blown MTA but it does reliably deliver mail to a smarthost. It's a 500kB (100kB binary) install with *all* the man pages and Debian documentation. Also, the flexibility you could call upon with 'mailx' (for example, grep a log file and append the +-20 lines to the email) is far more than what could be put into syslog-ng.
- While it is easy to troubleshoot the forwarder, the MTA is running on another host. That alone makes it slightly more complicated to troubleshoot it, as there's one more variable in the picture.
I think you are seeing a problem where one does not exist.
- I will either need a reasonably feature-full forwarder that can do TLS and authentication, so I can use the SMTP server of my choice, or I need an intermediate server that can accept mails from the syslog host, and forward it (using TLS & auth) to the real smtp server.
Nullmailer does auth whilst nbsmtp can do tls/auth/queue.
Having an built-in smtp destination, that will have tls & auth real soon, does not have these issues.
These are problems that do not exist. Can you tell me what problems you have seen (not theorised) there are by just forking sendmail (wrapper)/nbsmtp/whatever?
But the main point is, that I don't think that an internal SMTP destination will do any good for misconfiguration. I.e. misconfiguration is a problem in itself, and no matter how many destinations you add to syslog-ng, you won't be able to prevent misconfiguration.
Correct. But as far as I'm concerned, I find it easier to debug one program, on one host, than go through N hops to find which one is misbehaving.
It's far more difficult to ram the SMTP RFC's into a syslog daemon.
With the program() destination, in order for my alerts to leave my system, it would go through several hops.
Assuming a sendmail that can do TLS and auth (best case):
syslog-ng -> sendmail -> external SMTP host
'best' = 'typical'
With a bit of luck, the sendmail program will do proper logging, so I can troubleshoot it easily. Hopefully, it will also do queuing, either in memory, or on disk, so that when the SMTP host is unreachable, my alerts will not get lost.
'luck' = 'typical' 'queue' is down to personal preference, but as I keep hammering, nullmailer and nbsmtp both do queuing and have done so for *years*. I am guessing syslog-ng will not queue a 10kB email for me (as it would be nice to call on mailx's '~!').
There's too many ifs in there, and I'm really not interested in finding a small enough forwarder that can do all that. I know busybox can't, nor can nullmailer. Dragonfly Mail Agent (dma) might be a good candidate, but I'd need to review it. Esmtp is relay-only, and does not queue.
nullmailer can and does queue, I use it on my laptop to get mail off to a smarthost. It also logs.
With a built-in smtp destination, the setup would look like this:
syslog-ng -> external SMTP host
Boom. The smtp driver will detect if there's a network outage, and queue the mail, and retry later. It does this using syslog-ng's built-in queuing mechanism. I don't need to go an extra mile to make sure that any message sent from syslog-ng will be retried at least until the queue fills up.
nullmailer....nullmailer....nullmailer. :-/ If we say, "yes there is a void in lightweight MTA's that can do queuing, auth and TLS" the fix is to dedicate your efforts to produce one. Then *everything* can use it...not just syslog-ng. Putting a SMTP transport in syslog-ng helps no one, but the syslog-ng users.
A problem that was not yet mentioned is, that having an SMTP destination now makes syslog-ng dependent on yet another library, libesmtp.
It's optional, though. By that logic, we shouldn't have sql() destinations, either, since one can just use program() with a perl script. We wouldn't need tls, either, since one can set up a tunnel. We wouldn't need tcp() and the rest, either, since we can use netcat...
I am forced to use program to open 'psql' as syslog-ng cannot do simple "SELECT function('foobar', 'cheese', 'toys');" and also a Perl script to process squid log files too[1] as syslog-ng cannot do xyz; tasks which are syslog-ng specific... These work *extremely* well. Comparing smtp() to sql() is not quite like for like. You get additional functionality when accessing it natively such as prepared statements and such. Natively speaking SMTP does not buy you anything that I am aware of? Of course I guess it could be argued no one is forcing me to use smtp(), which would be a fair point. :)
All in all, it sounds like the perfect solution for a certain widely-used proprietary operating system, that has no proper built-in SMTP possibilities. And because the SMTP destination is configurable with ENABLE_SMTP, Linux builds can easily avoid the new dependency ;)
If you're talking about the OS I think you're talking about, let me assure you, I wouldn't touch it with a ten feet pole if I can avoid it (not to mention, that libestmp is most probably not available for said platform).
Ironically libesmtp is larger than nullmailer on my amd64 Debian laptop. I think the reference you are missing here is that with Windows it is *very* difficult (unsurprisingly) to apply the original UNIX philosophy of stringing together many small simple tools that do their singular task very well to perform the job you need. In Windows, if the application you are using does not support what you need it to do...you are generally out of luck. What do you do when given the task of parsing a CSV file (overlook escaping) to summing up the numbers in the fourth column and producing the average? Under UNIX would you reach for OpenOffice or awk? Under Windows you typically have no choice but to do the task in Excel. This is what I'm trying to communicate over, possibly poorly. I really am just very curious what the actual *real* problem is, and where all this supposed unreliability in calling sendmail comes from? On a passing note, what would be handy is some v6 netmask() action...much obliged if you find and scratch that itch :) Cheers [1] http://www.digriz.org.uk/syslog-ng-integration -- Alexander Clouter .sigmonster says: In the stairway of life, you'd best take the elevator.
On Mon, 2011-02-14 at 14:37 +0000, Alexander Clouter wrote:
Gergely Nagy <algernon@balabit.hu> wrote:
So isn't the built-in MTA a solution in search of a problem?
Nope, it isn't. I have routers that do not have a reliable MTA, and I don't intend to install one on them for the sole purpose of sending syslog alerts.
Then make them reliable (for me it was 'aptitude install nullmailer', not complicated).
nullmailer fails at supporting tls, so even if I install it, I still need an intermediate relay, or another tool to provide TLS for me. At which point, I can just add smtp() to syslog-ng, teach it TLS & auth (both of which is supported by libesmtp, I only need a few lines of glue code in the destination driver) and I'm better off that way. Less extra space, less stuff to configure, less points of potential failure.
How do you plan to send cron job alerts to the sysadmin if there is no local MTA? All UNIX hosts really should have a way to send email to their sysadmin, especially if they are a crucial part of the organisation's infrastructure.
Those alerts are logged to syslog, and my smtp destination can send the emails. I don't need a local MTA, nor do I desire one.
There might be N other ways to accomplish the same thing, a built-in smtp destination was the easiest for me, so I coded it up. All the other soultions I could think of would have taken me more time and effort.
N ways that already exist, and have worked for ~30 years.
N ways already exist, neither of which is suitable for my use-case. I might be the first guy in 30 years, in which case I feel honored.
I had an itch, I scratched it, I'm happy.
Well, that's a different and perfectly okay objective. The issue is that sending mail from a UNIX host is an already solved problem.
Provided that said host has the tools for it, yes. None of the tools available in debian were found to fit my needs. Unless combined, which IS the UNIX way, but is something I want to avoid in this case.
I'm pretty sure it won't be too hard to configure syslog-ng so, that it's pretty easy to troubleshoot the MTA that it sends it's messages to.
The thing is, I don't want a full-blown MTA on the same host that syslog-ng runs on. So at best, I can have a program that forwards alerts to an external host.
Nullmailer is *not* a full-blown MTA but it does reliably deliver mail to a smarthost. It's a 500kB (100kB binary) install with *all* the man pages and Debian documentation.
It does not support TLS, though, as far as I saw. Thus, I'd still need an intermediate relay, or another program to provide TLS for me.
Having an built-in smtp destination, that will have tls & auth real soon, does not have these issues.
These are problems that do not exist. Can you tell me what problems you have seen (not theorised) there are by just forking sendmail (wrapper)/nbsmtp/whatever?
Problem #1: I need a suitable sendmail on the host. Problem #2: Depending on the MTA, I will likely need something else to provide TLS support for it. Problem #3: External tools do not provide anything over my smtp driver, at least not in my use-case, while my smtp driver solves my immediate issue of not having to install external tools to get the job done. Yes, the problem can be solved with program() in most cases. My case is not the common case, though, and I do not see harm in catering to a few corner-cases when it does not disrupt anything else.
But the main point is, that I don't think that an internal SMTP destination will do any good for misconfiguration. I.e. misconfiguration is a problem in itself, and no matter how many destinations you add to syslog-ng, you won't be able to prevent misconfiguration.
Correct. But as far as I'm concerned, I find it easier to debug one program, on one host, than go through N hops to find which one is misbehaving.
It's far more difficult to ram the SMTP RFC's into a syslog daemon.
Which is why I didn't do that, but rely on libesmtp, which does the bulk of the job for me. I don't need all of the SMTP RFC, only a subset, which libesmtp implements. If I ever need more than that, I will either extend libesmtp, or find a way to use program(). Right now, program() would require more work from me, than the smtp destination did.
With the program() destination, in order for my alerts to leave my system, it would go through several hops.
Assuming a sendmail that can do TLS and auth (best case):
syslog-ng -> sendmail -> external SMTP host
'best' = 'typical'
Yes. But I see no harm in making corner-cases easy too. Typical installs can still use program() if so they wish, and they need the extra features (if any).
With a bit of luck, the sendmail program will do proper logging, so I can troubleshoot it easily. Hopefully, it will also do queuing, either in memory, or on disk, so that when the SMTP host is unreachable, my alerts will not get lost.
'luck' = 'typical'
'queue' is down to personal preference, but as I keep hammering, nullmailer and nbsmtp both do queuing and have done so for *years*. I am guessing syslog-ng will not queue a 10kB email for me (as it would be nice to call on mailx's '~!').
nullmailer doesn't have TLS, so I'd have to install yet another tool to fix that, or use an intermediate relay. I'd rather not do either. I tried to check what nbsmtp can do, but it's not available on neither Lucid, nor Squeeze, and the homepage returns a default Debian Apache page. As for queuing: syslog-ng will queue any log messages that it couldn't send via the smtp driver (the message, not the email), and will reconstruct the mail at a later point.
There's too many ifs in there, and I'm really not interested in finding a small enough forwarder that can do all that. I know busybox can't, nor can nullmailer. Dragonfly Mail Agent (dma) might be a good candidate, but I'd need to review it. Esmtp is relay-only, and does not queue.
nullmailer can and does queue, I use it on my laptop to get mail off to a smarthost. It also logs.
But, as I said, it doesn't do TLS. All of the listed solutions do part of what I want, but neither does it all. Thus, if I go with either, I will have to install something else aswell, which just complicates things. A built-in smtp() driver is far easier for me. It can do everything I want, in one place.
With a built-in smtp destination, the setup would look like this:
syslog-ng -> external SMTP host
Boom. The smtp driver will detect if there's a network outage, and queue the mail, and retry later. It does this using syslog-ng's built-in queuing mechanism. I don't need to go an extra mile to make sure that any message sent from syslog-ng will be retried at least until the queue fills up.
nullmailer....nullmailer....nullmailer. :-/
TLS... TLS... TLS.
If we say, "yes there is a void in lightweight MTA's that can do queuing, auth and TLS" the fix is to dedicate your efforts to produce one. Then *everything* can use it...not just syslog-ng. Putting a SMTP transport in syslog-ng helps no one, but the syslog-ng users.
I'm fine with helping syslog-ng users only. Producing a suitable MTA that can do everything I want would require far more effort than producing the smtp destination took.
I am forced to use program to open 'psql' as syslog-ng cannot do simple "SELECT function('foobar', 'cheese', 'toys');" and also a Perl script to process squid log files too[1] as syslog-ng cannot do xyz; tasks which are syslog-ng specific... These work *extremely* well.
So sql() has next to no use, let's remove it, it's useless. (I'd actually agree with that - sql-specific drivers, which exploit the features specific to said sql server would be of much more value, in my opinion)
Comparing smtp() to sql() is not quite like for like. You get additional functionality when accessing it natively such as prepared statements and such. Natively speaking SMTP does not buy you anything that I am aware of?
I already listed why I prefer a native driver over using program(), but let me reiterate: * I want TLS, auth and queuing, preferably in one place. None of the existing lightweight solutions I know of provide that in themselves. Yes, nullmailer comes close, but it does lack TLS, which forces me to setup or install yet another tool. * program() was designed with a long-running process in mind, and even though syslog-ng will restart a program that exists, that's not the intended usage. Yes, one can write a wrapper that doesn't exist, but then I could just use the smtp destination. It also restarts the program, if it exits, but personally, I find that an abuse, and if I'd use program(), I'd do that with a long-running wrapper instead. And then it's easier to use smtp()... * A built-in driver suited my immediate needs best, it's optional, noone's forced to use it if they're happy with something else.
All in all, it sounds like the perfect solution for a certain widely-used proprietary operating system, that has no proper built-in SMTP possibilities. And because the SMTP destination is configurable with ENABLE_SMTP, Linux builds can easily avoid the new dependency ;)
If you're talking about the OS I think you're talking about, let me assure you, I wouldn't touch it with a ten feet pole if I can avoid it (not to mention, that libestmp is most probably not available for said platform).
Ironically libesmtp is larger than nullmailer on my amd64 Debian laptop.
That's interesting. libesmtp.so is ~72k on my amd64 lucid, while nullmailer (/usr/lib/nullmailer/* + /usr/bin/nullmailer-inject + /usr/sbin/nullmailer*) is ~172k. I'm not sure how much of that I'd need for simple forwarding, but I assume I need at least nullmailer-queue and nullmailer-send, which are 60k. And indeed, 60k is smaller than 72k, but libesmtp also does TLS, which nullmailer does not.
I think the reference you are missing here is that with Windows it is *very* difficult (unsurprisingly) to apply the original UNIX philosophy of stringing together many small simple tools that do their singular task very well to perform the job you need. In Windows, if the application you are using does not support what you need it to do...you are generally out of luck.
I suppose you're right. I do not know Windows well enough to make an informed opinion (and I'd prefer to keep it that way).
What do you do when given the task of parsing a CSV file (overlook escaping) to summing up the numbers in the fourth column and producing the average? Under UNIX would you reach for OpenOffice or awk? Under Windows you typically have no choice but to do the task in Excel.
I'd go with awk, yes. But if I had an application that would need this data, and I had a library to parse CSV files, I probably wouldn't call out to awk, but use the parser library instead. It does depend on the use case, though. If I'm writing a shell script, then awk it is. If I'm writing a GUI application, I'd rather use a library instead of calling out to awk.
This is what I'm trying to communicate over, possibly poorly. I really am just very curious what the actual *real* problem is, and where all this supposed unreliability in calling sendmail comes from?
The actual problem is, that I need TLS + Auth. I could do that with program(), if I install an MTA that can do at least auth and queuing, and a wrapper for TLS. However, that's something I want to avoid for various reasons, ranging from limiting the installed and running programs to a minimum to finding it easier to debug & maintain when it's built into syslog-ng itself. If the smtp driver fails, I'll see that immediately from the logs, and can start debugging right away. If an external program fails, I will see the fail in the logs, then will have to start figuring out WHY it failed (which is something I already know from the smtp driver's logs), and can only start debugging then. Using an external program also means I will have to learn - sooner or later - how to efficiently debug it. I already know how to handle syslog-ng. It's a corner case, indeed, highly motivated by personal preferences. I do not think that this makes it a bad itch to scratch, though. It's merely an itch that few people have.
On a passing note, what would be handy is some v6 netmask() action...much obliged if you find and scratch that itch :)
Hnnngnhn. And I was so happy that I scratched this itch, you find me another one! Argh. (Added to my TODO list :P). -- |8]
On 02/12/2011 05:12:55 PM, Gergely Nagy wrote:
Hi!
I've had a little free time, and started toying with the idea of implementing an SMTP destination for syslog-ng. So far, it's progressing nicely, but there are a few shortcomings, as I will explain later.
For anyone interested, it can be found on the 'work/modules/afsmtp' branch of my syslog-ng-3.3 tree:
git://git.balabit.hu/algernon/syslog-ng-3.3.git
Or, one can browse the branch on the web:
http://git.balabit.hu/?p=algernon/syslog- ng-3.3.git;a=shortlog;h=work/modules/afsmtp
One will need the libesmtp library installed for the module to work. It should be available in most distributions, but if one doesn't have it, it can be downloaded from http://www.stafford.uklinux.net/libesmtp/ (although, as of this writing, the site seems to be down).
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
One can specify any number of recipients, and the mail will be sent to all of them. Custom headers can also be specified, but at this time, a selected few headers do not work as one would expect, due to libesmtp handling them specially.
These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines).
Specifying mail_from or subject multiple times will override any previous values. Adding the same custom header multiple times will only add the first (though, plans include changing that, so that one can include the same header multiple times, with different values - if I can find a use-case for that).
There are also a few other minor issues here and there, but nothing major, as far as I could check.
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
Authentication & starttls are nice things, but making e-mail sending available as patterndb actions would be much cooler, IMHO. (Even though there is probably a workaround for it even with this driver.) Nice work! Robert
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Sat, 2011-02-12 at 20:07 +0100, Robert Fekete wrote:
On 02/12/2011 05:12:55 PM, Gergely Nagy wrote:
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
Authentication & starttls are nice things, but making e-mail sending available as patterndb actions would be much cooler, IMHO. (Even though there is probably a workaround for it even with this driver.)
From what I see, actions generate messages to internal(), so one can catch them there, filter and route to a mail destination, much like the example in the syslog-ng docs that sends triggered messages to an external application: just replace the d_triggers destination.
I don't think implementing direct sending from patterndb would be a good idea, for various reasons. One being I have no idea how I'd go around coding it properly. -- |8]
Hi, Thanks for your contribution, some comments below. On Sat, 2011-02-12 at 17:12 +0100, Gergely Nagy wrote:
Hi!
I've had a little free time, and started toying with the idea of implementing an SMTP destination for syslog-ng. So far, it's progressing nicely, but there are a few shortcomings, as I will explain later.
For anyone interested, it can be found on the 'work/modules/afsmtp' branch of my syslog-ng-3.3 tree:
git://git.balabit.hu/algernon/syslog-ng-3.3.git
Or, one can browse the branch on the web:
http://git.balabit.hu/?p=algernon/syslog-ng-3.3.git;a=shortlog;h=work/module...
One will need the libesmtp library installed for the module to work. It should be available in most distributions, but if one doesn't have it, it can be downloaded from http://www.stafford.uklinux.net/libesmtp/ (although, as of this writing, the site seems to be down).
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
I'd love to get away a little bit from SMTP logic, and use option names that reflect more closely what the user sees in a MUA. We might give a chance to specify all the rest of the details, but they should be the exception and not the rule. smtp( host("localhost") port(25) from("syslog-ng alert service <noreply@example.com>") to("Admin #1 <admin@example.com>", "Admin #2 <admin2@example.com>") cc("Admin BOSS <admin.boss@example.com>") bcc("Blind CC <blindcc@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") extra-header("X-Program", "$PROGRAM") ); I doubt it'd be useful to use templates within To/Cc/Bcc, but the rest should be templates, with support for embedded newline characters. (headers can also contain embedded newlines, by embedding NL and adding whitespace in the front of the upcoming line, although I'm not sure how much MUAs like it). The same syntax could perhaps be used by a possible "sendmail()" destination, which could spawn the local sendmail binary, instead of directly using SMTP. Otherwise I really like the functionality, great stuff.
One can specify any number of recipients, and the mail will be sent to all of them. Custom headers can also be specified, but at this time, a selected few headers do not work as one would expect, due to libesmtp handling them specially.
These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines).
Specifying mail_from or subject multiple times will override any previous values. Adding the same custom header multiple times will only add the first (though, plans include changing that, so that one can include the same header multiple times, with different values - if I can find a use-case for that).
There are also a few other minor issues here and there, but nothing major, as far as I could check.
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
Great. Just let me know when I can pull it :) -- Bazsi
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
I'd love to get away a little bit from SMTP logic, and use option names that reflect more closely what the user sees in a MUA.
We might give a chance to specify all the rest of the details, but they should be the exception and not the rule.
smtp( host("localhost") port(25) from("syslog-ng alert service <noreply@example.com>") to("Admin #1 <admin@example.com>", "Admin #2 <admin2@example.com>") cc("Admin BOSS <admin.boss@example.com>") bcc("Blind CC <blindcc@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") extra-header("X-Program", "$PROGRAM") );
Noted, will adjust accordingly.
I doubt it'd be useful to use templates within To/Cc/Bcc, but the rest should be templates, with support for embedded newline characters. (headers can also contain embedded newlines, by embedding NL and adding whitespace in the front of the upcoming line, although I'm not sure how much MUAs like it).
That's a tricky one. I'd go the easy route and filter newlines, and replace them with spaces in the headers. I'm not quite sure whether libesmtp does it for us, I'll have to double check. The other option of mangling them into an SMTP-acceptable format doesn't seem to worth the complexity, unless it's in libesmtp already.
The same syntax could perhaps be used by a possible "sendmail()" destination, which could spawn the local sendmail binary, instead of directly using SMTP.
Rather do a variant of program() then, I'd say, which does a one-off fork & launch for each message it receives, as opposed to a long running process. That, and a few template functions to help with the formatting should do the trick. -- |8]
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
I'd love to get away a little bit from SMTP logic, and use option names that reflect more closely what the user sees in a MUA.
We might give a chance to specify all the rest of the details, but they should be the exception and not the rule.
smtp( host("localhost") port(25) from("syslog-ng alert service <noreply@example.com>") to("Admin #1 <admin@example.com>", "Admin #2 <admin2@example.com>") cc("Admin BOSS <admin.boss@example.com>") bcc("Blind CC <blindcc@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") extra-header("X-Program", "$PROGRAM") );
Noted, will adjust accordingly.
I doubt it'd be useful to use templates within To/Cc/Bcc, but the rest should be templates, with support for embedded newline characters. (headers can also contain embedded newlines, by embedding NL and adding whitespace in the front of the upcoming line, although I'm not sure how much MUAs like it).
That's a tricky one. I'd go the easy route and filter newlines, and replace them with spaces in the headers. I'm not quite sure whether libesmtp does it for us, I'll have to double check. The other option of mangling them into an SMTP-acceptable format doesn't seem to worth the complexity, unless it's in libesmpt al
The same syntax could perhaps be used by a possible "sendmail()" destination, which could spawn the local sendmail binary, instead of directly using SMTP.
Rather do a variant of program() then, I'd say, which does a one-off fork & launch for each message it receives, as opposed to a long running process. That, and a few template functions to help with the formatting should do the trick. -- |8]
(Quick note: I rebased my work/modules/afsmtp branch on top of the current master) Balazs Scheidler <bazsi@balabit.hu> writes:
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
I'd love to get away a little bit from SMTP logic, and use option names that reflect more closely what the user sees in a MUA.
Most of this has already been done on my branch, I just didn't send an update, I believe.
We might give a chance to specify all the rest of the details, but they should be the exception and not the rule.
smtp( host("localhost") port(25) from("syslog-ng alert service <noreply@example.com>") to("Admin #1 <admin@example.com>", "Admin #2 <admin2@example.com>") cc("Admin BOSS <admin.boss@example.com>") bcc("Blind CC <blindcc@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") extra-header("X-Program", "$PROGRAM") );
from(), to(), cc(), bcc(), reply_to() have the following syntax in my current tree: ,---- | from("admin@example.com") | OR | from("Admin #1" "admin@example.com") `---- Same goes for the other mentioned headers. The reason behind this, is that libesmtp expects things this way: the address and the name in two separate parameters. If specified with the first form, the e-mail address will be used as the name too. I could work around that, and do some parsing myself, but it's easier this way. I can extend the syntax to something like the following: ,---- | to("admin@example.com" ("Admin #2" "admin2@example.com")) `---- Although, I'd rather have multiple to(), cc() and bcc() lines than this.
I doubt it'd be useful to use templates within To/Cc/Bcc, but the rest should be templates, with support for embedded newline characters. (headers can also contain embedded newlines, by embedding NL and adding whitespace in the front of the upcoming line, although I'm not sure how much MUAs like it).
As of this writing, subject(), body() and header() (soon to be renamed to extra_header()) support templates, the rest don't. I haven't verified that embedded newlines work properly yet. I have a suspicion that they don't, and in that case I think I have a reasonably easy workaround (more on that below). I'll post an update once I know for sure.
The same syntax could perhaps be used by a possible "sendmail()" destination, which could spawn the local sendmail binary, instead of directly using SMTP.
I think I replied to this before - I'd rather extend program() to have an option that would make it always spawn a new instance of the program. Something like flags(always-spawn) or similar.
These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines).
This has partially been solved since: one can't set the mentioned headers with the headers() statement, and has to use from(), to(), etc. I do not have a solution for the date header yet (it can't be set in any way at the moment). Another problem is, that libesmtp parses the whole body, and tries to extract headers even if they appear after the body. Thus, having a body like the following will crash the module: ,---- | body("Yadayada\nDate: CRASH, BOOM, BANG!") `---- The workaround for this will be to replace "\n"s in the body with "\n ", to stop libesmtp from messing with the headers. Something similar will be done for the headers too, to prevent embedded newlines from injecting extra headers.
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
Great. Just let me know when I can pull it :)
I have both auth & starttls support locally, but... the solution is rather dumb, especially the tls one. They "Work For Me", but aren't merge ready yet, I believe. If it's possible, I'd like to request a two-stage merge: first the basic driver, without auth & tls, because that'll be ready within a reasonable time-frame, and the rest can be merged later, whenever I or anyone else gets around to do it properly. -- |8]
Hi, On Fri, 2011-03-04 at 15:16 +0100, Gergely Nagy wrote:
(Quick note: I rebased my work/modules/afsmtp branch on top of the current master)
I'll review it, thanks.
Balazs Scheidler <bazsi@balabit.hu> writes:
The usage is something like this:
destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply@example.com>") rcpt_to("Admin #1 <admin@example.com>") rcpt_to("Admin #2 <admin2@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); };
I'd love to get away a little bit from SMTP logic, and use option names that reflect more closely what the user sees in a MUA.
Most of this has already been done on my branch, I just didn't send an update, I believe.
We might give a chance to specify all the rest of the details, but they should be the exception and not the rule.
smtp( host("localhost") port(25) from("syslog-ng alert service <noreply@example.com>") to("Admin #1 <admin@example.com>", "Admin #2 <admin2@example.com>") cc("Admin BOSS <admin.boss@example.com>") bcc("Blind CC <blindcc@example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") extra-header("X-Program", "$PROGRAM") );
from(), to(), cc(), bcc(), reply_to() have the following syntax in my current tree:
,---- | from("admin@example.com") | OR | from("Admin #1" "admin@example.com") `----
Same goes for the other mentioned headers. The reason behind this, is that libesmtp expects things this way: the address and the name in two separate parameters. If specified with the first form, the e-mail address will be used as the name too.
I could work around that, and do some parsing myself, but it's easier this way.
I can extend the syntax to something like the following:
,---- | to("admin@example.com" ("Admin #2" "admin2@example.com")) `----
Although, I'd rather have multiple to(), cc() and bcc() lines than this.
I think it is ok that way.
I doubt it'd be useful to use templates within To/Cc/Bcc, but the rest should be templates, with support for embedded newline characters. (headers can also contain embedded newlines, by embedding NL and adding whitespace in the front of the upcoming line, although I'm not sure how much MUAs like it).
As of this writing, subject(), body() and header() (soon to be renamed to extra_header()) support templates, the rest don't.
I haven't verified that embedded newlines work properly yet. I have a suspicion that they don't, and in that case I think I have a reasonably easy workaround (more on that below). I'll post an update once I know for sure.
The same syntax could perhaps be used by a possible "sendmail()" destination, which could spawn the local sendmail binary, instead of directly using SMTP.
I think I replied to this before - I'd rather extend program() to have an option that would make it always spawn a new instance of the program. Something like flags(always-spawn) or similar.
That's ok, then sendmail() or better yet email() destination could become an SCL block, that uses a template() to format an email. The important stuff is to take care about proper escaping, so that users wouldn't have to care about that.
These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines).
This has partially been solved since: one can't set the mentioned headers with the headers() statement, and has to use from(), to(), etc.
I do not have a solution for the date header yet (it can't be set in any way at the moment).
Is there a point in setting that? I mean I would never want to specify the date header, or do you have a specific usecase?
Another problem is, that libesmtp parses the whole body, and tries to extract headers even if they appear after the body. Thus, having a body like the following will crash the module:
,---- | body("Yadayada\nDate: CRASH, BOOM, BANG!") `----
The workaround for this will be to replace "\n"s in the body with "\n ", to stop libesmtp from messing with the headers.
hmm... that doesn't sound too good? are you sure libesmtp is the correct library to use, then? two NLs terminate the message, that's such an ancient rule, that should really be working with an SMTP library, shouldn't it?
Something similar will be done for the headers too, to prevent embedded newlines from injecting extra headers.
Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet.
Great. Just let me know when I can pull it :)
I have both auth & starttls support locally, but... the solution is rather dumb, especially the tls one. They "Work For Me", but aren't merge ready yet, I believe.
If it's possible, I'd like to request a two-stage merge: first the basic driver, without auth & tls, because that'll be ready within a reasonable time-frame, and the rest can be merged later, whenever I or anyone else gets around to do it properly.
That's ok for me. I'll try to review the SMTP bits until then. -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
Hi,
On Fri, 2011-03-04 at 15:16 +0100, Gergely Nagy wrote:
(Quick note: I rebased my work/modules/afsmtp branch on top of the current master)
I'll review it, thanks.
I even pushed it now (along with the header()->extra_header() change).
The same syntax could perhaps be used by a possible "sendmail()" destination, which could spawn the local sendmail binary, instead of directly using SMTP.
I think I replied to this before - I'd rather extend program() to have an option that would make it always spawn a new instance of the program. Something like flags(always-spawn) or similar.
That's ok, then sendmail() or better yet email() destination could become an SCL block, that uses a template() to format an email. The important stuff is to take care about proper escaping, so that users wouldn't have to care about that.
Yeah, a few template functions here and there, and we're good to go. I'll happily prepare that once I'm done with the rest of my tasks :)
These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines).
This has partially been solved since: one can't set the mentioned headers with the headers() statement, and has to use from(), to(), etc.
I do not have a solution for the date header yet (it can't be set in any way at the moment).
Is there a point in setting that? I mean I would never want to specify the date header, or do you have a specific usecase?
Nope, no use case. I was thinking that people might want to set the message date to the date of the original receipt of the log message (by the time we get to send it, the clock might tick a few seconds!). On the other hand, solving the date problem (libesmtp has some annoying properties, date handling is one of them) seems like too much work for no gain, so Date will remain a forbidden header, unless someone presents a valid reason against that.
Another problem is, that libesmtp parses the whole body, and tries to extract headers even if they appear after the body. Thus, having a body like the following will crash the module:
,---- | body("Yadayada\nDate: CRASH, BOOM, BANG!") `----
The workaround for this will be to replace "\n"s in the body with "\n ", to stop libesmtp from messing with the headers.
hmm... that doesn't sound too good? are you sure libesmtp is the correct library to use, then?
It was lightweight! But the more I work with it, the less sure I am about this decision. I had another look at what's available, and libetpan (http://libetpan.sf.net/libetpan/) sounds like a decent alternative. It pulls in quite a bit of other dependencies, though. There's also libtinymail (http://tinymail.org/) which has a lot less dependencies, but a quick look through its API docs didn't seem promising.
two NLs terminate the message, that's such an ancient rule, that should really be working with an SMTP library, shouldn't it?
\n\n terminate the header, the message is terminated with \n.\n (which is guarded against by libesmtp). It's not as bad as it sounds. Let me try summing up how libesmtp works.. First of all, it stores the headers we explicitly set in a separate structure, which is fine. When setting header values, libesmtp itself does not guard against embedded newlines, but we can either replace newlines with a space, to clean up headers, or add a space after each newline: ,---- | Header: blah blah | blah blah | blah `---- This is valid MIME. Replacing newlines is easier, though. Anyway, when libesmtp gets around to process the body, it scans it for headers, in case some were set there too. Sadly, I couldn't find a way to stop it from doing so. Therefore, extra care will need to be taken to format the body correctly... Mngh. Now that I thought it through, I'll start to evaluate libetpan instead, even if it has more dependencies. -- |8]
Hi, I´m configuring user rights on a Balabit Storebox and would like to know if it is possible to give different users the right to analyse different destinations? Which means that user A can only look at the logs from destination A and user B can only look at the logs from destination B. Regards, René
Hi, By default, members of the 'search' group can view the stored messages online. You can specify different groups in the Access Control field of the logspace. HTH, Regards, Robert On Sunday, March 6, 2011 15:18 CET, Heinze, René <Rene.Heinze@BDR.de> wrote:
Hi,
I´m configuring user rights on a Balabit Storebox and would like to know if it is possible to give different users the right to analyse different destinations? Which means that user A can only look at the logs from destination A and user B can only look at the logs from destination B.
Regards,
René ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi, On Fri, 2011-03-04 at 16:53 +0100, Gergely Nagy wrote:
Balazs Scheidler <bazsi@balabit.hu> writes:
These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines).
This has partially been solved since: one can't set the mentioned headers with the headers() statement, and has to use from(), to(), etc.
I do not have a solution for the date header yet (it can't be set in any way at the moment).
Is there a point in setting that? I mean I would never want to specify the date header, or do you have a specific usecase?
Nope, no use case. I was thinking that people might want to set the message date to the date of the original receipt of the log message (by the time we get to send it, the clock might tick a few seconds!).
On the other hand, solving the date problem (libesmtp has some annoying properties, date handling is one of them) seems like too much work for no gain, so Date will remain a forbidden header, unless someone presents a valid reason against that.
I did review the libesmtp code this time, and as far as I can see, libesmtp tries to be smart by extracting headers from the message body, and it adds them only in case they are missing and would be required by SMTP.
Another problem is, that libesmtp parses the whole body, and tries to extract headers even if they appear after the body. Thus, having a body like the following will crash the module:
,---- | body("Yadayada\nDate: CRASH, BOOM, BANG!") `----
The workaround for this will be to replace "\n"s in the body with "\n ", to stop libesmtp from messing with the headers.
hmm... that doesn't sound too good? are you sure libesmtp is the correct library to use, then?
It was lightweight! But the more I work with it, the less sure I am about this decision.
I've checked this and the problem is that libesmtp is quite strict about that header/body separator. It must be "\r\n" and not just a mere "\n". I'm not sure why it crashes though.
I had another look at what's available, and libetpan (http://libetpan.sf.net/libetpan/) sounds like a decent alternative. It pulls in quite a bit of other dependencies, though.
I don't like that. It is complete with MIME envelopes and base64 encoding which seems a bit heavyweight.
There's also libtinymail (http://tinymail.org/) which has a lot less dependencies, but a quick look through its API docs didn't seem promising.
two NLs terminate the message, that's such an ancient rule, that should really be working with an SMTP library, shouldn't it?
\n\n terminate the header, the message is terminated with \n.\n (which is guarded against by libesmtp).
true, I wanted to write separate the header from the body.
It's not as bad as it sounds.
Let me try summing up how libesmtp works..
First of all, it stores the headers we explicitly set in a separate structure, which is fine. When setting header values, libesmtp itself does not guard against embedded newlines, but we can either replace newlines with a space, to clean up headers, or add a space after each newline:
,---- | Header: blah blah | blah blah | blah `----
This is valid MIME. Replacing newlines is easier, though.
Anyway, when libesmtp gets around to process the body, it scans it for headers, in case some were set there too. Sadly, I couldn't find a way to stop it from doing so. Therefore, extra care will need to be taken to format the body correctly...
using \r\n would work. Let me see if I can create a patch to fix thes problems. -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
Another problem is, that libesmtp parses the whole body, and tries to extract headers even if they appear after the body. Thus, having a body like the following will crash the module:
,---- | body("Yadayada\nDate: CRASH, BOOM, BANG!") `----
The workaround for this will be to replace "\n"s in the body with "\n ", to stop libesmtp from messing with the headers.
hmm... that doesn't sound too good? are you sure libesmtp is the correct library to use, then?
It was lightweight! But the more I work with it, the less sure I am about this decision.
I've checked this and the problem is that libesmtp is quite strict about that header/body separator. It must be "\r\n" and not just a mere "\n". I'm not sure why it crashes though.
It crashes, because it has a list of headers that it treats specially: they have special getter/setter functions, and the Date header is one of them, and it wants a time_t, yet, in the body, there's a string, and it does not attempt to parse it, it just treats the string pointer as if it was a time_t, and that leads to all kinds of nasty stuff, that in the end results in a crash. See the header_actions array in headers.c, and the set_date(), print_date() functions in the same file.
I had another look at what's available, and libetpan (http://libetpan.sf.net/libetpan/) sounds like a decent alternative. It pulls in quite a bit of other dependencies, though.
I don't like that. It is complete with MIME envelopes and base64 encoding which seems a bit heavyweight.
Yep, agreed. I'll be sticking with libesmtp, and either fixing its bugs and submitting them upstream, or working them around.
It's not as bad as it sounds.
Let me try summing up how libesmtp works..
First of all, it stores the headers we explicitly set in a separate structure, which is fine. When setting header values, libesmtp itself does not guard against embedded newlines, but we can either replace newlines with a space, to clean up headers, or add a space after each newline:
,---- | Header: blah blah | blah blah | blah `----
This is valid MIME. Replacing newlines is easier, though.
Anyway, when libesmtp gets around to process the body, it scans it for headers, in case some were set there too. Sadly, I couldn't find a way to stop it from doing so. Therefore, extra care will need to be taken to format the body correctly...
using \r\n would work. Let me see if I can create a patch to fix thes problems.
Hrm, that's a possibility, yeah, I didn't check \r\n, nor that part of the libesmtp code (yet). -- |8]
First of all, it stores the headers we explicitly set in a separate structure, which is fine. When setting header values, libesmtp itself does not guard against embedded newlines, but we can either replace newlines with a space, to clean up headers, or add a space after each newline:
,---- | Header: blah blah | blah blah | blah `----
This is valid MIME. Replacing newlines is easier, though.
Anyway, when libesmtp gets around to process the body, it scans it for headers, in case some were set there too. Sadly, I couldn't find a way to stop it from doing so. Therefore, extra care will need to be taken to format the body correctly...
using \r\n would work. Let me see if I can create a patch to fix thes problems.
Hrm, that's a possibility, yeah, I didn't check \r\n, nor that part of the libesmtp code (yet).
It appears that I fixed this issue by coincidence.. at least I can't reproduce the problem I saw before. An smtp destination like this: ,---- | destination d_smtp { | smtp( | host("127.0.0.1") | port(25) | from("syslog-ng alert service <algernon@luthien>") | to("algernon@luthien") | subject("[ALERT] $PROGRAM[$PID]") | body("Date: $S_DATA\nMsg: $MSG\n\nOh, hey there!\nWe received this message, and thought you might be interested.") | header("X-Program" "$PROGRAM") | header("X-Facility" "$FACILITY") | ); | }; `---- Did not produce extra headers, nor did it crash anymore. Furthermore, I fixed the header injection bug by replacing any \rs and \ns in the headers with a space. Also ported the driver to latest git head. It's available on my work/modules/afsmtp branch, and a consolidated patch will be available on the integration/afsmtp branch soon, too. This current code seems to be stable and reliable now, but a quick review probably wouldn't hurt. -- |8]
participants (8)
-
Alexander Clouter
-
Balazs Scheidler
-
Bill Anderson
-
Fekete Róbert
-
Gergely Nagy
-
Heinze, René
-
Robert Fekete
-
Valentijn Sessink