[syslog-ng] Macro expansion in program destination

[Balazs Scheidler]

On Mon, 2009-12-07 at 11:20 -0800, Evan Rempel wrote:
> Balazs Scheidler wrote:
> > On Sat, 2009-12-05 at 16:52 -0800, Evan Rempel wrote:
> >>> When Syslog-NG starts up, it executes all of the program() statements
> >>> with something like an execve system call (I'll defer to the Balabit
> >>> guys for specifics).
> >> I understand that this is the way that syslog-ng works today, but I am
> >> discussing the why and what if about macro expansion in program
> >> destinations. Is there any reason to prevent someone from using
> >> macro expansions in program destinations?
> >>
> >>>  In doing so, it will pass the program()
> >>> configuration to the system call.  This happens just once at startup.
> >>> This forked program will have its own PID and in your case would look
> >>> like "program -options >> /tmp/log.30" or something similar if you
> >>> looked at a process list using ps aux.  The key is that the "30" (or
> >>> whatever the day was) is sent at program startup and becomes part of
> >>> the program.
> >> No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value
> >> for $S_DAY do the command line, and thus the 30 does *not* appear at all.
> >>
> >>> You want that macro to be resolved on a per-message
> >>> basis, which would mean that the operating system would have to go
> >>> back in time to when the program first started and change the
> >>> arguments it was given.  I think time travel is planned in the 2.10
> >>> Linux kernel, but that may have been space travel--I can't remember
> >>> now.
> >> If what you are saying is true, then the same issue exists for file names.
> >> What actually happens is that syslog-ng *evaluates* the destination template
> >> for each message, and if 
> >>
> >> 1. The template is not open, opens it and writes the message to it.
> >> 2. if it is already open, write the message to it.
> >>
> >> Additionally, if the destination does not receive a message for a certain time period, then
> >> the destination is closed.
> >>
> >> All of this works fine for file destinations, but is not performed at all for program destinations.
> >>
> >> I can not see any reason why this process would not work equally well for program destinations.
> >> I thought that I must have been missing something obvious in my evaluation of how things
> >> could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
> >>
> >>
> >>> Happily, the power of Syslog-NG can save you quite easily here and
> >>> perform the "general function" you're looking for.  The key is that
> >>> Syslog-NG lets you format the messages being sent to your program()
> >>> any way you want to using templates, so just tack on the $S_DAY macro
> >>> to the log format you're using like this:
> >>> template t_msg_with_day { template("$S_DAY|$MSG"); };
> >>> destination mydest{ program("/some/path/to/filter/program -options",
> >>> template(t_msg_with_day) ) );};
> >>>
> >>> Then open your file in your program using the $S_DAY argument.
> >> You could do it this way, but since syslog-ng has already parsed the
> >> syslog line, and already figured out what the value for $S_DAY is,
> >> it is more CPU efficient for syslog-ng to perform the determination of
> >> when a new instance of the program should be started. All of this code
> >> is already in syslog-ng, but it is limited to files, and excluded from program
> >> destinations.
> >>
> >>
> >>
> >>> My rule of thumb for where to put Syslog-NG in the log delivery chain
> >>> is to use it solely as a receiver/parser/forwarder and to leave all of
> >>> the actual disk writing and logic to a program written in a language
> >>> more suited for decision making.
> >> You have identified exactly what I am talking about. Syslog-ng has done the receiving,
> >> and the parsing, and then needs to forward the message to
> >> an appropriate program destination based on the parsed information.
> >>
> >>>  If you don't need any advanced
> >>> decision-making done with your logs, then using the great feature of
> >>> macro file destinations like
> >>>
> >>> destination d_file { file("/tmp/log.$S_DAY"); };
> >>>
> >>> or even
> >>>
> >>> destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
> >>>
> >>> is really handy.
> >> And it would be even more handy if it could be used on program destinations :-)
> >>
> >>
> >> On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel at uvic.ca> wrote:
> >>> I understand that the program destination should only be opened once
> >>> and the messages sent to it but why does that mean the macro expansion
> >>> should not be permitted?
> >>>
> >>> The only work around right now is for the program to parse each
> >>> message to extract the parts that are needed to manage the log
> >>> rotation. Syslog-ng already does this AND syslog-ng already has the
> >>> necessary logic to open a different destination based on the macro
> >>> expantion.
> >>>
> >>> General purpose tools need to perform general function and do them in
> >>> the most intelegent manner. Disabling functionality in specefic
> >>> situations in the name of protecting me from my own stupidity results
> >>> in situations wher the tool could solve the problem but does not
> >>> because of an arbitrary division of the author.
> >>>
> >>> Unless there is an more important consideration such as security then
> >>> I think the disallow of macro expantion in program destinations should
> >>> be permitted.
> >>>
> >>> Evan Rempel
> >>> University Systems
> >>>
> >>> On 2009-12-04, at 14:20, "chris packham" <chris.packham at alliedtelesis.co.nz
> >>>  > wrote:
> >>>
> >>>> This is intentional behaviour.
> >>>>
> >>>> Syslog-ng starts any "program()" when syslog-ng is started and it
> >>>> expects that the program will perform read from stdin and block
> >>>> waiting for data. This protects syslog-ng againts the overhead of
> >>>> forking a new process everytime a matching log message is seen.
> >>>>
> >>>> You can still use filters so I'd suggest modifying your program so
> >>>> that it can have the $S_DAY fed to it at the start of each log
> >>>> message.
> >>>>
> >>>>
> >>>>>>> Evan Rempel <erempel at uvic.ca> 12/04/09 4:07 PM >>>
> >>>> Is it possible to use macro expansion in the program destination.
> >>>> I wanted to write something like
> >>>>
> >>>> destination mydest{ program("/some/path/to/filter/program -options
> >>>>>> /path/log.$S_DAY"));};
> >>>> I was hoping that syslog-ng would not open this destination until
> >>>> the first message
> >>>> was ready for this destination.
> >>>>
> >>>> I was hoping that syslog-ng would open a new one when the $S_DAY
> >>>> changed, allowing
> >>>> me to have each day of logs processed by my program with outputs
> >>>> based on day.
> >>>>
> >>>> Does anyone know if macro expantion can be used on program
> >>>> destinations?
> >>>>
> >>>> If not, can someone explain what I am overlooking that makes this a
> >>>> bad thing?
> >>>>
> > 
> > This could be implemented, however the current funcionality is
> > completely related to file destinations, it is not generic in any way.
> > It could be generalized though. I'd add this idea to our roadmap, but
> > could you please describe your exact use case?
> 
> We have a tool that does filtering based on a hierarchical configuration of
> regular expressions. The tool can also do actions based on matches etc.
> We use this tool for processing virtually all of our log files (apache, tomcat,
> ldap, custom apps and syslog). This tool is a filter in the true definition, so
> it only works on one input stream, and produces one output stream.
> 
> I wanted to use syslog-ng's ability to "automatically" do log file rotation. OK,
> it's not really rotation, but it works, and IMHO is even better than rotation.
> I wanted to send a syslog-ng macro value to the application at startup
> and then when the macro value changes, start another instance of the application.
> 
> destination { program("/my/filter/application $S_YEAR $S_MONTH $S_DAY > /some/output/file"); };
> or
> destination { program("/my/filter/application > /some/output/file$S_YEAR$S_MONTH$S_DAY"); };
> 
> We have a second tool that performs statistical reporting on log streams.
> I wanted to have this tool produce statistics for each day, so need to feed it
> a single day of logs.
> 
> I wanted to use syslog-ng's ability to invoke a new script each day;
> 
> destination { program("/my/statistics/application $S_YEAR $S_MONTH $S_DAY"); };
> 
> At the end of the day, the statistical data can be placed into a database or e-mailed to
> some place. They key is how to get syslog-ng to start another instance of the program for each
> day.
> 
> 
> There are lots of workarounds, but just like all workarounds they are complicated and
> do not provide the exact same functionality. I am not going to list the workarounds because
> they make this discussion more complicated.
> 
> 
> An different issue is that if I configure a program destination using a macro variable,
> the variable is expanded by the SHELL, not by syslog-ng. This is not obvious at all, and
> can lead to very odd, and possibly security related issues.

Thanks for the explanation. I've added this item to our internal roadmap
(which is a SCRUM backlog these days), not yet scheduled to a release
though, however I agree that such a feature should be useful in a number
of cases.


-- 
Bazsi