[syslog-ng] Macro expansion in program destination

[Balazs Scheidler]

On Sat, 2009-12-05 at 16:52 -0800, Evan Rempel wrote:
> > When Syslog-NG starts up, it executes all of the program() statements
> > with something like an execve system call (I'll defer to the Balabit
> > guys for specifics).
> 
> I understand that this is the way that syslog-ng works today, but I am
> discussing the why and what if about macro expansion in program
> destinations. Is there any reason to prevent someone from using
> macro expansions in program destinations?
> 
> >  In doing so, it will pass the program()
> > configuration to the system call.  This happens just once at startup.
> > This forked program will have its own PID and in your case would look
> > like "program -options >> /tmp/log.30" or something similar if you
> > looked at a process list using ps aux.  The key is that the "30" (or
> > whatever the day was) is sent at program startup and becomes part of
> > the program.
> 
> No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value
> for $S_DAY do the command line, and thus the 30 does *not* appear at all.
> 
> > You want that macro to be resolved on a per-message
> > basis, which would mean that the operating system would have to go
> > back in time to when the program first started and change the
> > arguments it was given.  I think time travel is planned in the 2.10
> > Linux kernel, but that may have been space travel--I can't remember
> > now.
> 
> If what you are saying is true, then the same issue exists for file names.
> What actually happens is that syslog-ng *evaluates* the destination template
> for each message, and if 
> 
> 1. The template is not open, opens it and writes the message to it.
> 2. if it is already open, write the message to it.
> 
> Additionally, if the destination does not receive a message for a certain time period, then
> the destination is closed.
> 
> All of this works fine for file destinations, but is not performed at all for program destinations.
> 
> I can not see any reason why this process would not work equally well for program destinations.
> I thought that I must have been missing something obvious in my evaluation of how things
> could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
> 
> 
> > Happily, the power of Syslog-NG can save you quite easily here and
> > perform the "general function" you're looking for.  The key is that
> > Syslog-NG lets you format the messages being sent to your program()
> > any way you want to using templates, so just tack on the $S_DAY macro
> > to the log format you're using like this:
> 
> > template t_msg_with_day { template("$S_DAY|$MSG"); };
> > destination mydest{ program("/some/path/to/filter/program -options",
> > template(t_msg_with_day) ) );};
> >
> > Then open your file in your program using the $S_DAY argument.
> 
> You could do it this way, but since syslog-ng has already parsed the
> syslog line, and already figured out what the value for $S_DAY is,
> it is more CPU efficient for syslog-ng to perform the determination of
> when a new instance of the program should be started. All of this code
> is already in syslog-ng, but it is limited to files, and excluded from program
> destinations.
> 
> 
> 
> > My rule of thumb for where to put Syslog-NG in the log delivery chain
> > is to use it solely as a receiver/parser/forwarder and to leave all of
> > the actual disk writing and logic to a program written in a language
> > more suited for decision making.
> 
> You have identified exactly what I am talking about. Syslog-ng has done the receiving,
> and the parsing, and then needs to forward the message to
> an appropriate program destination based on the parsed information.
> 
> >  If you don't need any advanced
> > decision-making done with your logs, then using the great feature of
> > macro file destinations like
> >
> > destination d_file { file("/tmp/log.$S_DAY"); };
> >
> > or even
> >
> > destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
> >
> > is really handy.
> 
> And it would be even more handy if it could be used on program destinations :-)
> 
> 
> On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel at uvic.ca> wrote:
> > I understand that the program destination should only be opened once
> > and the messages sent to it but why does that mean the macro expansion
> > should not be permitted?
> >
> > The only work around right now is for the program to parse each
> > message to extract the parts that are needed to manage the log
> > rotation. Syslog-ng already does this AND syslog-ng already has the
> > necessary logic to open a different destination based on the macro
> > expantion.
> >
> > General purpose tools need to perform general function and do them in
> > the most intelegent manner. Disabling functionality in specefic
> > situations in the name of protecting me from my own stupidity results
> > in situations wher the tool could solve the problem but does not
> > because of an arbitrary division of the author.
> >
> > Unless there is an more important consideration such as security then
> > I think the disallow of macro expantion in program destinations should
> > be permitted.
> >
> > Evan Rempel
> > University Systems
> >
> > On 2009-12-04, at 14:20, "chris packham" <chris.packham at alliedtelesis.co.nz
> >  > wrote:
> >
> >> This is intentional behaviour.
> >>
> >> Syslog-ng starts any "program()" when syslog-ng is started and it
> >> expects that the program will perform read from stdin and block
> >> waiting for data. This protects syslog-ng againts the overhead of
> >> forking a new process everytime a matching log message is seen.
> >>
> >> You can still use filters so I'd suggest modifying your program so
> >> that it can have the $S_DAY fed to it at the start of each log
> >> message.
> >>
> >>
> >>>>> Evan Rempel <erempel at uvic.ca> 12/04/09 4:07 PM >>>
> >>
> >> Is it possible to use macro expansion in the program destination.
> >> I wanted to write something like
> >>
> >> destination mydest{ program("/some/path/to/filter/program -options
> >> >> /path/log.$S_DAY"));};
> >>
> >> I was hoping that syslog-ng would not open this destination until
> >> the first message
> >> was ready for this destination.
> >>
> >> I was hoping that syslog-ng would open a new one when the $S_DAY
> >> changed, allowing
> >> me to have each day of logs processed by my program with outputs
> >> based on day.
> >>
> >> Does anyone know if macro expantion can be used on program
> >> destinations?
> >>
> >> If not, can someone explain what I am overlooking that makes this a
> >> bad thing?
> >>

This could be implemented, however the current funcionality is
completely related to file destinations, it is not generic in any way.
It could be generalized though. I'd add this idea to our roadmap, but
could you please describe your exact use case?

Thanks.


-- 
Bazsi