[syslog-ng] Macro expansion in program destination

[Evan Rempel]

Balazs Scheidler wrote:
> On Sat, 2009-12-05 at 16:52 -0800, Evan Rempel wrote:
>>> When Syslog-NG starts up, it executes all of the program() statements
>>> with something like an execve system call (I'll defer to the Balabit
>>> guys for specifics).
>> I understand that this is the way that syslog-ng works today, but I am
>> discussing the why and what if about macro expansion in program
>> destinations. Is there any reason to prevent someone from using
>> macro expansions in program destinations?
>>
>>>  In doing so, it will pass the program()
>>> configuration to the system call.  This happens just once at startup.
>>> This forked program will have its own PID and in your case would look
>>> like "program -options >> /tmp/log.30" or something similar if you
>>> looked at a process list using ps aux.  The key is that the "30" (or
>>> whatever the day was) is sent at program startup and becomes part of
>>> the program.
>> No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value
>> for $S_DAY do the command line, and thus the 30 does *not* appear at all.
>>
>>> You want that macro to be resolved on a per-message
>>> basis, which would mean that the operating system would have to go
>>> back in time to when the program first started and change the
>>> arguments it was given.  I think time travel is planned in the 2.10
>>> Linux kernel, but that may have been space travel--I can't remember
>>> now.
>> If what you are saying is true, then the same issue exists for file names.
>> What actually happens is that syslog-ng *evaluates* the destination template
>> for each message, and if 
>>
>> 1. The template is not open, opens it and writes the message to it.
>> 2. if it is already open, write the message to it.
>>
>> Additionally, if the destination does not receive a message for a certain time period, then
>> the destination is closed.
>>
>> All of this works fine for file destinations, but is not performed at all for program destinations.
>>
>> I can not see any reason why this process would not work equally well for program destinations.
>> I thought that I must have been missing something obvious in my evaluation of how things
>> could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
>>
>>
>>> Happily, the power of Syslog-NG can save you quite easily here and
>>> perform the "general function" you're looking for.  The key is that
>>> Syslog-NG lets you format the messages being sent to your program()
>>> any way you want to using templates, so just tack on the $S_DAY macro
>>> to the log format you're using like this:
>>> template t_msg_with_day { template("$S_DAY|$MSG"); };
>>> destination mydest{ program("/some/path/to/filter/program -options",
>>> template(t_msg_with_day) ) );};
>>>
>>> Then open your file in your program using the $S_DAY argument.
>> You could do it this way, but since syslog-ng has already parsed the
>> syslog line, and already figured out what the value for $S_DAY is,
>> it is more CPU efficient for syslog-ng to perform the determination of
>> when a new instance of the program should be started. All of this code
>> is already in syslog-ng, but it is limited to files, and excluded from program
>> destinations.
>>
>>
>>
>>> My rule of thumb for where to put Syslog-NG in the log delivery chain
>>> is to use it solely as a receiver/parser/forwarder and to leave all of
>>> the actual disk writing and logic to a program written in a language
>>> more suited for decision making.
>> You have identified exactly what I am talking about. Syslog-ng has done the receiving,
>> and the parsing, and then needs to forward the message to
>> an appropriate program destination based on the parsed information.
>>
>>>  If you don't need any advanced
>>> decision-making done with your logs, then using the great feature of
>>> macro file destinations like
>>>
>>> destination d_file { file("/tmp/log.$S_DAY"); };
>>>
>>> or even
>>>
>>> destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
>>>
>>> is really handy.
>> And it would be even more handy if it could be used on program destinations :-)
>>
>>
>> On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>> I understand that the program destination should only be opened once
>>> and the messages sent to it but why does that mean the macro expansion
>>> should not be permitted?
>>>
>>> The only work around right now is for the program to parse each
>>> message to extract the parts that are needed to manage the log
>>> rotation. Syslog-ng already does this AND syslog-ng already has the
>>> necessary logic to open a different destination based on the macro
>>> expantion.
>>>
>>> General purpose tools need to perform general function and do them in
>>> the most intelegent manner. Disabling functionality in specefic
>>> situations in the name of protecting me from my own stupidity results
>>> in situations wher the tool could solve the problem but does not
>>> because of an arbitrary division of the author.
>>>
>>> Unless there is an more important consideration such as security then
>>> I think the disallow of macro expantion in program destinations should
>>> be permitted.
>>>
>>> Evan Rempel
>>> University Systems
>>>
>>> On 2009-12-04, at 14:20, "chris packham" <chris.packham at alliedtelesis.co.nz
>>>  > wrote:
>>>
>>>> This is intentional behaviour.
>>>>
>>>> Syslog-ng starts any "program()" when syslog-ng is started and it
>>>> expects that the program will perform read from stdin and block
>>>> waiting for data. This protects syslog-ng againts the overhead of
>>>> forking a new process everytime a matching log message is seen.
>>>>
>>>> You can still use filters so I'd suggest modifying your program so
>>>> that it can have the $S_DAY fed to it at the start of each log
>>>> message.
>>>>
>>>>
>>>>>>> Evan Rempel <erempel at uvic.ca> 12/04/09 4:07 PM >>>
>>>> Is it possible to use macro expansion in the program destination.
>>>> I wanted to write something like
>>>>
>>>> destination mydest{ program("/some/path/to/filter/program -options
>>>>>> /path/log.$S_DAY"));};
>>>> I was hoping that syslog-ng would not open this destination until
>>>> the first message
>>>> was ready for this destination.
>>>>
>>>> I was hoping that syslog-ng would open a new one when the $S_DAY
>>>> changed, allowing
>>>> me to have each day of logs processed by my program with outputs
>>>> based on day.
>>>>
>>>> Does anyone know if macro expantion can be used on program
>>>> destinations?
>>>>
>>>> If not, can someone explain what I am overlooking that makes this a
>>>> bad thing?
>>>>
> 
> This could be implemented, however the current funcionality is
> completely related to file destinations, it is not generic in any way.
> It could be generalized though. I'd add this idea to our roadmap, but
> could you please describe your exact use case?

We have a tool that does filtering based on a hierarchical configuration of
regular expressions. The tool can also do actions based on matches etc.
We use this tool for processing virtually all of our log files (apache, tomcat,
ldap, custom apps and syslog). This tool is a filter in the true definition, so
it only works on one input stream, and produces one output stream.

I wanted to use syslog-ng's ability to "automatically" do log file rotation. OK,
it's not really rotation, but it works, and IMHO is even better than rotation.
I wanted to send a syslog-ng macro value to the application at startup
and then when the macro value changes, start another instance of the application.

destination { program("/my/filter/application $S_YEAR $S_MONTH $S_DAY > /some/output/file"); };
or
destination { program("/my/filter/application > /some/output/file$S_YEAR$S_MONTH$S_DAY"); };

We have a second tool that performs statistical reporting on log streams.
I wanted to have this tool produce statistics for each day, so need to feed it
a single day of logs.

I wanted to use syslog-ng's ability to invoke a new script each day;

destination { program("/my/statistics/application $S_YEAR $S_MONTH $S_DAY"); };

At the end of the day, the statistical data can be placed into a database or e-mailed to
some place. They key is how to get syslog-ng to start another instance of the program for each
day.


There are lots of workarounds, but just like all workarounds they are complicated and
do not provide the exact same functionality. I am not going to list the workarounds because
they make this discussion more complicated.


An different issue is that if I configure a program destination using a macro variable,
the variable is expanded by the SHELL, not by syslog-ng. This is not obvious at all, and
can lead to very odd, and possibly security related issues.


-- 
Evan Rempel