Evan, I'll try to elaborate on what Chris has said: When Syslog-NG starts up, it executes all of the program() statements with something like an execve system call (I'll defer to the Balabit guys for specifics). In doing so, it will pass the program() configuration to the system call. This happens just once at startup. This forked program will have its own PID and in your case would look like "program -options >> /tmp/log.30" or something similar if you looked at a process list using ps aux. The key is that the "30" (or whatever the day was) is sent at program startup and becomes part of the program. You want that macro to be resolved on a per-message basis, which would mean that the operating system would have to go back in time to when the program first started and change the arguments it was given. I think time travel is planned in the 2.10 Linux kernel, but that may have been space travel--I can't remember now. Happily, the power of Syslog-NG can save you quite easily here and perform the "general function" you're looking for. The key is that Syslog-NG lets you format the messages being sent to your program() any way you want to using templates, so just tack on the $S_DAY macro to the log format you're using like this: template t_msg_with_day { template("$S_DAY|$MSG"); }; destination mydest{ program("/some/path/to/filter/program -options", template(t_msg_with_day) ) );}; Then open your file in your program using the $S_DAY argument. In Perl, I'd do something like this: $| = 1; my $current_day; my $fh; while (<>){ my ($day, $msg) = split(/\|/, $_); unless ($current_day eq $day){ $current_day = $day; close($fh) if $fh; open($fh, ">> /path/log.$day") or die($!); } print $fh $msg; } My rule of thumb for where to put Syslog-NG in the log delivery chain is to use it solely as a receiver/parser/forwarder and to leave all of the actual disk writing and logic to a program written in a language more suited for decision making. If you don't need any advanced decision-making done with your logs, then using the great feature of macro file destinations like destination d_file { file("/tmp/log.$S_DAY"); }; or even destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); }; is really handy. On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted?
The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion.
General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author.
Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted.
Evan Rempel University Systems
On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz > wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>>
Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
/path/log.$S_DAY"));};
I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
Thanks,
-- Evan Rempel ______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html