[syslog-ng] Macro expansion in program destination

[Martin Holste]

Evan,

I'll try to elaborate on what Chris has said:

When Syslog-NG starts up, it executes all of the program() statements
with something like an execve system call (I'll defer to the Balabit
guys for specifics).  In doing so, it will pass the program()
configuration to the system call.  This happens just once at startup.
This forked program will have its own PID and in your case would look
like "program -options >> /tmp/log.30" or something similar if you
looked at a process list using ps aux.  The key is that the "30" (or
whatever the day was) is sent at program startup and becomes part of
the program.  You want that macro to be resolved on a per-message
basis, which would mean that the operating system would have to go
back in time to when the program first started and change the
arguments it was given.  I think time travel is planned in the 2.10
Linux kernel, but that may have been space travel--I can't remember
now.

Happily, the power of Syslog-NG can save you quite easily here and
perform the "general function" you're looking for.  The key is that
Syslog-NG lets you format the messages being sent to your program()
any way you want to using templates, so just tack on the $S_DAY macro
to the log format you're using like this:

template t_msg_with_day { template("$S_DAY|$MSG"); };
destination mydest{ program("/some/path/to/filter/program -options",
template(t_msg_with_day) ) );};

Then open your file in your program using the $S_DAY argument.  In
Perl, I'd do something like this:
$| = 1;
my $current_day;
my $fh;
while (<>){
  my ($day, $msg) = split(/\|/, $_);
  unless ($current_day eq $day){
    $current_day = $day;
    close($fh) if $fh;
    open($fh, ">> /path/log.$day") or die($!);
  }
  print $fh $msg;
}

My rule of thumb for where to put Syslog-NG in the log delivery chain
is to use it solely as a receiver/parser/forwarder and to leave all of
the actual disk writing and logic to a program written in a language
more suited for decision making.  If you don't need any advanced
decision-making done with your logs, then using the great feature of
macro file destinations like

destination d_file { file("/tmp/log.$S_DAY"); };

or even

destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };

is really handy.


On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel at uvic.ca> wrote:
> I understand that the program destination should only be opened once
> and the messages sent to it but why does that mean the macro expansion
> should not be permitted?
>
> The only work around right now is for the program to parse each
> message to extract the parts that are needed to manage the log
> rotation. Syslog-ng already does this AND syslog-ng already has the
> necessary logic to open a different destination based on the macro
> expantion.
>
> General purpose tools need to perform general function and do them in
> the most intelegent manner. Disabling functionality in specefic
> situations in the name of protecting me from my own stupidity results
> in situations wher the tool could solve the problem but does not
> because of an arbitrary division of the author.
>
> Unless there is an more important consideration such as security then
> I think the disallow of macro expantion in program destinations should
> be permitted.
>
> Evan Rempel
> University Systems
>
> On 2009-12-04, at 14:20, "chris packham" <chris.packham at alliedtelesis.co.nz
>  > wrote:
>
>> This is intentional behaviour.
>>
>> Syslog-ng starts any "program()" when syslog-ng is started and it
>> expects that the program will perform read from stdin and block
>> waiting for data. This protects syslog-ng againts the overhead of
>> forking a new process everytime a matching log message is seen.
>>
>> You can still use filters so I'd suggest modifying your program so
>> that it can have the $S_DAY fed to it at the start of each log
>> message.
>>
>>
>>>>> Evan Rempel <erempel at uvic.ca> 12/04/09 4:07 PM >>>
>>
>> Is it possible to use macro expansion in the program destination.
>> I wanted to write something like
>>
>> destination mydest{ program("/some/path/to/filter/program -options
>> >> /path/log.$S_DAY"));};
>>
>> I was hoping that syslog-ng would not open this destination until
>> the first message
>> was ready for this destination.
>>
>> I was hoping that syslog-ng would open a new one when the $S_DAY
>> changed, allowing
>> me to have each day of logs processed by my program with outputs
>> based on day.
>>
>> Does anyone know if macro expantion can be used on program
>> destinations?
>>
>> If not, can someone explain what I am overlooking that makes this a
>> bad thing?
>>
>> Thanks,
>>
>> --
>> Evan Rempel
>> ______________________________________________________________________________
>
>
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>> ______________________________________________________________________________
>
>
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>