Re: [syslog-ng] Macro expansion in program destination
This is intentional behaviour. Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen. You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>>
Is it possible to use macro expansion in the program destination. I wanted to write something like destination mydest{ program("/some/path/to/filter/program -options >> /path/log.$S_DAY"));}; I was hoping that syslog-ng would not open this destination until the first message was ready for this destination. I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day. Does anyone know if macro expantion can be used on program destinations? If not, can someone explain what I am overlooking that makes this a bad thing? Thanks, -- Evan Rempel ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted? The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion. General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author. Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted. Evan Rempel University Systems On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz
wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>>
Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
/path/log.$S_DAY"));};
I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
Thanks,
-- Evan Rempel ______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Evan, I'll try to elaborate on what Chris has said: When Syslog-NG starts up, it executes all of the program() statements with something like an execve system call (I'll defer to the Balabit guys for specifics). In doing so, it will pass the program() configuration to the system call. This happens just once at startup. This forked program will have its own PID and in your case would look like "program -options >> /tmp/log.30" or something similar if you looked at a process list using ps aux. The key is that the "30" (or whatever the day was) is sent at program startup and becomes part of the program. You want that macro to be resolved on a per-message basis, which would mean that the operating system would have to go back in time to when the program first started and change the arguments it was given. I think time travel is planned in the 2.10 Linux kernel, but that may have been space travel--I can't remember now. Happily, the power of Syslog-NG can save you quite easily here and perform the "general function" you're looking for. The key is that Syslog-NG lets you format the messages being sent to your program() any way you want to using templates, so just tack on the $S_DAY macro to the log format you're using like this: template t_msg_with_day { template("$S_DAY|$MSG"); }; destination mydest{ program("/some/path/to/filter/program -options", template(t_msg_with_day) ) );}; Then open your file in your program using the $S_DAY argument. In Perl, I'd do something like this: $| = 1; my $current_day; my $fh; while (<>){ my ($day, $msg) = split(/\|/, $_); unless ($current_day eq $day){ $current_day = $day; close($fh) if $fh; open($fh, ">> /path/log.$day") or die($!); } print $fh $msg; } My rule of thumb for where to put Syslog-NG in the log delivery chain is to use it solely as a receiver/parser/forwarder and to leave all of the actual disk writing and logic to a program written in a language more suited for decision making. If you don't need any advanced decision-making done with your logs, then using the great feature of macro file destinations like destination d_file { file("/tmp/log.$S_DAY"); }; or even destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); }; is really handy. On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted?
The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion.
General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author.
Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted.
Evan Rempel University Systems
On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz > wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>>
Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
/path/log.$S_DAY"));};
I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
Thanks,
-- Evan Rempel ______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
When Syslog-NG starts up, it executes all of the program() statements with something like an execve system call (I'll defer to the Balabit guys for specifics).
I understand that this is the way that syslog-ng works today, but I am discussing the why and what if about macro expansion in program destinations. Is there any reason to prevent someone from using macro expansions in program destinations?
In doing so, it will pass the program() configuration to the system call. This happens just once at startup. This forked program will have its own PID and in your case would look like "program -options >> /tmp/log.30" or something similar if you looked at a process list using ps aux. The key is that the "30" (or whatever the day was) is sent at program startup and becomes part of the program.
No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value for $S_DAY do the command line, and thus the 30 does *not* appear at all.
You want that macro to be resolved on a per-message basis, which would mean that the operating system would have to go back in time to when the program first started and change the arguments it was given. I think time travel is planned in the 2.10 Linux kernel, but that may have been space travel--I can't remember now.
If what you are saying is true, then the same issue exists for file names. What actually happens is that syslog-ng *evaluates* the destination template for each message, and if 1. The template is not open, opens it and writes the message to it. 2. if it is already open, write the message to it. Additionally, if the destination does not receive a message for a certain time period, then the destination is closed. All of this works fine for file destinations, but is not performed at all for program destinations. I can not see any reason why this process would not work equally well for program destinations. I thought that I must have been missing something obvious in my evaluation of how things could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
Happily, the power of Syslog-NG can save you quite easily here and perform the "general function" you're looking for. The key is that Syslog-NG lets you format the messages being sent to your program() any way you want to using templates, so just tack on the $S_DAY macro to the log format you're using like this:
template t_msg_with_day { template("$S_DAY|$MSG"); }; destination mydest{ program("/some/path/to/filter/program -options", template(t_msg_with_day) ) );};
Then open your file in your program using the $S_DAY argument.
You could do it this way, but since syslog-ng has already parsed the syslog line, and already figured out what the value for $S_DAY is, it is more CPU efficient for syslog-ng to perform the determination of when a new instance of the program should be started. All of this code is already in syslog-ng, but it is limited to files, and excluded from program destinations.
My rule of thumb for where to put Syslog-NG in the log delivery chain is to use it solely as a receiver/parser/forwarder and to leave all of the actual disk writing and logic to a program written in a language more suited for decision making.
You have identified exactly what I am talking about. Syslog-ng has done the receiving, and the parsing, and then needs to forward the message to an appropriate program destination based on the parsed information.
If you don't need any advanced decision-making done with your logs, then using the great feature of macro file destinations like
destination d_file { file("/tmp/log.$S_DAY"); };
or even
destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
is really handy.
And it would be even more handy if it could be used on program destinations :-) On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted?
The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion.
General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author.
Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted.
Evan Rempel University Systems
On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz
wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>>
Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
/path/log.$S_DAY"));};
I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
Thanks,
-- Evan Rempel
No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value for $S_DAY do the command line, and thus the 30 does *not* appear at all.
Yes, you are right--I thought it would but it doesn't. I still don't understand why you would need to do that. You want to send a log to a given program depending on the value of $S_DAY?
You have identified exactly what I am talking about. Syslog-ng has done the receiving, and the parsing, and then needs to forward the message to an appropriate program destination based on the parsed information.
Then use a filter(). If your setup is so specific that filter() isn't enough, then use something like Perl to route those messages. That's advanced decision making and should be done in a program. I'd rather see the Balabit guys focusing their efforts on other features.
On Sat, 2009-12-05 at 16:52 -0800, Evan Rempel wrote:
When Syslog-NG starts up, it executes all of the program() statements with something like an execve system call (I'll defer to the Balabit guys for specifics).
I understand that this is the way that syslog-ng works today, but I am discussing the why and what if about macro expansion in program destinations. Is there any reason to prevent someone from using macro expansions in program destinations?
In doing so, it will pass the program() configuration to the system call. This happens just once at startup. This forked program will have its own PID and in your case would look like "program -options >> /tmp/log.30" or something similar if you looked at a process list using ps aux. The key is that the "30" (or whatever the day was) is sent at program startup and becomes part of the program.
No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value for $S_DAY do the command line, and thus the 30 does *not* appear at all.
You want that macro to be resolved on a per-message basis, which would mean that the operating system would have to go back in time to when the program first started and change the arguments it was given. I think time travel is planned in the 2.10 Linux kernel, but that may have been space travel--I can't remember now.
If what you are saying is true, then the same issue exists for file names. What actually happens is that syslog-ng *evaluates* the destination template for each message, and if
1. The template is not open, opens it and writes the message to it. 2. if it is already open, write the message to it.
Additionally, if the destination does not receive a message for a certain time period, then the destination is closed.
All of this works fine for file destinations, but is not performed at all for program destinations.
I can not see any reason why this process would not work equally well for program destinations. I thought that I must have been missing something obvious in my evaluation of how things could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
Happily, the power of Syslog-NG can save you quite easily here and perform the "general function" you're looking for. The key is that Syslog-NG lets you format the messages being sent to your program() any way you want to using templates, so just tack on the $S_DAY macro to the log format you're using like this:
template t_msg_with_day { template("$S_DAY|$MSG"); }; destination mydest{ program("/some/path/to/filter/program -options", template(t_msg_with_day) ) );};
Then open your file in your program using the $S_DAY argument.
You could do it this way, but since syslog-ng has already parsed the syslog line, and already figured out what the value for $S_DAY is, it is more CPU efficient for syslog-ng to perform the determination of when a new instance of the program should be started. All of this code is already in syslog-ng, but it is limited to files, and excluded from program destinations.
My rule of thumb for where to put Syslog-NG in the log delivery chain is to use it solely as a receiver/parser/forwarder and to leave all of the actual disk writing and logic to a program written in a language more suited for decision making.
You have identified exactly what I am talking about. Syslog-ng has done the receiving, and the parsing, and then needs to forward the message to an appropriate program destination based on the parsed information.
If you don't need any advanced decision-making done with your logs, then using the great feature of macro file destinations like
destination d_file { file("/tmp/log.$S_DAY"); };
or even
destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
is really handy.
And it would be even more handy if it could be used on program destinations :-)
On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted?
The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion.
General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author.
Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted.
Evan Rempel University Systems
On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz
wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>>
Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
/path/log.$S_DAY"));};
I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
This could be implemented, however the current funcionality is completely related to file destinations, it is not generic in any way. It could be generalized though. I'd add this idea to our roadmap, but could you please describe your exact use case? Thanks. -- Bazsi
Balazs Scheidler wrote:
On Sat, 2009-12-05 at 16:52 -0800, Evan Rempel wrote:
When Syslog-NG starts up, it executes all of the program() statements with something like an execve system call (I'll defer to the Balabit guys for specifics). I understand that this is the way that syslog-ng works today, but I am discussing the why and what if about macro expansion in program destinations. Is there any reason to prevent someone from using macro expansions in program destinations?
In doing so, it will pass the program() configuration to the system call. This happens just once at startup. This forked program will have its own PID and in your case would look like "program -options >> /tmp/log.30" or something similar if you looked at a process list using ps aux. The key is that the "30" (or whatever the day was) is sent at program startup and becomes part of the program. No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value for $S_DAY do the command line, and thus the 30 does *not* appear at all.
You want that macro to be resolved on a per-message basis, which would mean that the operating system would have to go back in time to when the program first started and change the arguments it was given. I think time travel is planned in the 2.10 Linux kernel, but that may have been space travel--I can't remember now. If what you are saying is true, then the same issue exists for file names. What actually happens is that syslog-ng *evaluates* the destination template for each message, and if
1. The template is not open, opens it and writes the message to it. 2. if it is already open, write the message to it.
Additionally, if the destination does not receive a message for a certain time period, then the destination is closed.
All of this works fine for file destinations, but is not performed at all for program destinations.
I can not see any reason why this process would not work equally well for program destinations. I thought that I must have been missing something obvious in my evaluation of how things could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
Happily, the power of Syslog-NG can save you quite easily here and perform the "general function" you're looking for. The key is that Syslog-NG lets you format the messages being sent to your program() any way you want to using templates, so just tack on the $S_DAY macro to the log format you're using like this: template t_msg_with_day { template("$S_DAY|$MSG"); }; destination mydest{ program("/some/path/to/filter/program -options", template(t_msg_with_day) ) );};
Then open your file in your program using the $S_DAY argument. You could do it this way, but since syslog-ng has already parsed the syslog line, and already figured out what the value for $S_DAY is, it is more CPU efficient for syslog-ng to perform the determination of when a new instance of the program should be started. All of this code is already in syslog-ng, but it is limited to files, and excluded from program destinations.
My rule of thumb for where to put Syslog-NG in the log delivery chain is to use it solely as a receiver/parser/forwarder and to leave all of the actual disk writing and logic to a program written in a language more suited for decision making. You have identified exactly what I am talking about. Syslog-ng has done the receiving, and the parsing, and then needs to forward the message to an appropriate program destination based on the parsed information.
If you don't need any advanced decision-making done with your logs, then using the great feature of macro file destinations like
destination d_file { file("/tmp/log.$S_DAY"); };
or even
destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
is really handy. And it would be even more handy if it could be used on program destinations :-)
On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted?
The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion.
General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author.
Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted.
Evan Rempel University Systems
On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz
wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
> Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>> Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
/path/log.$S_DAY"));}; I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
This could be implemented, however the current funcionality is completely related to file destinations, it is not generic in any way. It could be generalized though. I'd add this idea to our roadmap, but could you please describe your exact use case?
We have a tool that does filtering based on a hierarchical configuration of regular expressions. The tool can also do actions based on matches etc. We use this tool for processing virtually all of our log files (apache, tomcat, ldap, custom apps and syslog). This tool is a filter in the true definition, so it only works on one input stream, and produces one output stream. I wanted to use syslog-ng's ability to "automatically" do log file rotation. OK, it's not really rotation, but it works, and IMHO is even better than rotation. I wanted to send a syslog-ng macro value to the application at startup and then when the macro value changes, start another instance of the application. destination { program("/my/filter/application $S_YEAR $S_MONTH $S_DAY > /some/output/file"); }; or destination { program("/my/filter/application > /some/output/file$S_YEAR$S_MONTH$S_DAY"); }; We have a second tool that performs statistical reporting on log streams. I wanted to have this tool produce statistics for each day, so need to feed it a single day of logs. I wanted to use syslog-ng's ability to invoke a new script each day; destination { program("/my/statistics/application $S_YEAR $S_MONTH $S_DAY"); }; At the end of the day, the statistical data can be placed into a database or e-mailed to some place. They key is how to get syslog-ng to start another instance of the program for each day. There are lots of workarounds, but just like all workarounds they are complicated and do not provide the exact same functionality. I am not going to list the workarounds because they make this discussion more complicated. An different issue is that if I configure a program destination using a macro variable, the variable is expanded by the SHELL, not by syslog-ng. This is not obvious at all, and can lead to very odd, and possibly security related issues. -- Evan Rempel
On Mon, 2009-12-07 at 11:20 -0800, Evan Rempel wrote:
Balazs Scheidler wrote:
On Sat, 2009-12-05 at 16:52 -0800, Evan Rempel wrote:
When Syslog-NG starts up, it executes all of the program() statements with something like an execve system call (I'll defer to the Balabit guys for specifics). I understand that this is the way that syslog-ng works today, but I am discussing the why and what if about macro expansion in program destinations. Is there any reason to prevent someone from using macro expansions in program destinations?
In doing so, it will pass the program() configuration to the system call. This happens just once at startup. This forked program will have its own PID and in your case would look like "program -options >> /tmp/log.30" or something similar if you looked at a process list using ps aux. The key is that the "30" (or whatever the day was) is sent at program startup and becomes part of the program. No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value for $S_DAY do the command line, and thus the 30 does *not* appear at all.
You want that macro to be resolved on a per-message basis, which would mean that the operating system would have to go back in time to when the program first started and change the arguments it was given. I think time travel is planned in the 2.10 Linux kernel, but that may have been space travel--I can't remember now. If what you are saying is true, then the same issue exists for file names. What actually happens is that syslog-ng *evaluates* the destination template for each message, and if
1. The template is not open, opens it and writes the message to it. 2. if it is already open, write the message to it.
Additionally, if the destination does not receive a message for a certain time period, then the destination is closed.
All of this works fine for file destinations, but is not performed at all for program destinations.
I can not see any reason why this process would not work equally well for program destinations. I thought that I must have been missing something obvious in my evaluation of how things could work. Telling me how syslog-ng works does *not* inform me of *why* it works that way.
Happily, the power of Syslog-NG can save you quite easily here and perform the "general function" you're looking for. The key is that Syslog-NG lets you format the messages being sent to your program() any way you want to using templates, so just tack on the $S_DAY macro to the log format you're using like this: template t_msg_with_day { template("$S_DAY|$MSG"); }; destination mydest{ program("/some/path/to/filter/program -options", template(t_msg_with_day) ) );};
Then open your file in your program using the $S_DAY argument. You could do it this way, but since syslog-ng has already parsed the syslog line, and already figured out what the value for $S_DAY is, it is more CPU efficient for syslog-ng to perform the determination of when a new instance of the program should be started. All of this code is already in syslog-ng, but it is limited to files, and excluded from program destinations.
My rule of thumb for where to put Syslog-NG in the log delivery chain is to use it solely as a receiver/parser/forwarder and to leave all of the actual disk writing and logic to a program written in a language more suited for decision making. You have identified exactly what I am talking about. Syslog-ng has done the receiving, and the parsing, and then needs to forward the message to an appropriate program destination based on the parsed information.
If you don't need any advanced decision-making done with your logs, then using the great feature of macro file destinations like
destination d_file { file("/tmp/log.$S_DAY"); };
or even
destination d_file { file("/tmp/logs/$S_YEAR/$S_MONTH/$S_DAY.log"); };
is really handy. And it would be even more handy if it could be used on program destinations :-)
On Fri, Dec 4, 2009 at 7:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I understand that the program destination should only be opened once and the messages sent to it but why does that mean the macro expansion should not be permitted?
The only work around right now is for the program to parse each message to extract the parts that are needed to manage the log rotation. Syslog-ng already does this AND syslog-ng already has the necessary logic to open a different destination based on the macro expantion.
General purpose tools need to perform general function and do them in the most intelegent manner. Disabling functionality in specefic situations in the name of protecting me from my own stupidity results in situations wher the tool could solve the problem but does not because of an arbitrary division of the author.
Unless there is an more important consideration such as security then I think the disallow of macro expantion in program destinations should be permitted.
Evan Rempel University Systems
On 2009-12-04, at 14:20, "chris packham" <chris.packham@alliedtelesis.co.nz
wrote:
This is intentional behaviour.
Syslog-ng starts any "program()" when syslog-ng is started and it expects that the program will perform read from stdin and block waiting for data. This protects syslog-ng againts the overhead of forking a new process everytime a matching log message is seen.
You can still use filters so I'd suggest modifying your program so that it can have the $S_DAY fed to it at the start of each log message.
>> Evan Rempel <erempel@uvic.ca> 12/04/09 4:07 PM >>> Is it possible to use macro expansion in the program destination. I wanted to write something like
destination mydest{ program("/some/path/to/filter/program -options
> /path/log.$S_DAY"));}; I was hoping that syslog-ng would not open this destination until the first message was ready for this destination.
I was hoping that syslog-ng would open a new one when the $S_DAY changed, allowing me to have each day of logs processed by my program with outputs based on day.
Does anyone know if macro expantion can be used on program destinations?
If not, can someone explain what I am overlooking that makes this a bad thing?
This could be implemented, however the current funcionality is completely related to file destinations, it is not generic in any way. It could be generalized though. I'd add this idea to our roadmap, but could you please describe your exact use case?
We have a tool that does filtering based on a hierarchical configuration of regular expressions. The tool can also do actions based on matches etc. We use this tool for processing virtually all of our log files (apache, tomcat, ldap, custom apps and syslog). This tool is a filter in the true definition, so it only works on one input stream, and produces one output stream.
I wanted to use syslog-ng's ability to "automatically" do log file rotation. OK, it's not really rotation, but it works, and IMHO is even better than rotation. I wanted to send a syslog-ng macro value to the application at startup and then when the macro value changes, start another instance of the application.
destination { program("/my/filter/application $S_YEAR $S_MONTH $S_DAY > /some/output/file"); }; or destination { program("/my/filter/application > /some/output/file$S_YEAR$S_MONTH$S_DAY"); };
We have a second tool that performs statistical reporting on log streams. I wanted to have this tool produce statistics for each day, so need to feed it a single day of logs.
I wanted to use syslog-ng's ability to invoke a new script each day;
destination { program("/my/statistics/application $S_YEAR $S_MONTH $S_DAY"); };
At the end of the day, the statistical data can be placed into a database or e-mailed to some place. They key is how to get syslog-ng to start another instance of the program for each day.
There are lots of workarounds, but just like all workarounds they are complicated and do not provide the exact same functionality. I am not going to list the workarounds because they make this discussion more complicated.
An different issue is that if I configure a program destination using a macro variable, the variable is expanded by the SHELL, not by syslog-ng. This is not obvious at all, and can lead to very odd, and possibly security related issues.
Thanks for the explanation. I've added this item to our internal roadmap (which is a SCRUM backlog these days), not yet scheduled to a release though, however I agree that such a feature should be useful in a number of cases. -- Bazsi
participants (4)
-
Balazs Scheidler
-
chris packham
-
Evan Rempel
-
Martin Holste