That will probably be more resource-intensive than I'd like with all of the regexp, but that is definitely a legitimate work-around. I suppose I could try to write a patch which would create a macro to apply a given modulo to the existing R_UNIXTIME source if the workaround doesn't scale. Thanks! On Thu, Jun 18, 2009 at 11:54 PM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2009-06-17 at 15:51 -0500, Martin Holste wrote:
The indexing system I'm writing will work best if it can read in syslog-ng logs as regular files (as opposed to fifos, sockets or SQL). In order to have multiple indexing workers going at the same time, I'm finding it easiest if the output from syslog-ng can be divided into chunks like this:
destination d_file { file("/tmp/slices/$R_UNIXTIME.${.classifier.rule_id}" template(t_db_parsed)); };
The workers index the chunks as syslog-ng spits them into the slices buffer directory. This works pretty well, but I'm limited to tuning only between one second intervals provided by $R_UNIXTIME, or minute, hour, etc. intervals by concatenating other time macros together. Is there some way I could get syslog-ng to spit out something lke 5 second slices? Also, will syslog-ng know to close the file handle on a log slice when the $R_UNIXTIME second has elapsed?
syslog-ng has no builtin macro for that, but since you are already using 3.0, you could achieve the same by cutting the appropriate value from the syslog message itself.
For instance:
rewrite p_date_to_values { set("$R_DATE", value("rdate")); };
filter f_get_second_chunk { match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$' type(pcre) value('rdate')); };
The point of the first rewrite rule is to convert the R_DATE macro to a value (see bugzilla #37) and the second filter uses PCRE regular expressions to parse the first digit of the seconds into a value called "rdate.second_tens"
Then, you can use this as a macro in your filename:
file("/tmp/slices/${rdate.second_tens}.${.classifier.rule_id}" template(t_db_parsed));
This should be made easier syntax wise, but should do the job.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html