On Thu, Jun 18, 2009 at 11:54 PM, Balazs Scheidler <bazsi@balabit.hu> wrote:

On Wed, 2009-06-17 at 15:51 -0500, Martin Holste wrote:
> The indexing system I'm writing will work best if it can read in
> syslog-ng logs as regular files (as opposed to fifos, sockets or SQL).
> In order to have multiple indexing workers going at the same time, I'm
> finding it easiest if the output from syslog-ng can be divided into
> chunks like this:
>
> destination d_file
> { file("/tmp/slices/$R_UNIXTIME.${.classifier.rule_id}"
> template(t_db_parsed)); };
>
> The workers index the chunks as syslog-ng spits them into the slices
> buffer directory. This works pretty well, but I'm limited to tuning
> only between one second intervals provided by $R_UNIXTIME, or minute,
> hour, etc. intervals by concatenating other time macros together. Is
> there some way I could get syslog-ng to spit out something lke 5
> second slices? Also, will syslog-ng know to close the file handle on
> a log slice when the $R_UNIXTIME second has elapsed?
>

syslog-ng has no builtin macro for that, but since you are already using
3.0, you could achieve the same by cutting the appropriate value from
the syslog message itself.

For instance:

rewrite p_date_to_values {
set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$' type(pcre) value('rdate'));
};

The point of the first rewrite rule is to convert the R_DATE macro to a value
(see bugzilla #37) and the second filter uses PCRE regular expressions
to parse the first digit of the seconds into a value called "rdate.second_tens"

Then, you can use this as a macro in your filename:

file("/tmp/slices/${rdate.second_tens}.${.classifier.rule_id}" template(t_db_parsed));

This should be made easier syntax wise, but should do the job.

--
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html