[syslog-ng] Time slice destination files

[Balazs Scheidler]

On Wed, 2009-06-17 at 15:51 -0500, Martin Holste wrote:
> The indexing system I'm writing will work best if it can read in
> syslog-ng logs as regular files (as opposed to fifos, sockets or SQL).
> In order to have multiple indexing workers going at the same time, I'm
> finding it easiest if the output from syslog-ng can be divided into
> chunks like this:
> 
> destination d_file
> { file("/tmp/slices/$R_UNIXTIME.${.classifier.rule_id}"
> template(t_db_parsed)); };
> 
> The workers index the chunks as syslog-ng spits them into the slices
> buffer directory.  This works pretty well, but I'm limited to tuning
> only between one second intervals provided by $R_UNIXTIME, or minute,
> hour, etc. intervals by concatenating other time macros together.  Is
> there some way I could get syslog-ng to spit out something lke 5
> second slices?  Also, will syslog-ng know to close the file handle on
> a log slice when the $R_UNIXTIME second has elapsed?
> 

syslog-ng has no builtin macro for that, but since you are already using
3.0, you could achieve the same by cutting the appropriate value from
the syslog message itself.

For instance:

rewrite p_date_to_values { 
	set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
	match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$' type(pcre) value('rdate'));
};

The point of the first rewrite rule is to convert the R_DATE macro to a value 
(see bugzilla #37) and the second filter uses PCRE regular expressions
to parse the first digit of the seconds into a value called "rdate.second_tens"

Then, you can use this as a macro in your filename:

file("/tmp/slices/${rdate.second_tens}.${.classifier.rule_id}" template(t_db_parsed));

This should be made easier syntax wise, but should do the job.

-- 
Bazsi