[syslog-ng] Time slice destination files

[Martin Holste]

That will probably be more resource-intensive than I'd like with all of the
regexp, but that is definitely a legitimate work-around.  I suppose I could
try to write a patch which would create a macro to apply a given modulo to
the existing R_UNIXTIME source if the workaround doesn't scale.  Thanks!

On Thu, Jun 18, 2009 at 11:54 PM, Balazs Scheidler <bazsi at balabit.hu> wrote:

> On Wed, 2009-06-17 at 15:51 -0500, Martin Holste wrote:
> > The indexing system I'm writing will work best if it can read in
> > syslog-ng logs as regular files (as opposed to fifos, sockets or SQL).
> > In order to have multiple indexing workers going at the same time, I'm
> > finding it easiest if the output from syslog-ng can be divided into
> > chunks like this:
> >
> > destination d_file
> > { file("/tmp/slices/$R_UNIXTIME.${.classifier.rule_id}"
> > template(t_db_parsed)); };
> >
> > The workers index the chunks as syslog-ng spits them into the slices
> > buffer directory.  This works pretty well, but I'm limited to tuning
> > only between one second intervals provided by $R_UNIXTIME, or minute,
> > hour, etc. intervals by concatenating other time macros together.  Is
> > there some way I could get syslog-ng to spit out something lke 5
> > second slices?  Also, will syslog-ng know to close the file handle on
> > a log slice when the $R_UNIXTIME second has elapsed?
> >
>
> syslog-ng has no builtin macro for that, but since you are already using
> 3.0, you could achieve the same by cutting the appropriate value from
> the syslog message itself.
>
> For instance:
>
> rewrite p_date_to_values {
>        set("$R_DATE", value("rdate"));
> };
>
> filter f_get_second_chunk {
>        match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$'
> type(pcre) value('rdate'));
> };
>
> The point of the first rewrite rule is to convert the R_DATE macro to a
> value
> (see bugzilla #37) and the second filter uses PCRE regular expressions
> to parse the first digit of the seconds into a value called
> "rdate.second_tens"
>
> Then, you can use this as a macro in your filename:
>
> file("/tmp/slices/${rdate.second_tens}.${.classifier.rule_id}"
> template(t_db_parsed));
>
> This should be made easier syntax wise, but should do the job.
>
> --
> Bazsi
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090619/ec85ccbc/attachment.htm