That will probably be more resource-intensive than I'd like with all of the regexp, but that is definitely a legitimate work-around. I suppose I could try to write a patch which would create a macro to apply a given modulo to the existing R_UNIXTIME source if the workaround doesn't scale. Thanks!<br>
<br><div class="gmail_quote">On Thu, Jun 18, 2009 at 11:54 PM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div></div><div class="h5">On Wed, 2009-06-17 at 15:51 -0500, Martin Holste wrote:<br>
> The indexing system I'm writing will work best if it can read in<br>
> syslog-ng logs as regular files (as opposed to fifos, sockets or SQL).<br>
> In order to have multiple indexing workers going at the same time, I'm<br>
> finding it easiest if the output from syslog-ng can be divided into<br>
> chunks like this:<br>
><br>
> destination d_file<br>
> { file("/tmp/slices/$R_UNIXTIME.${.classifier.rule_id}"<br>
> template(t_db_parsed)); };<br>
><br>
> The workers index the chunks as syslog-ng spits them into the slices<br>
> buffer directory. This works pretty well, but I'm limited to tuning<br>
> only between one second intervals provided by $R_UNIXTIME, or minute,<br>
> hour, etc. intervals by concatenating other time macros together. Is<br>
> there some way I could get syslog-ng to spit out something lke 5<br>
> second slices? Also, will syslog-ng know to close the file handle on<br>
> a log slice when the $R_UNIXTIME second has elapsed?<br>
><br>
<br>
</div></div>syslog-ng has no builtin macro for that, but since you are already using<br>
3.0, you could achieve the same by cutting the appropriate value from<br>
the syslog message itself.<br>
<br>
For instance:<br>
<br>
rewrite p_date_to_values {<br>
set("$R_DATE", value("rdate"));<br>
};<br>
<br>
filter f_get_second_chunk {<br>
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$' type(pcre) value('rdate'));<br>
};<br>
<br>
The point of the first rewrite rule is to convert the R_DATE macro to a value<br>
(see bugzilla #37) and the second filter uses PCRE regular expressions<br>
to parse the first digit of the seconds into a value called "rdate.second_tens"<br>
<br>
Then, you can use this as a macro in your filename:<br>
<br>
file("/tmp/slices/${rdate.second_tens}.${.classifier.rule_id}" template(t_db_parsed));<br>
<br>
This should be made easier syntax wise, but should do the job.<br>
<font color="#888888"><br>
--<br>
Bazsi<br>
<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</font></blockquote></div><br>