[syslog-ng] set tag in log statement
Martin Holste
mcholste at gmail.com
Mon Jan 9 20:04:47 CET 2012
This is very cool stuff. There are a lot of times when using a csv
parser would be best, but deviating from an already-present db-parser
is unmanageable. It would also be really great to be able to parse
first with db-parser and then further parse a given extracted field
with the csv parser, or vice-versa.
On Mon, Jan 9, 2012 at 4:35 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> On Thu, 2012-01-05 at 14:03 -0500, Patrick Hemmer wrote:
>> Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST)
>> From: Balazs Scheidler <bazsi at balabit.hu>
>> To: Syslog-ng users' and developers' mailing list
>> <syslog-ng at lists.balabit.hu>
>> Subject: Re: [syslog-ng] set tag in log statement
>> > On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
>> >> On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng at feystorm.net> wrote:
>> >>
>> >>> Would it be possible to set tags on a message when it is matched by a
>> >>> `log {}` statement? This would be useful for complex log configurations
>> >>> so you could filter out messages that were logged by previous `log {}`
>> >>> statements without having to write really long `filter {}`s.
>> >>>
>> >>> I'm not sure if this is possible or not. Would depend on the order of
>> >>> operations, like if a filter block in a latter log statement is
>> >>> evaluated at the same time as an earlier filter block, or if its
>> >>> evaluated when it reaches the `log {}`.
>> >> Hi,
>> >>
>> >> currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
>> >>
>> >> Something like:
>> >> rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
>> >>
>> > No, this wouldn't work. $TAGS is read-only right now.
>> >
>> >> Then a log statement would look like:
>> >>
>> >> log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
>> >>
>> >> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/modifying-messages.html
>> >>
>> >> You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
>> >>
>> >> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/conditional-rewrite.html
>> > Would it not be enought to use the flags(final) rule on log statements,
>> > perhaps using embedded log {} statements?
>> >
>> > This is equivalent to your configuration:
>> >
>> > filter f_a { facility(mail); };
>> > log { source(s_local); filter(f_a); destination(d_a); flags(final); };
>> >
>> > filter f_b { facility(local0); };
>> > log { source(s_local); filter(f_b); destination(d_b); flags(final); };
>> >
>> > log { source(s_local); destination (d_c); };
>> Hrm, missed this reply somehow. Anyway, yes that would work for that
>> example configuration. But it doesnt work so cleanly on more complex
>> configurations.
>>
>> filter f_prog1 { program("prog1"); };
>> log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); };
>> filter f_prog2 { program("prog2"}; };
>> log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
>>
>> filter f_crit { level(crit); };
>> log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); };
>> filter f_notcrit { not tags("tag_crit"); };
>> log { source(s_local); filter(f_notcrit); destination(d_notcrit); };
>>
>> filter f_other { not tags("tag_prog"); };
>> log { source(s_local); filter(f_other); destination(d_other); };
>>
>>
>> That is a bit of a bizarre config, but was just an example. You could
>> probably work some combination of fallback and final plus reordering to
>> get it working, but you end up with really hard to read configs. Plus if
>> you ever have to go and add a new log statement in, if you dont add it
>> in the right place it might not work or it might end up breaking the
>> fallback/final rules. It just becomes very difficult to configure.
>
> I think I do understand what you need, although I think it needs a
> different solution.
>
> For syslog-ng, a log {} statement is a one-way street, changes on a log
> statement would not affect the next one. Each fork along the log
> processing tree is independent, changes on one should not affect the
> other.
>
> However I'm working on a change on the configuration model that would
> probably solve your problem too.
>
> This is called "junctions" in the lack of a better name.
>
> Right now, syslog-ng is sending messages along a message processing
> tree, e.g.:
>
> source ---> log stmt1(filter, parser, rewrite, destination)
> |
> ---> log stmt2(filter, parser, rewrite, destination)
> |
> ---> log stmt3(filter, parser, rewrite, destination)
>
> This is equivalent to this configuration:
>
> log { source(src); filter(f1); parser(p1); rewrite(r1); destination(d1); };
> log { source(src); filter(f2); parser(p2); rewrite(r2); destination(d2); };
> log { source(src); filter(f3); parser(p3); rewrite(r3); destination(d3); };
>
> Also, a log statement can add further forks, so the structure is a tree.
>
> source ---> log stmt(filter, parser, rewrite)
> \
> -> log stmt1(filter, parser, rewrite, destination)
> |
> -> log stmt2(filter, parser, rewrite, destination)
> |
> -> log stmt3(filter, parser, rewrite, destination)
>
> Which is equivalent to this one:
>
> log { source(src); filter(f); parser(p); rewrite(r);
> log { filter(f1); parser(p1); rewrite(r1); destination(d1); };
> log { filter(f2); parser(p2); rewrite(r2); destination(d2); };
> log { filter(f3); parser(p3); rewrite(r3); destination(d3); };
> };
>
>
> Each of the forks is independent from the neighbouring ones, so any
> changes propagate only further down the tree, towards the leaves (e.g.
> destinations).
>
> Even in the current syslog-ng versions, further forks can be introduced
> by adding another layer of log { } statements.
>
> syslog-ng doesn't support cycles within this graph, e.g. once a fork is
> made, those become independent chains of processing, and any changes
> that happen on these branches are lost to the rest of the configruation.
>
> The junction model changes that and allows joining branches.
>
> log {
> source(src);
> junction {
> log { filter(f1); parser(p1); rewrite(r1); destination(d1); };
> log { filter(f2); parser(p2); rewrite(r2); destination(d2); };
> log { filter(f3); parser(p3); rewrite(r3); destination(d3); };
> };
> destination(dall);
> };
>
> This config will result in "dall" receiving all the messages that fall
> off the three branches above.
>
> This can be used for interesting things like:
>
> - doing db-parser() based information extraction in one case
> - and doing csv-parser() in another.
>
> junction {
> log { filter(f_apache_msg); parser(p_apache_parser); flags(final); };
> log { parser(p_dbparser()); };
> };
>
> This junction can be pasted into a log statement and the end results of
> the two branches can be used for further processing.
>
> For your usecase, the solution is to create such a junction that
> categorizes messages using filters, attaches tags and such things. And
> once done, destinations are hooked to the end result:
>
> filter f_prog1 { program("prog1"); };
> filter f_prog2 { program("prog2"}; };
> filter f_crit { level(crit); };
> filter f_notcrit { not tags("tag_crit"); };
> filter f_other { not tags("tag_prog"); };
>
> log { source(s_local);
> junction {
> log { filter(f_prog1); destination(d_prog1); tags("tag_prog"); };
> log { filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
> log { filter(f_crit); destination(d_crit); tags("tag_crit"); };
> };
> # from this point, tags are properly set
> junction {
> log { filter(f_notcrit); destination(d_notcrit); };
> log { filter(f_other); destination(d_other); };
> }
> };
>
> One thing to note though, is that multiple branches that emerge from a
> junction point _each_ receive a copy of a message, so unless the filters are
> completely independent, multiple messages will fall off the first junction.
>
> There are other interesting stuff to do with junctions: it'll be possible to
> associate rewrite/parser statements with sources and use it as a source.
>
> For example:
>
> source s_apache {
> log {
> source { file("/var/log/apache2/access.log"); };
> parser { csv-parser(...); };
> };
> };
>
> log { source(s_apache); destination(...); };
>
> And voila, the s_apache source will emit structured information, instead
> of mere log lines.
>
> I have this working in a local git tree of mine, but I didn't like the
> implementation, we're talking about code that have been in syslog-ng
> in more than a decade. It badly needs some refactoring. I already did
> that too, but the refactored version doesn't work yet.
>
> This feature is the most important item on my 3.4 roadmap, so once complete,
> I'd publish 3.4 alpha, and since much less changes have happened in 3.4 than
> in 3.3, I hope to publish a final version before the summer.
>
> At least that's the plan.
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list