[syslog-ng] set tag in log statement
Balazs Scheidler
bazsi at balabit.hu
Mon Jan 9 11:35:00 CET 2012
On Thu, 2012-01-05 at 14:03 -0500, Patrick Hemmer wrote:
> Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST)
> From: Balazs Scheidler <bazsi at balabit.hu>
> To: Syslog-ng users' and developers' mailing list
> <syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] set tag in log statement
> > On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
> >> On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng at feystorm.net> wrote:
> >>
> >>> Would it be possible to set tags on a message when it is matched by a
> >>> `log {}` statement? This would be useful for complex log configurations
> >>> so you could filter out messages that were logged by previous `log {}`
> >>> statements without having to write really long `filter {}`s.
> >>>
> >>> I'm not sure if this is possible or not. Would depend on the order of
> >>> operations, like if a filter block in a latter log statement is
> >>> evaluated at the same time as an earlier filter block, or if its
> >>> evaluated when it reaches the `log {}`.
> >> Hi,
> >>
> >> currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
> >>
> >> Something like:
> >> rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
> >>
> > No, this wouldn't work. $TAGS is read-only right now.
> >
> >> Then a log statement would look like:
> >>
> >> log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
> >>
> >> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/modifying-messages.html
> >>
> >> You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
> >>
> >> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/conditional-rewrite.html
> > Would it not be enought to use the flags(final) rule on log statements,
> > perhaps using embedded log {} statements?
> >
> > This is equivalent to your configuration:
> >
> > filter f_a { facility(mail); };
> > log { source(s_local); filter(f_a); destination(d_a); flags(final); };
> >
> > filter f_b { facility(local0); };
> > log { source(s_local); filter(f_b); destination(d_b); flags(final); };
> >
> > log { source(s_local); destination (d_c); };
> Hrm, missed this reply somehow. Anyway, yes that would work for that
> example configuration. But it doesnt work so cleanly on more complex
> configurations.
>
> filter f_prog1 { program("prog1"); };
> log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); };
> filter f_prog2 { program("prog2"}; };
> log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
>
> filter f_crit { level(crit); };
> log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); };
> filter f_notcrit { not tags("tag_crit"); };
> log { source(s_local); filter(f_notcrit); destination(d_notcrit); };
>
> filter f_other { not tags("tag_prog"); };
> log { source(s_local); filter(f_other); destination(d_other); };
>
>
> That is a bit of a bizarre config, but was just an example. You could
> probably work some combination of fallback and final plus reordering to
> get it working, but you end up with really hard to read configs. Plus if
> you ever have to go and add a new log statement in, if you dont add it
> in the right place it might not work or it might end up breaking the
> fallback/final rules. It just becomes very difficult to configure.
I think I do understand what you need, although I think it needs a
different solution.
For syslog-ng, a log {} statement is a one-way street, changes on a log
statement would not affect the next one. Each fork along the log
processing tree is independent, changes on one should not affect the
other.
However I'm working on a change on the configuration model that would
probably solve your problem too.
This is called "junctions" in the lack of a better name.
Right now, syslog-ng is sending messages along a message processing
tree, e.g.:
source ---> log stmt1(filter, parser, rewrite, destination)
|
---> log stmt2(filter, parser, rewrite, destination)
|
---> log stmt3(filter, parser, rewrite, destination)
This is equivalent to this configuration:
log { source(src); filter(f1); parser(p1); rewrite(r1); destination(d1); };
log { source(src); filter(f2); parser(p2); rewrite(r2); destination(d2); };
log { source(src); filter(f3); parser(p3); rewrite(r3); destination(d3); };
Also, a log statement can add further forks, so the structure is a tree.
source ---> log stmt(filter, parser, rewrite)
\
-> log stmt1(filter, parser, rewrite, destination)
|
-> log stmt2(filter, parser, rewrite, destination)
|
-> log stmt3(filter, parser, rewrite, destination)
Which is equivalent to this one:
log { source(src); filter(f); parser(p); rewrite(r);
log { filter(f1); parser(p1); rewrite(r1); destination(d1); };
log { filter(f2); parser(p2); rewrite(r2); destination(d2); };
log { filter(f3); parser(p3); rewrite(r3); destination(d3); };
};
Each of the forks is independent from the neighbouring ones, so any
changes propagate only further down the tree, towards the leaves (e.g.
destinations).
Even in the current syslog-ng versions, further forks can be introduced
by adding another layer of log { } statements.
syslog-ng doesn't support cycles within this graph, e.g. once a fork is
made, those become independent chains of processing, and any changes
that happen on these branches are lost to the rest of the configruation.
The junction model changes that and allows joining branches.
log {
source(src);
junction {
log { filter(f1); parser(p1); rewrite(r1); destination(d1); };
log { filter(f2); parser(p2); rewrite(r2); destination(d2); };
log { filter(f3); parser(p3); rewrite(r3); destination(d3); };
};
destination(dall);
};
This config will result in "dall" receiving all the messages that fall
off the three branches above.
This can be used for interesting things like:
- doing db-parser() based information extraction in one case
- and doing csv-parser() in another.
junction {
log { filter(f_apache_msg); parser(p_apache_parser); flags(final); };
log { parser(p_dbparser()); };
};
This junction can be pasted into a log statement and the end results of
the two branches can be used for further processing.
For your usecase, the solution is to create such a junction that
categorizes messages using filters, attaches tags and such things. And
once done, destinations are hooked to the end result:
filter f_prog1 { program("prog1"); };
filter f_prog2 { program("prog2"}; };
filter f_crit { level(crit); };
filter f_notcrit { not tags("tag_crit"); };
filter f_other { not tags("tag_prog"); };
log { source(s_local);
junction {
log { filter(f_prog1); destination(d_prog1); tags("tag_prog"); };
log { filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
log { filter(f_crit); destination(d_crit); tags("tag_crit"); };
};
# from this point, tags are properly set
junction {
log { filter(f_notcrit); destination(d_notcrit); };
log { filter(f_other); destination(d_other); };
}
};
One thing to note though, is that multiple branches that emerge from a
junction point _each_ receive a copy of a message, so unless the filters are
completely independent, multiple messages will fall off the first junction.
There are other interesting stuff to do with junctions: it'll be possible to
associate rewrite/parser statements with sources and use it as a source.
For example:
source s_apache {
log {
source { file("/var/log/apache2/access.log"); };
parser { csv-parser(...); };
};
};
log { source(s_apache); destination(...); };
And voila, the s_apache source will emit structured information, instead
of mere log lines.
I have this working in a local git tree of mine, but I didn't like the
implementation, we're talking about code that have been in syslog-ng
in more than a decade. It badly needs some refactoring. I already did
that too, but the refactored version doesn't work yet.
This feature is the most important item on my 3.4 roadmap, so once complete,
I'd publish 3.4 alpha, and since much less changes have happened in 3.4 than
in 3.3, I hope to publish a final version before the summer.
At least that's the plan.
--
Bazsi
More information about the syslog-ng
mailing list