[syslog-ng] set tag in log statement
Fekete Róbert
frobert at balabit.hu
Mon Jan 9 20:21:05 CET 2012
On Monday, January 9, 2012 20:04 CET, Martin Holste <mcholste at gmail.com> wrote:
> This is very cool stuff. There are a lot of times when using a csv
> parser would be best, but deviating from an already-present db-parser
> is unmanageable. It would also be really great to be able to parse
> first with db-parser and then further parse a given extracted field
> with the csv parser, or vice-versa.
AFAIK, running a CSV parser on a field extracted by patterndb should be possible even now, you just have to specify a template for the csv parser so that it runs on the specified field, and not on MSG. (Although in 3.3.3 this seems to be broken, but this will be hopefully corrected.)
Robert
>
> On Mon, Jan 9, 2012 at 4:35 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > On Thu, 2012-01-05 at 14:03 -0500, Patrick Hemmer wrote:
> >> Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST)
> >> From: Balazs Scheidler <bazsi at balabit.hu>
> >> To: Syslog-ng users' and developers' mailing list
> >> <syslog-ng at lists.balabit.hu>
> >> Subject: Re: [syslog-ng] set tag in log statement
> >> > On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
> >> >> On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng at feystorm.net> wrote:
> >> >>
> >> >>> Would it be possible to set tags on a message when it is matched by a
> >> >>> `log {}` statement? This would be useful for complex log configurations
> >> >>> so you could filter out messages that were logged by previous `log {}`
> >> >>> statements without having to write really long `filter {}`s.
> >> >>>
> >> >>> I'm not sure if this is possible or not. Would depend on the order of
> >> >>> operations, like if a filter block in a latter log statement is
> >> >>> evaluated at the same time as an earlier filter block, or if its
> >> >>> evaluated when it reaches the `log {}`.
> >> >> Hi,
> >> >>
> >> >> currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
> >> >>
> >> >> Something like:
> >> >> rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
> >> >>
> >> > No, this wouldn't work. $TAGS is read-only right now.
> >> >
> >> >> Then a log statement would look like:
> >> >>
> >> >> log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
> >> >>
> >> >> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/modifying-messages.html
> >> >>
> >> >> You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
> >> >>
> >> >> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/conditional-rewrite.html
> >> > Would it not be enought to use the flags(final) rule on log statements,
> >> > perhaps using embedded log {} statements?
> >> >
> >> > This is equivalent to your configuration:
> >> >
> >> > filter f_a { facility(mail); };
> >> > log { source(s_local); filter(f_a); destination(d_a); flags(final); };
> >> >
> >> > filter f_b { facility(local0); };
> >> > log { source(s_local); filter(f_b); destination(d_b); flags(final); };
> >> >
> >> > log { source(s_local); destination (d_c); };
> >> Hrm, missed this reply somehow. Anyway, yes that would work for that
> >> example configuration. But it doesnt work so cleanly on more complex
> >> configurations.
> >>
> >> filter f_prog1 { program("prog1"); };
> >> log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); };
> >> filter f_prog2 { program("prog2"}; };
> >> log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
> >>
> >> filter f_crit { level(crit); };
> >> log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); };
> >> filter f_notcrit { not tags("tag_crit"); };
> >> log { source(s_local); filter(f_notcrit); destination(d_notcrit); };
> >>
> >> filter f_other { not tags("tag_prog"); };
> >> log { source(s_local); filter(f_other); destination(d_other); };
> >>
> >>
> >> That is a bit of a bizarre config, but was just an example. You could
> >> probably work some combination of fallback and final plus reordering to
> >> get it working, but you end up with really hard to read configs. Plus if
> >> you ever have to go and add a new log statement in, if you dont add it
> >> in the right place it might not work or it might end up breaking the
> >> fallback/final rules. It just becomes very difficult to configure.
> >
> > I think I do understand what you need, although I think it needs a
> > different solution.
> >
> > For syslog-ng, a log {} statement is a one-way street, changes on a log
> > statement would not affect the next one. Each fork along the log
> > processing tree is independent, changes on one should not affect the
> > other.
> >
> > However I'm working on a change on the configuration model that would
> > probably solve your problem too.
> >
> > This is called "junctions" in the lack of a better name.
> >
> > Right now, syslog-ng is sending messages along a message processing
> > tree, e.g.:
> >
> > source ---> log stmt1(filter, parser, rewrite, destination)
> > |
> > ---> log stmt2(filter, parser, rewrite, destination)
> > |
> > ---> log stmt3(filter, parser, rewrite, destination)
> >
> > This is equivalent to this configuration:
> >
> > log { source(src); filter(f1); parser(p1); rewrite(r1); destination(d1); };
> > log { source(src); filter(f2); parser(p2); rewrite(r2); destination(d2); };
> > log { source(src); filter(f3); parser(p3); rewrite(r3); destination(d3); };
> >
> > Also, a log statement can add further forks, so the structure is a tree.
> >
> > source ---> log stmt(filter, parser, rewrite)
> > \
> > -> log stmt1(filter, parser, rewrite, destination)
> > |
> > -> log stmt2(filter, parser, rewrite, destination)
> > |
> > -> log stmt3(filter, parser, rewrite, destination)
> >
> > Which is equivalent to this one:
> >
> > log { source(src); filter(f); parser(p); rewrite(r);
> > log { filter(f1); parser(p1); rewrite(r1); destination(d1); };
> > log { filter(f2); parser(p2); rewrite(r2); destination(d2); };
> > log { filter(f3); parser(p3); rewrite(r3); destination(d3); };
> > };
> >
> >
> > Each of the forks is independent from the neighbouring ones, so any
> > changes propagate only further down the tree, towards the leaves (e.g.
> > destinations).
> >
> > Even in the current syslog-ng versions, further forks can be introduced
> > by adding another layer of log { } statements.
> >
> > syslog-ng doesn't support cycles within this graph, e.g. once a fork is
> > made, those become independent chains of processing, and any changes
> > that happen on these branches are lost to the rest of the configruation.
> >
> > The junction model changes that and allows joining branches.
> >
> > log {
> > source(src);
> > junction {
> > log { filter(f1); parser(p1); rewrite(r1); destination(d1); };
> > log { filter(f2); parser(p2); rewrite(r2); destination(d2); };
> > log { filter(f3); parser(p3); rewrite(r3); destination(d3); };
> > };
> > destination(dall);
> > };
> >
> > This config will result in "dall" receiving all the messages that fall
> > off the three branches above.
> >
> > This can be used for interesting things like:
> >
> > - doing db-parser() based information extraction in one case
> > - and doing csv-parser() in another.
> >
> > junction {
> > log { filter(f_apache_msg); parser(p_apache_parser); flags(final); };
> > log { parser(p_dbparser()); };
> > };
> >
> > This junction can be pasted into a log statement and the end results of
> > the two branches can be used for further processing.
> >
> > For your usecase, the solution is to create such a junction that
> > categorizes messages using filters, attaches tags and such things. And
> > once done, destinations are hooked to the end result:
> >
> > filter f_prog1 { program("prog1"); };
> > filter f_prog2 { program("prog2"}; };
> > filter f_crit { level(crit); };
> > filter f_notcrit { not tags("tag_crit"); };
> > filter f_other { not tags("tag_prog"); };
> >
> > log { source(s_local);
> > junction {
> > log { filter(f_prog1); destination(d_prog1); tags("tag_prog"); };
> > log { filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
> > log { filter(f_crit); destination(d_crit); tags("tag_crit"); };
> > };
> > # from this point, tags are properly set
> > junction {
> > log { filter(f_notcrit); destination(d_notcrit); };
> > log { filter(f_other); destination(d_other); };
> > }
> > };
> >
> > One thing to note though, is that multiple branches that emerge from a
> > junction point _each_ receive a copy of a message, so unless the filters are
> > completely independent, multiple messages will fall off the first junction.
> >
> > There are other interesting stuff to do with junctions: it'll be possible to
> > associate rewrite/parser statements with sources and use it as a source.
> >
> > For example:
> >
> > source s_apache {
> > log {
> > source { file("/var/log/apache2/access.log"); };
> > parser { csv-parser(...); };
> > };
> > };
> >
> > log { source(s_apache); destination(...); };
> >
> > And voila, the s_apache source will emit structured information, instead
> > of mere log lines.
> >
> > I have this working in a local git tree of mine, but I didn't like the
> > implementation, we're talking about code that have been in syslog-ng
> > in more than a decade. It badly needs some refactoring. I already did
> > that too, but the refactored version doesn't work yet.
> >
> > This feature is the most important item on my 3.4 roadmap, so once complete,
> > I'd publish 3.4 alpha, and since much less changes have happened in 3.4 than
> > in 3.3, I hope to publish a final version before the summer.
> >
> > At least that's the plan.
> >
> > --
> > Bazsi
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
More information about the syslog-ng
mailing list