set tag in log statement
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s. I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Such a config would look like filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); tags("tag1"); }; filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); tags("tag1"); }; filter f_c { not tags("tag1"); }; log { source(s_local); filter(f_c); destination (d_c); };
On Friday, December 16, 2011 21:59 CET, Patrick Hemmer <syslogng@feystorm.net> wrote:
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s.
I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`.
Hi, currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message. Something like: rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));}; Then a log statement would look like: log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); }; http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement. http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... Regards, Robert
Such a config would look like
filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); tags("tag1"); };
filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); tags("tag1"); };
filter f_c { not tags("tag1"); }; log { source(s_local); filter(f_c); destination (d_c); }; ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
On Friday, December 16, 2011 21:59 CET, Patrick Hemmer <syslogng@feystorm.net> wrote:
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s.
I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`.
Hi,
currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
Something like: rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
No, this wouldn't work. $TAGS is read-only right now.
Then a log statement would look like:
log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
Would it not be enought to use the flags(final) rule on log statements, perhaps using embedded log {} statements? This is equivalent to your configuration: filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); flags(final); }; filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); flags(final); }; log { source(s_local); destination (d_c); }; -- Bazsi
Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] set tag in log statement
On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng@feystorm.net> wrote:
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s.
I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Hi,
currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
Something like: rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
No, this wouldn't work. $TAGS is read-only right now.
Then a log statement would look like:
log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... Would it not be enought to use the flags(final) rule on log statements, perhaps using embedded log {} statements?
This is equivalent to your configuration:
filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); flags(final); };
filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); flags(final); };
log { source(s_local); destination (d_c); }; Hrm, missed this reply somehow. Anyway, yes that would work for that example configuration. But it doesnt work so cleanly on more complex configurations.
filter f_prog1 { program("prog1"); }; log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; filter f_prog2 { program("prog2"}; }; log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); }; filter f_crit { level(crit); }; log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); }; filter f_notcrit { not tags("tag_crit"); }; log { source(s_local); filter(f_notcrit); destination(d_notcrit); }; filter f_other { not tags("tag_prog"); }; log { source(s_local); filter(f_other); destination(d_other); }; That is a bit of a bizarre config, but was just an example. You could probably work some combination of fallback and final plus reordering to get it working, but you end up with really hard to read configs. Plus if you ever have to go and add a new log statement in, if you dont add it in the right place it might not work or it might end up breaking the fallback/final rules. It just becomes very difficult to configure. This
On Thu, 2012-01-05 at 14:03 -0500, Patrick Hemmer wrote:
Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] set tag in log statement
On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng@feystorm.net> wrote:
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s.
I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Hi,
currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
Something like: rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
No, this wouldn't work. $TAGS is read-only right now.
Then a log statement would look like:
log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... Would it not be enought to use the flags(final) rule on log statements, perhaps using embedded log {} statements?
This is equivalent to your configuration:
filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); flags(final); };
filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); flags(final); };
log { source(s_local); destination (d_c); }; Hrm, missed this reply somehow. Anyway, yes that would work for that example configuration. But it doesnt work so cleanly on more complex configurations.
filter f_prog1 { program("prog1"); }; log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; filter f_prog2 { program("prog2"}; }; log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
filter f_crit { level(crit); }; log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); }; filter f_notcrit { not tags("tag_crit"); }; log { source(s_local); filter(f_notcrit); destination(d_notcrit); };
filter f_other { not tags("tag_prog"); }; log { source(s_local); filter(f_other); destination(d_other); };
That is a bit of a bizarre config, but was just an example. You could probably work some combination of fallback and final plus reordering to get it working, but you end up with really hard to read configs. Plus if you ever have to go and add a new log statement in, if you dont add it in the right place it might not work or it might end up breaking the fallback/final rules. It just becomes very difficult to configure.
I think I do understand what you need, although I think it needs a different solution. For syslog-ng, a log {} statement is a one-way street, changes on a log statement would not affect the next one. Each fork along the log processing tree is independent, changes on one should not affect the other. However I'm working on a change on the configuration model that would probably solve your problem too. This is called "junctions" in the lack of a better name. Right now, syslog-ng is sending messages along a message processing tree, e.g.: source ---> log stmt1(filter, parser, rewrite, destination) | ---> log stmt2(filter, parser, rewrite, destination) | ---> log stmt3(filter, parser, rewrite, destination) This is equivalent to this configuration: log { source(src); filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { source(src); filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { source(src); filter(f3); parser(p3); rewrite(r3); destination(d3); }; Also, a log statement can add further forks, so the structure is a tree. source ---> log stmt(filter, parser, rewrite) \ -> log stmt1(filter, parser, rewrite, destination) | -> log stmt2(filter, parser, rewrite, destination) | -> log stmt3(filter, parser, rewrite, destination) Which is equivalent to this one: log { source(src); filter(f); parser(p); rewrite(r); log { filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { filter(f3); parser(p3); rewrite(r3); destination(d3); }; }; Each of the forks is independent from the neighbouring ones, so any changes propagate only further down the tree, towards the leaves (e.g. destinations). Even in the current syslog-ng versions, further forks can be introduced by adding another layer of log { } statements. syslog-ng doesn't support cycles within this graph, e.g. once a fork is made, those become independent chains of processing, and any changes that happen on these branches are lost to the rest of the configruation. The junction model changes that and allows joining branches. log { source(src); junction { log { filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { filter(f3); parser(p3); rewrite(r3); destination(d3); }; }; destination(dall); }; This config will result in "dall" receiving all the messages that fall off the three branches above. This can be used for interesting things like: - doing db-parser() based information extraction in one case - and doing csv-parser() in another. junction { log { filter(f_apache_msg); parser(p_apache_parser); flags(final); }; log { parser(p_dbparser()); }; }; This junction can be pasted into a log statement and the end results of the two branches can be used for further processing. For your usecase, the solution is to create such a junction that categorizes messages using filters, attaches tags and such things. And once done, destinations are hooked to the end result: filter f_prog1 { program("prog1"); }; filter f_prog2 { program("prog2"}; }; filter f_crit { level(crit); }; filter f_notcrit { not tags("tag_crit"); }; filter f_other { not tags("tag_prog"); }; log { source(s_local); junction { log { filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; log { filter(f_prog2); destination(d_prog2); tags("tag_prog"); }; log { filter(f_crit); destination(d_crit); tags("tag_crit"); }; }; # from this point, tags are properly set junction { log { filter(f_notcrit); destination(d_notcrit); }; log { filter(f_other); destination(d_other); }; } }; One thing to note though, is that multiple branches that emerge from a junction point _each_ receive a copy of a message, so unless the filters are completely independent, multiple messages will fall off the first junction. There are other interesting stuff to do with junctions: it'll be possible to associate rewrite/parser statements with sources and use it as a source. For example: source s_apache { log { source { file("/var/log/apache2/access.log"); }; parser { csv-parser(...); }; }; }; log { source(s_apache); destination(...); }; And voila, the s_apache source will emit structured information, instead of mere log lines. I have this working in a local git tree of mine, but I didn't like the implementation, we're talking about code that have been in syslog-ng in more than a decade. It badly needs some refactoring. I already did that too, but the refactored version doesn't work yet. This feature is the most important item on my 3.4 roadmap, so once complete, I'd publish 3.4 alpha, and since much less changes have happened in 3.4 than in 3.3, I hope to publish a final version before the summer. At least that's the plan. -- Bazsi
This is very cool stuff. There are a lot of times when using a csv parser would be best, but deviating from an already-present db-parser is unmanageable. It would also be really great to be able to parse first with db-parser and then further parse a given extracted field with the csv parser, or vice-versa. On Mon, Jan 9, 2012 at 4:35 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2012-01-05 at 14:03 -0500, Patrick Hemmer wrote:
Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] set tag in log statement
On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng@feystorm.net> wrote:
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s.
I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Hi,
currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
Something like: rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
No, this wouldn't work. $TAGS is read-only right now.
Then a log statement would look like:
log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... Would it not be enought to use the flags(final) rule on log statements, perhaps using embedded log {} statements?
This is equivalent to your configuration:
filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); flags(final); };
filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); flags(final); };
log { source(s_local); destination (d_c); }; Hrm, missed this reply somehow. Anyway, yes that would work for that example configuration. But it doesnt work so cleanly on more complex configurations.
filter f_prog1 { program("prog1"); }; log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; filter f_prog2 { program("prog2"}; }; log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
filter f_crit { level(crit); }; log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); }; filter f_notcrit { not tags("tag_crit"); }; log { source(s_local); filter(f_notcrit); destination(d_notcrit); };
filter f_other { not tags("tag_prog"); }; log { source(s_local); filter(f_other); destination(d_other); };
That is a bit of a bizarre config, but was just an example. You could probably work some combination of fallback and final plus reordering to get it working, but you end up with really hard to read configs. Plus if you ever have to go and add a new log statement in, if you dont add it in the right place it might not work or it might end up breaking the fallback/final rules. It just becomes very difficult to configure.
I think I do understand what you need, although I think it needs a different solution.
For syslog-ng, a log {} statement is a one-way street, changes on a log statement would not affect the next one. Each fork along the log processing tree is independent, changes on one should not affect the other.
However I'm working on a change on the configuration model that would probably solve your problem too.
This is called "junctions" in the lack of a better name.
Right now, syslog-ng is sending messages along a message processing tree, e.g.:
source ---> log stmt1(filter, parser, rewrite, destination) | ---> log stmt2(filter, parser, rewrite, destination) | ---> log stmt3(filter, parser, rewrite, destination)
This is equivalent to this configuration:
log { source(src); filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { source(src); filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { source(src); filter(f3); parser(p3); rewrite(r3); destination(d3); };
Also, a log statement can add further forks, so the structure is a tree.
source ---> log stmt(filter, parser, rewrite) \ -> log stmt1(filter, parser, rewrite, destination) | -> log stmt2(filter, parser, rewrite, destination) | -> log stmt3(filter, parser, rewrite, destination)
Which is equivalent to this one:
log { source(src); filter(f); parser(p); rewrite(r); log { filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { filter(f3); parser(p3); rewrite(r3); destination(d3); }; };
Each of the forks is independent from the neighbouring ones, so any changes propagate only further down the tree, towards the leaves (e.g. destinations).
Even in the current syslog-ng versions, further forks can be introduced by adding another layer of log { } statements.
syslog-ng doesn't support cycles within this graph, e.g. once a fork is made, those become independent chains of processing, and any changes that happen on these branches are lost to the rest of the configruation.
The junction model changes that and allows joining branches.
log { source(src); junction { log { filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { filter(f3); parser(p3); rewrite(r3); destination(d3); }; }; destination(dall); };
This config will result in "dall" receiving all the messages that fall off the three branches above.
This can be used for interesting things like:
- doing db-parser() based information extraction in one case - and doing csv-parser() in another.
junction { log { filter(f_apache_msg); parser(p_apache_parser); flags(final); }; log { parser(p_dbparser()); }; };
This junction can be pasted into a log statement and the end results of the two branches can be used for further processing.
For your usecase, the solution is to create such a junction that categorizes messages using filters, attaches tags and such things. And once done, destinations are hooked to the end result:
filter f_prog1 { program("prog1"); }; filter f_prog2 { program("prog2"}; }; filter f_crit { level(crit); }; filter f_notcrit { not tags("tag_crit"); }; filter f_other { not tags("tag_prog"); };
log { source(s_local); junction { log { filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; log { filter(f_prog2); destination(d_prog2); tags("tag_prog"); }; log { filter(f_crit); destination(d_crit); tags("tag_crit"); }; }; # from this point, tags are properly set junction { log { filter(f_notcrit); destination(d_notcrit); }; log { filter(f_other); destination(d_other); }; } };
One thing to note though, is that multiple branches that emerge from a junction point _each_ receive a copy of a message, so unless the filters are completely independent, multiple messages will fall off the first junction.
There are other interesting stuff to do with junctions: it'll be possible to associate rewrite/parser statements with sources and use it as a source.
For example:
source s_apache { log { source { file("/var/log/apache2/access.log"); }; parser { csv-parser(...); }; }; };
log { source(s_apache); destination(...); };
And voila, the s_apache source will emit structured information, instead of mere log lines.
I have this working in a local git tree of mine, but I didn't like the implementation, we're talking about code that have been in syslog-ng in more than a decade. It badly needs some refactoring. I already did that too, but the refactored version doesn't work yet.
This feature is the most important item on my 3.4 roadmap, so once complete, I'd publish 3.4 alpha, and since much less changes have happened in 3.4 than in 3.3, I hope to publish a final version before the summer.
At least that's the plan.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Monday, January 9, 2012 20:04 CET, Martin Holste <mcholste@gmail.com> wrote:
This is very cool stuff. There are a lot of times when using a csv parser would be best, but deviating from an already-present db-parser is unmanageable. It would also be really great to be able to parse first with db-parser and then further parse a given extracted field with the csv parser, or vice-versa. AFAIK, running a CSV parser on a field extracted by patterndb should be possible even now, you just have to specify a template for the csv parser so that it runs on the specified field, and not on MSG. (Although in 3.3.3 this seems to be broken, but this will be hopefully corrected.)
Robert
On Mon, Jan 9, 2012 at 4:35 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2012-01-05 at 14:03 -0500, Patrick Hemmer wrote:
Sent: Wed Dec 21 2011 08:13:13 GMT-0500 (EST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] set tag in log statement
On Sat, 2011-12-17 at 20:26 +0100, Fekete Róbert wrote:
On Friday, December 16, 2011 21:59 CET, Patrick Hemmer<syslogng@feystorm.net> wrote:
Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s.
I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Hi,
currently that is possible only with a small workaround: using a rewrite rule to modify the $TAGS field of the message.
Something like: rewrite r_rewrite_tag1 { set("$TAGS,tag1", value("TAGS"));};
No, this wouldn't work. $TAGS is read-only right now.
Then a log statement would look like:
log { source(s_local); filter(f_a); r_rewrite_tag1; destination(d_a); };
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
You can probably combine the filter and the rewrite into a single conditional rewrite rule to simplify the log statement.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... Would it not be enought to use the flags(final) rule on log statements, perhaps using embedded log {} statements?
This is equivalent to your configuration:
filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); flags(final); };
filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); flags(final); };
log { source(s_local); destination (d_c); }; Hrm, missed this reply somehow. Anyway, yes that would work for that example configuration. But it doesnt work so cleanly on more complex configurations.
filter f_prog1 { program("prog1"); }; log { source(s_local); filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; filter f_prog2 { program("prog2"}; }; log { source(s_local); filter(f_prog2); destination(d_prog2); tags("tag_prog"); };
filter f_crit { level(crit); }; log { source(s_local); filter(f_crit); destination(d_crit); tags("tag_crit"); }; filter f_notcrit { not tags("tag_crit"); }; log { source(s_local); filter(f_notcrit); destination(d_notcrit); };
filter f_other { not tags("tag_prog"); }; log { source(s_local); filter(f_other); destination(d_other); };
That is a bit of a bizarre config, but was just an example. You could probably work some combination of fallback and final plus reordering to get it working, but you end up with really hard to read configs. Plus if you ever have to go and add a new log statement in, if you dont add it in the right place it might not work or it might end up breaking the fallback/final rules. It just becomes very difficult to configure.
I think I do understand what you need, although I think it needs a
different solution.
For syslog-ng, a log {} statement is a one-way street, changes on a log statement would not affect the next one. Each fork along the log processing tree is independent, changes on one should not affect the other.
However I'm working on a change on the configuration model that would probably solve your problem too.
This is called "junctions" in the lack of a better name.
Right now, syslog-ng is sending messages along a message processing tree, e.g.:
source ---> log stmt1(filter, parser, rewrite, destination) | ---> log stmt2(filter, parser, rewrite, destination) | ---> log stmt3(filter, parser, rewrite, destination)
This is equivalent to this configuration:
log { source(src); filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { source(src); filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { source(src); filter(f3); parser(p3); rewrite(r3); destination(d3); };
Also, a log statement can add further forks, so the structure is a tree.
source ---> log stmt(filter, parser, rewrite) \ -> log stmt1(filter, parser, rewrite, destination) | -> log stmt2(filter, parser, rewrite, destination) | -> log stmt3(filter, parser, rewrite, destination)
Which is equivalent to this one:
log { source(src); filter(f); parser(p); rewrite(r); log { filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { filter(f3); parser(p3); rewrite(r3); destination(d3); }; };
Each of the forks is independent from the neighbouring ones, so any changes propagate only further down the tree, towards the leaves (e.g. destinations).
Even in the current syslog-ng versions, further forks can be introduced by adding another layer of log { } statements.
syslog-ng doesn't support cycles within this graph, e.g. once a fork is made, those become independent chains of processing, and any changes that happen on these branches are lost to the rest of the configruation.
The junction model changes that and allows joining branches.
log { source(src); junction { log { filter(f1); parser(p1); rewrite(r1); destination(d1); }; log { filter(f2); parser(p2); rewrite(r2); destination(d2); }; log { filter(f3); parser(p3); rewrite(r3); destination(d3); }; }; destination(dall); };
This config will result in "dall" receiving all the messages that fall off the three branches above.
This can be used for interesting things like:
- doing db-parser() based information extraction in one case
- and doing csv-parser() in another.
junction { log { filter(f_apache_msg); parser(p_apache_parser); flags(final); }; log { parser(p_dbparser()); }; };
This junction can be pasted into a log statement and the end results of the two branches can be used for further processing.
For your usecase, the solution is to create such a junction that categorizes messages using filters, attaches tags and such things. And once done, destinations are hooked to the end result:
filter f_prog1 { program("prog1"); }; filter f_prog2 { program("prog2"}; }; filter f_crit { level(crit); }; filter f_notcrit { not tags("tag_crit"); }; filter f_other { not tags("tag_prog"); };
log { source(s_local); junction { log { filter(f_prog1); destination(d_prog1); tags("tag_prog"); }; log { filter(f_prog2); destination(d_prog2); tags("tag_prog"); }; log { filter(f_crit); destination(d_crit); tags("tag_crit"); }; }; # from this point, tags are properly set junction { log { filter(f_notcrit); destination(d_notcrit); }; log { filter(f_other); destination(d_other); }; } };
One thing to note though, is that multiple branches that emerge from a junction point _each_ receive a copy of a message, so unless the filters are completely independent, multiple messages will fall off the first junction.
There are other interesting stuff to do with junctions: it'll be possible to associate rewrite/parser statements with sources and use it as a source.
For example:
source s_apache { log { source { file("/var/log/apache2/access.log"); }; parser { csv-parser(...); }; }; };
log { source(s_apache); destination(...); };
And voila, the s_apache source will emit structured information, instead of mere log lines.
I have this working in a local git tree of mine, but I didn't like the implementation, we're talking about code that have been in syslog-ng in more than a decade. It badly needs some refactoring. I already did that too, but the refactored version doesn't work yet.
This feature is the most important item on my 3.4 roadmap, so once complete, I'd publish 3.4 alpha, and since much less changes have happened in 3.4 than in 3.3, I hope to publish a final version before the summer.
At least that's the plan.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
AFAIK, running a CSV parser on a field extracted by patterndb should be possible even now, you just have to specify a template for the csv parser so that it runs on the specified field, and not on MSG. (Although in 3.3.3 this seems to be broken, but this will be hopefully corrected.)
Ok, but what about Patterndb on macros?
On Tuesday, January 10, 2012 18:54 CET, Martin Holste <mcholste@gmail.com> wrote:
AFAIK, running a CSV parser on a field extracted by patterndb should be possible even now, you just have to specify a template for the csv parser so that it runs on the specified field, and not on MSG. (Although in 3.3.3 this seems to be broken, but this will be hopefully corrected.)
Ok, but what about Patterndb on macros? That's not possible yet, but Bazsi mentioned that he wants to do it (or something like that) for 3.4.
Robert
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
Fekete Róbert
-
Martin Holste
-
Patrick Hemmer