using correlation to filter out some messages ?
Hello. I've had a look at message correlation in the 3.2 manual, but it's still a bit obscure for me to understand if it could allow me to achieve filtering of pairs of log messages, as described here: https://lists.balabit.hu/pipermail/syslog-ng/2010-January/013857.html -- BOFH excuse #360: Your parity check is overdrawn and you're out of cache.
Hi, Sorry for not answering any sooner, but as this post shows I usually process my backlog every now-and-then. Matthew and Martin usually helps a lot of people encountering this stuff, but certainly they may not be able to help everyone :) The question itself is great, this is how we can cover with newer-and-newer problems with syslog-ng functionality. Your use-case is not simply as db-parser() was never meant to be used to drop messages. So, here are some ideas how to cope with your problem. 1) you should create a very specific rule for the first message (e.g. it should contain the IP address as literal string, instead of a parser). This will make the db-parser() engine prefer this rule over the one with a parser reference in it (assuming of course that there's another similar rule) In this first rule, use the following correllation options: context-scope="program" context-id="lbprobe_${slapd.connection_id}" context-timeout="5" # or the time you'd assume you'll see the second message Attach a tag to the message, which you can use to drop the message later on using a filter (e.g. tag "dropthis"). Then, create a timeout action (e.g. <action trigger="timeout">, which should generate a message identical to the matched one, but this time without the tag which indicates that the message should be dropped. 2) create a rule for the 2nd message the correllation options of both rules should be the same as the 1st rule, except that the timeout can be 0. Attach a tag to the message, which you can use to drop the message later on using a filter (e.g. tag "dropthis"). Then, create a matched action (e.g. <action trigger="match">), with a filter condition attached (e.g. condition="filter-expr"). This condition should check that there was is no other message in the correllation state (for example: ${MESSAGE}@1 == ""). The action will be fired, when the closed message is coming from a non-probe client. In this case the content of the action should just repeat the original message. (e.g. it should be emitted) without the use of the dropthis tag. I know this is complicated. It could be made simpler a number of ways. 1) possibly an explicit "drop" action for the main message (makes the explicit tag, plus filtering in the configuration unnecessary). This should be trivial to do. 2) possibly build some stack based mini-language into the db-parser() engine, with stuff like push/pop/apply a surprising number of operations can be implemented easily. I'll give a thought to this. On Wed, 2010-12-29 at 15:58 +0100, Guillaume Rousse wrote:
Hello.
I've had a look at message correlation in the 3.2 manual, but it's still a bit obscure for me to understand if it could allow me to achieve filtering of pairs of log messages, as described here: https://lists.balabit.hu/pipermail/syslog-ng/2010-January/013857.html
-- Bazsi
Sorry for not answering any sooner, but as this post shows I usually process my backlog every now-and-then. Matthew and Martin usually helps a lot of people encountering this stuff, but certainly they may not be able to help everyone :)
Ha, this one is well beyond me--I'm also struggling with the correlation stuff.
Then, create a timeout action (e.g. <action trigger="timeout">, which should generate a message identical to the matched one, but this time without the tag which indicates that the message should be dropped.
This is where you're losing me. Can you provide a working example of the whole thing?
On Thu, 2011-01-13 at 19:44 -0600, Martin Holste wrote:
Sorry for not answering any sooner, but as this post shows I usually process my backlog every now-and-then. Matthew and Martin usually helps a lot of people encountering this stuff, but certainly they may not be able to help everyone :)
Ha, this one is well beyond me--I'm also struggling with the correlation stuff.
Then, create a timeout action (e.g. <action trigger="timeout">, which should generate a message identical to the matched one, but this time without the tag which indicates that the message should be dropped.
This is where you're losing me. Can you provide a working example of the whole thing?
something like this: <action trigger="timeout"> <message> <values> <!-- HOST, PROGRAM, PID are filled automatically based on the current message --> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action> while the body of the rule contains this: <tags> <tag>dropthis</tag> <tags> This tag can then be used to drop messages by a syslog-ng filter, something like this: filter f_drop { tags("dropthis"); }; parser p_db { db-parser(); }; log { parser(p_db); log { filter(f_drop); flags(final); }; log { destination(normal-destination); }; }; Or an equivalent. When db-parser() matches a rule, it does the following: 1) process the current message (matching the rule) by adding name-value pairs and tags as described by the rule 2) possibly create new messages based on actions associated with the rule. The body of the rule (e.g. the contents inside the main <rule></rule> tag) are applied to the current message. Actions are processed when their specified trigger is hit, e.g. either on timeout (trigger="timeout") or at the same time the rule is matched (trigger="match") This way a single rule match can in theory result in several messages to be emitted. 1) the message coming in and triggering the rule 2) all the messages generated by associated actions. -- Bazsi
Ok, I'm with you so far on this. So when this is run, there will be two messages, one which has a tag of "dropthis" and gets filtered out by the f_drop, and the other proceeds normally? If that's correct, then I don't understand the strategy but I do understand the configuration. On Fri, Jan 14, 2011 at 5:44 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2011-01-13 at 19:44 -0600, Martin Holste wrote:
Sorry for not answering any sooner, but as this post shows I usually process my backlog every now-and-then. Matthew and Martin usually helps a lot of people encountering this stuff, but certainly they may not be able to help everyone :)
Ha, this one is well beyond me--I'm also struggling with the correlation stuff.
Then, create a timeout action (e.g. <action trigger="timeout">, which should generate a message identical to the matched one, but this time without the tag which indicates that the message should be dropped.
This is where you're losing me. Can you provide a working example of the whole thing?
something like this:
<action trigger="timeout"> <message> <values> <!-- HOST, PROGRAM, PID are filled automatically based on the current message --> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action>
while the body of the rule contains this:
<tags> <tag>dropthis</tag> <tags>
This tag can then be used to drop messages by a syslog-ng filter, something like this:
filter f_drop { tags("dropthis"); };
parser p_db { db-parser(); };
log { parser(p_db); log { filter(f_drop); flags(final); }; log { destination(normal-destination); }; };
Or an equivalent.
When db-parser() matches a rule, it does the following: 1) process the current message (matching the rule) by adding name-value pairs and tags as described by the rule 2) possibly create new messages based on actions associated with the rule.
The body of the rule (e.g. the contents inside the main <rule></rule> tag) are applied to the current message. Actions are processed when their specified trigger is hit, e.g. either on timeout (trigger="timeout") or at the same time the rule is matched (trigger="match")
This way a single rule match can in theory result in several messages to be emitted. 1) the message coming in and triggering the rule 2) all the messages generated by associated actions.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Fri, 2011-01-14 at 13:18 -0600, Martin Holste wrote:
Ok, I'm with you so far on this. So when this is run, there will be two messages, one which has a tag of "dropthis" and gets filtered out by the f_drop, and the other proceeds normally? If that's correct, then I don't understand the strategy but I do understand the configuration.
The aim was to drop the lines unless proved to be part of a real session. It is part of a real session, if: 1) the closure message comes 5 seconds after the first one (that's the correllation timeout) 2) or if the connection request came from a different host from the load balancer. In effect, it'll hold the connection open/close messages until either of the above conditions will prove that the message can be written out. But in order to do this, we need to drop the 'raw' log message first and reproduce it once we know we have to.
On Fri, Jan 14, 2011 at 5:44 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2011-01-13 at 19:44 -0600, Martin Holste wrote:
Sorry for not answering any sooner, but as this post shows I usually process my backlog every now-and-then. Matthew and Martin usually helps a lot of people encountering this stuff, but certainly they may not be able to help everyone :)
Ha, this one is well beyond me--I'm also struggling with the correlation stuff.
Then, create a timeout action (e.g. <action trigger="timeout">, which should generate a message identical to the matched one, but this time without the tag which indicates that the message should be dropped.
This is where you're losing me. Can you provide a working example of the whole thing?
something like this:
<action trigger="timeout"> <message> <values> <!-- HOST, PROGRAM, PID are filled automatically based on the current message --> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action>
while the body of the rule contains this:
<tags> <tag>dropthis</tag> <tags>
This tag can then be used to drop messages by a syslog-ng filter, something like this:
filter f_drop { tags("dropthis"); };
parser p_db { db-parser(); };
log { parser(p_db); log { filter(f_drop); flags(final); }; log { destination(normal-destination); }; };
Or an equivalent.
When db-parser() matches a rule, it does the following: 1) process the current message (matching the rule) by adding name-value pairs and tags as described by the rule 2) possibly create new messages based on actions associated with the rule.
The body of the rule (e.g. the contents inside the main <rule></rule> tag) are applied to the current message. Actions are processed when their specified trigger is hit, e.g. either on timeout (trigger="timeout") or at the same time the rule is matched (trigger="match")
This way a single rule match can in theory result in several messages to be emitted. 1) the message coming in and triggering the rule 2) all the messages generated by associated actions.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
Ok, so the first time the raw message is seen, it is not forwarded through normally then and has effectively been put in limbo, not duplicated, right? On Sat, Jan 15, 2011 at 5:41 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Fri, 2011-01-14 at 13:18 -0600, Martin Holste wrote:
Ok, I'm with you so far on this. So when this is run, there will be two messages, one which has a tag of "dropthis" and gets filtered out by the f_drop, and the other proceeds normally? If that's correct, then I don't understand the strategy but I do understand the configuration.
The aim was to drop the lines unless proved to be part of a real session. It is part of a real session, if:
1) the closure message comes 5 seconds after the first one (that's the correllation timeout) 2) or if the connection request came from a different host from the load balancer.
In effect, it'll hold the connection open/close messages until either of the above conditions will prove that the message can be written out. But in order to do this, we need to drop the 'raw' log message first and reproduce it once we know we have to.
On Fri, Jan 14, 2011 at 5:44 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2011-01-13 at 19:44 -0600, Martin Holste wrote:
Sorry for not answering any sooner, but as this post shows I usually process my backlog every now-and-then. Matthew and Martin usually helps a lot of people encountering this stuff, but certainly they may not be able to help everyone :)
Ha, this one is well beyond me--I'm also struggling with the correlation stuff.
Then, create a timeout action (e.g. <action trigger="timeout">, which should generate a message identical to the matched one, but this time without the tag which indicates that the message should be dropped.
This is where you're losing me. Can you provide a working example of the whole thing?
something like this:
<action trigger="timeout"> <message> <values> <!-- HOST, PROGRAM, PID are filled automatically based on the current message --> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action>
while the body of the rule contains this:
<tags> <tag>dropthis</tag> <tags>
This tag can then be used to drop messages by a syslog-ng filter, something like this:
filter f_drop { tags("dropthis"); };
parser p_db { db-parser(); };
log { parser(p_db); log { filter(f_drop); flags(final); }; log { destination(normal-destination); }; };
Or an equivalent.
When db-parser() matches a rule, it does the following: 1) process the current message (matching the rule) by adding name-value pairs and tags as described by the rule 2) possibly create new messages based on actions associated with the rule.
The body of the rule (e.g. the contents inside the main <rule></rule> tag) are applied to the current message. Actions are processed when their specified trigger is hit, e.g. either on timeout (trigger="timeout") or at the same time the rule is matched (trigger="match")
This way a single rule match can in theory result in several messages to be emitted. 1) the message coming in and triggering the rule 2) all the messages generated by associated actions.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Sat, 2011-01-15 at 11:16 -0600, Martin Holste wrote:
Ok, so the first time the raw message is seen, it is not forwarded through normally then and has effectively been put in limbo, not duplicated, right?
yes. -- Bazsi
Ok, got it. So here's the patterndb correlation challenge I'm working on: Cisco's Ironport is an email/spam filter that produces verbose logging. The logs have two ID's of interest, the ICID which goes with the connection and the MID which goes with each individual email. Usually, a single email will produce between 15 and 30 log messages. I'm looking to use correlation to produce one "meta" message which has the most important details from all messages so that when searching logs, you don't have to do follow-up searches to find all of the info you're looking for. Here's what I've got so far: <ruleset> <rules> <rule class="10" id="10" context-id="ironport-icid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: New SMTP ICID @NUMBER:icid:@ interface @ESTRING:interface_name: @(@IPv4:interface_ip:@) address @IPv4:sender_ip:@ reverse dns host @ESTRING:sender_dns: @verified yes</pattern> </patterns> <examples> <example> <test_message program="ironport_mail_logs">Info: New SMTP ICID 696117306 interface InternalNet (x.x.88.227) address 10.x.x.x reverse dns host xx verified yes</test_message> <test_value name="icid">696117306</test_value> <test_value name="interface_name">InternalNet</test_value> <test_value name="interface_ip">x.x.88.227</test_value> <test_value name="sender_ip">10.x.x.x</test_value> <test_value name="sender_dns">xx</test_value> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-icid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: ICID @NUMBER:icid:@ close</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">IronPort message complete: icid: $icid, mid: $mid, interface_name: $interface_name, interface_ip: $interface_ip, sender_ip: $sender_ip, sender_dns: $sender_dns</value> </values> </message> </action> </actions> <examples> <example> <test_message program="ironport_mail_logs">Info: ICID 696117306 close</test_message> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Start MID @NUMBER:mid:@ ICID @NUMBER:icid:@</pattern> </patterns> <values> <value name="icid">$icid</value> </values> <examples> <example> <test_message program="ironport_mail_logs">Info: Start MID 144753300 ICID 696117306</test_message> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Message finished MID @NUMBER:mid:@ done</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">IronPort message complete: icid: $icid@2, mid: $mid</value> </values> </message> </action> </actions> <examples> <example> <test_message program="ironport_mail_logs">Info: Message finished MID 144753300 done</test_message> </example> </examples> </rule> </rules> </ruleset> So how do I use tagging to propagate the information contained from previous messages? Previously, you had suggested I use grep, but that's proving to be tough for me to implement. On Sun, Jan 16, 2011 at 9:06 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Sat, 2011-01-15 at 11:16 -0600, Martin Holste wrote:
Ok, so the first time the raw message is seen, it is not forwarded through normally then and has effectively been put in limbo, not duplicated, right?
yes.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Sorry, I didn't have enough time to understand the problem, but I'll try harder in the near future, promise :) On Sun, 2011-01-16 at 11:54 -0600, Martin Holste wrote:
Ok, got it. So here's the patterndb correlation challenge I'm working on: Cisco's Ironport is an email/spam filter that produces verbose logging. The logs have two ID's of interest, the ICID which goes with the connection and the MID which goes with each individual email. Usually, a single email will produce between 15 and 30 log messages. I'm looking to use correlation to produce one "meta" message which has the most important details from all messages so that when searching logs, you don't have to do follow-up searches to find all of the info you're looking for. Here's what I've got so far:
<ruleset> <rules> <rule class="10" id="10" context-id="ironport-icid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: New SMTP ICID @NUMBER:icid:@ interface @ESTRING:interface_name: @(@IPv4:interface_ip:@) address @IPv4:sender_ip:@ reverse dns host @ESTRING:sender_dns: @verified yes</pattern> </patterns> <examples>
-- Bazsi
Le 15/01/2011 12:41, Balazs Scheidler a écrit :
The aim was to drop the lines unless proved to be part of a real session. It is part of a real session, if:
1) the closure message comes 5 seconds after the first one (that's the correllation timeout) 2) or if the connection request came from a different host from the load balancer.
In effect, it'll hold the connection open/close messages until either of the above conditions will prove that the message can be written out. But in order to do this, we need to drop the 'raw' log message first and reproduce it once we know we have to. Thanks for your explanations.
However, in my case the problem can be simplified, as the load-balancer (a cisco switch) never initiates LDAP connections itself, so the logic can be simplified to: - any initial connection message from the load balancer should be dropped - any connection closure message correlated with an initial connection message from the load balancer should be dropped too The following rules database should be enough for this. I tested it to match first and second messages with pdbtool. However, it still doesn't filter the closure messages, with the following configuration: filter f_ldap { facility(local4); }; filter f_slb_ldap_probe { tags("dropthis"); }; parser p_db { db-parser(); }; log { source(s_sys); parser(p_db); filter(f_ldap); filter(f_slb_ldap_probe); destination(d_drop); flags(final); }; What am I missing ? -- BOFH excuse #24: network packets travelling uphill (use a carrier pigeon)
Le 18/01/2011 13:23, Guillaume Rousse a écrit :
What am I missing ? Actually, it was a configuration issue on my side, it works perfectly now.
And I just realized I could almost achieve the same by filtering on the '(connection lost)' message which is always present for basic tcp probes (I could eventually have lost other unrelated messages, touch). Many thanks for your help. I guess this could be added as practical examples to the syslog-ng admin guide. -- BOFH excuse #311: transient bus protocol violation
Le 18/01/2011 15:24, Guillaume Rousse a écrit :
Le 18/01/2011 13:23, Guillaume Rousse a écrit :
What am I missing ? Actually, it was a configuration issue on my side, it works perfectly now.
And I just realized I could almost achieve the same by filtering on the '(connection lost)' message which is always present for basic tcp probes (I could eventually have lost other unrelated messages, touch). And I also realized than I am probably filtering all messages matching the second pattern (conn=@NUMBER:slapd_connection_id@ fd=@NUMBER@ closed (connection lost)), wether they are related to the load balancer or not, as the context-id attribute does not act like a filter.
As you suggested earlier, the following action in the second rule should ensure a message in an empty context is getting re-emited: <action trigger="match" condition="${MESSAGE}@1 == ''"> <message> <values> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action> -- BOFH excuse #412: Radial Telemetry Infiltration
Le 18/01/2011 18:13, Guillaume Rousse a écrit :
As you suggested earlier, the following action in the second rule should ensure a message in an empty context is getting re-emited: <action trigger="match" condition="${MESSAGE}@1 == ''"> <message> <values> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action> This one triggers an error:
Error parsing filter expression, syntax error, unexpected $undefined in <string> at line 1, column 2: ${ ^ ... Error parsing pattern database file; filename='/var/lib/syslog-ng/patterndb.xml', error='Error compiling conditional expression' I tried playing with other macros, such as ${HOST}, without success. Also, I guess the re-emited message should be the current one (${MSG}), not the previous one (${MSG}@1), for the value element content ? -- BOFH excuse #15: temporary routing anomaly
On Wed, 2011-01-19 at 10:58 +0100, Guillaume Rousse wrote:
Le 18/01/2011 18:13, Guillaume Rousse a écrit :
As you suggested earlier, the following action in the second rule should ensure a message in an empty context is getting re-emited: <action trigger="match" condition="${MESSAGE}@1 == ''"> <message> <values> <value name="MESSAGE">${MSG}@1</value> </values> </message> </action> This one triggers an error:
Error parsing filter expression, syntax error, unexpected $undefined in <string> at line 1, column 2:
${ ^ ... Error parsing pattern database file; filename='/var/lib/syslog-ng/patterndb.xml', error='Error compiling conditional expression'
I tried playing with other macros, such as ${HOST}, without success.
Also, I guess the re-emited message should be the current one (${MSG}), not the previous one (${MSG}@1), for the value element content ?
you should enclose the macro reference in quotes like this: condition="'${MESSAGE}@1' == ''" ^ ^ in a filter expression, all strings are assumed to be templates, and then you can use operators like you did. but macro references also need to be enclosed in quotes (either apostrophes or double quotes will work), this time it was easier to use apostrophes because the XML attribute used quotes. -- Bazsi
Le 24/01/2011 17:35, Balazs Scheidler a écrit :
you should enclose the macro reference in quotes like this:
condition="'${MESSAGE}@1' == ''" ^ ^
in a filter expression, all strings are assumed to be templates, and then you can use operators like you did. but macro references also need to be enclosed in quotes (either apostrophes or double quotes will work), this time it was easier to use apostrophes because the XML attribute used quotes. OK, this time syslog-ng doesn't choke, but the re-emited message is leaking to stdout (actually, to the console used to launch it, I just presume it's syslog-ng stdout), which is quite painful:
[root@avron1 ~]# service syslog-ng start Lancement de syslog-ng : [ OK ] [root@avron1 ~]# 2011 Jan 26 11:16:21 avron1 conn=1569812 fd=39 closed (connection lost) 2011 Jan 26 11:16:21 avron1 conn=1569813 fd=60 closed (connection lost) 2011 Jan 26 11:16:23 avron1 conn=1569814 fd=39 closed (connection lost) 2011 Jan 26 11:16:23 avron1 conn=1569815 fd=60 closed (connection lost) Morevoer, it also suggested the condition used doesn't work, as those messages shouldn't have been re-emited at all. I'm attaching patterndb and syslog-ng configuration related fragments. -- BOFH excuse #211: Lightning strikes.
Hi, How is the d_drop destination configured? As far as I know you can't use flags(final) in embedded log statements so the tagged messages would still reach d_ldap. You can simply change the filter to match on messages not having the 'dropthis' tag and use it in the embedded log section. Regards, Sandor On Wed, Jan 26, 2011 at 11:21 AM, Guillaume Rousse <guillomovitch@gmail.com> wrote:
Le 24/01/2011 17:35, Balazs Scheidler a écrit :
you should enclose the macro reference in quotes like this:
condition="'${MESSAGE}@1' == ''" ^ ^
in a filter expression, all strings are assumed to be templates, and then you can use operators like you did. but macro references also need to be enclosed in quotes (either apostrophes or double quotes will work), this time it was easier to use apostrophes because the XML attribute used quotes. OK, this time syslog-ng doesn't choke, but the re-emited message is leaking to stdout (actually, to the console used to launch it, I just presume it's syslog-ng stdout), which is quite painful:
[root@avron1 ~]# service syslog-ng start Lancement de syslog-ng : [ OK ] [root@avron1 ~]# 2011 Jan 26 11:16:21 avron1 conn=1569812 fd=39 closed (connection lost) 2011 Jan 26 11:16:21 avron1 conn=1569813 fd=60 closed (connection lost) 2011 Jan 26 11:16:23 avron1 conn=1569814 fd=39 closed (connection lost) 2011 Jan 26 11:16:23 avron1 conn=1569815 fd=60 closed (connection lost)
Morevoer, it also suggested the condition used doesn't work, as those messages shouldn't have been re-emited at all.
I'm attaching patterndb and syslog-ng configuration related fragments. -- BOFH excuse #211:
Lightning strikes.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2011-01-26 at 11:21 +0100, Guillaume Rousse wrote:
Le 24/01/2011 17:35, Balazs Scheidler a écrit :
you should enclose the macro reference in quotes like this:
condition="'${MESSAGE}@1' == ''" ^ ^
in a filter expression, all strings are assumed to be templates, and then you can use operators like you did. but macro references also need to be enclosed in quotes (either apostrophes or double quotes will work), this time it was easier to use apostrophes because the XML attribute used quotes. OK, this time syslog-ng doesn't choke, but the re-emited message is leaking to stdout (actually, to the console used to launch it, I just presume it's syslog-ng stdout), which is quite painful:
[root@avron1 ~]# service syslog-ng start Lancement de syslog-ng : [ OK ] [root@avron1 ~]# 2011 Jan 26 11:16:21 avron1 conn=1569812 fd=39 closed (connection lost) 2011 Jan 26 11:16:21 avron1 conn=1569813 fd=60 closed (connection lost) 2011 Jan 26 11:16:23 avron1 conn=1569814 fd=39 closed (connection lost) 2011 Jan 26 11:16:23 avron1 conn=1569815 fd=60 closed (connection lost)
Morevoer, it also suggested the condition used doesn't work, as those messages shouldn't have been re-emited at all.
I'm attaching patterndb and syslog-ng configuration related fragments.
hmm... reemitted messages are coming in the internal() source, and in case you don't have that in your configuration file anywhere, it might come out on stdout. -- Bazsi
Le 05/02/2011 14:27, Balazs Scheidler a écrit :
hmm... reemitted messages are coming in the internal() source, and in case you don't have that in your configuration file anywhere, it might come out on stdout. I do.
However, I found the issue reading the documentation about embedded log statements (big fat warning in 8.1.2 section): The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths. Just moving the flags(final) statement from second inner log bloc to the outer bloc was enough to fix this specific issue. May I suggest a slight rephrasing of the warning, tough ? final flags inside embedded log path do have an effect on other log path at the same depth. In this first case, the final flag prevent filtered message in the first inner log path to be passed to the second inner path, for instance: log { source(s_sys); log { filter(f_drop); destination(d_drop); flags(final); }; log { destination(d_ldap); }; }; -- BOFH excuse #372: Forced to support NT servers; sysadmins quit.
Hi Guillamue, Thanks for noticing it. We'll clarify the warning in the next doc release. Regards, Robert On 02/22/2011 03:35 PM, Guillaume Rousse wrote:
Le 05/02/2011 14:27, Balazs Scheidler a écrit :
hmm... reemitted messages are coming in the internal() source, and in case you don't have that in your configuration file anywhere, it might come out on stdout. I do.
However, I found the issue reading the documentation about embedded log statements (big fat warning in 8.1.2 section): The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.
Just moving the flags(final) statement from second inner log bloc to the outer bloc was enough to fix this specific issue.
May I suggest a slight rephrasing of the warning, tough ? final flags inside embedded log path do have an effect on other log path at the same depth.
In this first case, the final flag prevent filtered message in the first inner log path to be passed to the second inner path, for instance: log { source(s_sys);
log { filter(f_drop); destination(d_drop); flags(final); };
log { destination(d_ldap); }; };
participants (5)
-
Balazs Scheidler
-
Fekete Robert
-
Guillaume Rousse
-
Martin Holste
-
Sandor Geller