Bazsi's blog: syslog-ng correllation updated
syslog-ng correllation updated I’m trying to push syslog-ng 3.2beta1 out on the door, but as I was writing the NEWS entry I had to realize that the latest state of the patterndb correllation functions are undocumented so far. So here goes a blog post which tries to summarize how it works, so that I can include it in the NEWS entry :) My previous post on the topic used a syntax that included explicit “store” and “join” elements, but I’ve decided to drop those, as they stood in the way for some more juicy functionality. What remained is that the correllation is still focused around a “correllation state”, or as I’ve called internally a “context”. A context consists of a series of log messages related to each other in some way. As new messages come in, they may be associated with a context (e.g. added to the context). Also, when an incoming message is identified it can trigger some actions to be performed. And these actions can use all the information that was stored previously in the context. Let’s see how this work out with concrete examples: each rule in the patterndb has a “context-id” attribute telling db-parser() which context the given message should be associated with. This example covers an SSH login message: <rule id=”…” context-id=”$HOST:$PROGRAM:$PID” context-timeout=”86400″ context-scope=”global”> <patterns> <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern> </patterns> … </rule> Since multiple rules can reference the same context, multiple different kind of messages may be added into the same context as a result. E.g. the logout event looks like this: <rule context-id=”$HOST:$PROGRAM:$PID” context-timeout=”0″ context-scope=”global”> <patterns> <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:usracct.username:@</pattern> </patterns> … </rule> As you can see a “session” is identified using the triplet ($HOST, $PROGRAM, $PID) and these two rules correllate the login/logout events into the same context, which means that you can create a derived event that contains information from both of them. Please note that it is fairly common that messages only need to be correllated if they originate from the same host: e.g. this SSH login message needs only to be correllated to its logout counterpart if they both originate from the same host. In the previous example this was achieved using explicit macros in the context-id attribute, however since this is quite often the case, this was worked into a function of its own right: each rule can have a context-scope attribute: <rule id=”…” context-scope=”process” context-id=”ssh-login” context-timeout=”5″> </rule> The context-scope tells syslog-ng which messages need to be considered when looking for correllations: * process: only consider messages that have matching $HOST, $PROGRAM and $PID values * program: only consider messages that have matching $HOST and $PROGRAM values * host: only consider messages that have matching $HOST values * global: any kind of message is fine The default is to use “process”, which means that if it is true that the same process is emitting all the messages that you want to correllate, then you don’t need to use a variable part in your context-id attribute. But it is also important to know that it is way faster to specify the scope this way than it’d be to add all relevant macros to your context-id attribute. So far so good, we have all the functions that we used to have with the previous versions of the functionality. But I mentioned something about “actions” to be performed. Until now a patterndb rule basically only identified the incoming message, possibly associated tags and name-value pairs, but didn’t perform anything else. This is being changed: one or more actions can be associated with a patterndb rule in order to make it possible to react to more complex situations. Here’s an example action: <action rate=”1/86400″> <message> <values> <value name=”MESSAGE”>a patterndb rule matched</value> </values> </message> </action> Right now the only real response to a message is to generate another message, but this allows us to do a couple of powerful transformations, especially with the following options that you can specify for an action tag: * condition: specifies a syslog-ng filter expression that needs to be matched in order to really perform the action. It is evaluated on the current message that matched the rule. * rate: <num>/<period> specifies how much messages are to be generated (num), in the specified time period (period). Excess messages are dropped. For example: “1/60″ allows 1 message / minute. Rates apply to the given scope for the given rule/action. E.g. context-scope=”host”, rate=”1/60″ means that one message gets generated for _each host_ per minute. * trigger: specifies when to execute the action, there are two possible triggers right now: * match: execute immediately once the rule matches * timeout: execute when the correllation timer expires I’d like to highlight two things: * it is possible to react to the expiration of a correllation timer (e.g. trigger=”timeout”) * it is possible to generate a message only in case a given condition is met (e.g. “$PID” == “”) Right now new messages are posted to the internal() driver. This is not the way I wanted it to be, but doing my original plan would require an enormous refactorization of the code, and it is too late for that too happen. My original idea was to let the db-parser() emit multiple messages, but since the current state of affairs in syslog-ng assumes that only sources generate messages, that needs a lot of work. But hey, we need something to do for syslog-ng 3.3, right?
I’ve got a Windows syslog client (from Q1 Labs) that wants to send multiple syslog messages within a single tcp packet to syslog-ng. The messages file on the syslog-ng side looks like this (Note the “^M<13>” separating the individual messages): [user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58^M<13>Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58^M<13>Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58^M Is it possible to configure syslog-ng to separate the messages out into individual ones like these? [user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58 [user] [notice] Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58 [user] [notice] Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58 I am using the syslog-ng ose client version 3.0.3. Thanks. Steve ------------- Steve Lee Technical Operations Center University Technology Services Emory University ------------- This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).
Hello Steve, The client should be sending Line Feed '\n' or ^J not Carriage Return '\r' or ^M. That's what the TCP Syslog RFC requires. So if you can get that problem fixed it might start working. You might want to look through the packet dumps to verify if the upstream is sending '\n' or '\r' or '\r\n'. Regards, Matthew Hall. On Mon, Oct 11, 2010 at 05:48:53PM -0400, Lee, Steve wrote:
I’ve got a Windows syslog client (from Q1 Labs) that wants to send multiple syslog messages within a single tcp packet to syslog-ng. The messages file on the syslog-ng side looks like this (Note the “^M<13>” separating the individual messages):
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58^M<13>Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58^M<13>Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58^M
Is it possible to configure syslog-ng to separate the messages out into individual ones like these?
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58 [user] [notice] Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58 [user] [notice] Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58
I am using the syslog-ng ose client version 3.0.3.
Thanks.
Steve
------------- Steve Lee Technical Operations Center University Technology Services Emory University -------------
This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited.
If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
What you might try is to create a filter that takes all incoming data on the tcp socket, replaces ^M with \n, and then pipes it back into another source driver (socket, pipe, whatever) for syslog-ng to process again, but without the filter expression (^M is probably \r as thats what most editors will display \r as). I'm not sure if that'll work, but I think it should. -Patrick Sent: Mon Oct 11 2010 15:48:53 GMT-0600 (Mountain Daylight Time) From: Lee, Steve <steve.lee@emory.edu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multiple syslog messages in one tcp packet
I’ve got a Windows syslog client (from Q1 Labs) that wants to send multiple syslog messages within a single tcp packet to syslog-ng. The messages file on the syslog-ng side looks like this (Note the “^M<13>” separating the individual messages):
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58^M<13>Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58^M<13>Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58^M
Is it possible to configure syslog-ng to separate the messages out into individual ones like these?
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58 [user] [notice] Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58 [user] [notice] Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58
I am using the syslog-ng ose client version 3.0.3.
Thanks.
Steve
------------- Steve Lee Technical Operations Center University Technology Services Emory University -------------
This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited.
If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
How do you create a filter for ^M and other control characters? Matthew. On Mon, Oct 11, 2010 at 04:27:59PM -0600, Patrick H. wrote:
What you might try is to create a filter that takes all incoming data on the tcp socket, replaces ^M with \n, and then pipes it back into another source driver (socket, pipe, whatever) for syslog-ng to process again, but without the filter expression (^M is probably \r as thats what most editors will display \r as). I'm not sure if that'll work, but I think it should.
-Patrick
Sent: Mon Oct 11 2010 15:48:53 GMT-0600 (Mountain Daylight Time) From: Lee, Steve <steve.lee@emory.edu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multiple syslog messages in one tcp packet
I’ve got a Windows syslog client (from Q1 Labs) that wants to send multiple syslog messages within a single tcp packet to syslog-ng. The messages file on the syslog-ng side looks like this (Note the “^M<13>” separating the individual messages):
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58^M<13>Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58^M<13>Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58^M
Is it possible to configure syslog-ng to separate the messages out into individual ones like these?
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58 [user] [notice] Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58 [user] [notice] Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58
I am using the syslog-ng ose client version 3.0.3.
Thanks.
Steve
------------- Steve Lee Technical Operations Center University Technology Services Emory University -------------
This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited.
If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
rewrite r_fixmsg { subst("\r","\n",value("MESSAGE") type("string") flags("global")); }; I dont know if syslog-ng will translate the \r or not. If not try inserting a raw \r char in there (this is all assuming ^M really is \r). -Patrick Sent: Mon Oct 11 2010 16:32:22 GMT-0600 (Mountain Daylight Time) From: Matthew Hall <mhall@mhcomputing.net> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multiple syslog messages in one tcp packet
How do you create a filter for ^M and other control characters?
Matthew.
On Mon, Oct 11, 2010 at 04:27:59PM -0600, Patrick H. wrote:
What you might try is to create a filter that takes all incoming data on the tcp socket, replaces ^M with \n, and then pipes it back into another source driver (socket, pipe, whatever) for syslog-ng to process again, but without the filter expression (^M is probably \r as thats what most editors will display \r as). I'm not sure if that'll work, but I think it should.
-Patrick
Sent: Mon Oct 11 2010 15:48:53 GMT-0600 (Mountain Daylight Time) From: Lee, Steve <steve.lee@emory.edu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multiple syslog messages in one tcp packet
I’ve got a Windows syslog client (from Q1 Labs) that wants to send multiple syslog messages within a single tcp packet to syslog-ng. The messages file on the syslog-ng side looks like this (Note the “^M<13>” separating the individual messages):
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58^M<13>Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58^M<13>Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58^M
Is it possible to configure syslog-ng to separate the messages out into individual ones like these?
[user] [notice] Oct 11 16:25:05 10.40.3.16 10.40.3.16 ation_logfile.txt Payload=server.Emory.Edu, The Operations Manager agent processes are using too much processor time SEVERITY:2 STATE: New; Custom Oct 11 15:15:58 [user] [notice] Oct 11 15:17:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile=logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: New; StateCollection Oct 11 15:17:58 [user] [notice] Oct 11 15:19:58 server2.emory.edu AgentDevice=FileForwarder AgentLogFile= logfile.txt Payload=Microsoft.SystemCenter.AgentWatchersGroup, Health Service Heartbeat Failure SEVERITY:2 STATE: Closed; StateCollection Oct 11 15:19:58
I am using the syslog-ng ose client version 3.0.3.
Thanks.
Steve
------------- Steve Lee Technical Operations Center University Technology Services Emory University -------------
This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited.
If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Mon, Oct 11, 2010 at 04:53:50PM -0600, Patrick H. wrote:
I dont know if syslog-ng will translate the \r or not. If not try inserting a raw \r char in there
(this is all assuming ^M really is \r).
I confirmed that before posting my original mails, in a table which explains which UNIX Ctrl-Letter corresponds to which C-escaped character. ^M and \r are one and the same.
-Patrick
On Mon, 2010-10-11 at 15:32 -0700, Matthew Hall wrote:
How do you create a filter for ^M and other control characters?
syslog-ng 3.0 & 3.1 supports C-like \r for ^M (ASCII 13) syslog-ng 3.2 also supports \x escapes like: "\x0d" -- Bazsi
Thanks for the examples, this helps. However, I do have a question. The best use I can think of for this is to correlate our email gateway logs, which currently spew about 20 log entries per email. I'd love for all of the data to be printed out in one line like you've demonstrated the action feature can accomplish. The problem that I foresee is that many log entries do not have $PID available, just $HOST and $PROGRAM, and that will not be unique enough. Our mail gateways have message ID's built into the log entry, but it would have to be parsed out with a pattern. Can this be done and still work within the system you've created? If so, can you show an example? On Mon, Oct 11, 2010 at 9:18 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
syslog-ng correllation updated
I’m trying to push syslog-ng 3.2beta1 out on the door, but as I was writing the NEWS entry I had to realize that the latest state of the patterndb correllation functions are undocumented so far. So here goes a blog post which tries to summarize how it works, so that I can include it in the NEWS entry :)
My previous post on the topic used a syntax that included explicit “store” and “join” elements, but I’ve decided to drop those, as they stood in the way for some more juicy functionality. What remained is that the correllation is still focused around a “correllation state”, or as I’ve called internally a “context”.
A context consists of a series of log messages related to each other in some way. As new messages come in, they may be associated with a context (e.g. added to the context). Also, when an incoming message is identified it can trigger some actions to be performed. And these actions can use all the information that was stored previously in the context.
Let’s see how this work out with concrete examples: each rule in the patterndb has a “context-id” attribute telling db-parser() which context the given message should be associated with.
This example covers an SSH login message:
<rule id=”…” context-id=”$HOST:$PROGRAM:$PID” context-timeout=”86400″ context-scope=”global”> <patterns> <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern> </patterns> … </rule>
Since multiple rules can reference the same context, multiple different kind of messages may be added into the same context as a result. E.g. the logout event looks like this:
<rule context-id=”$HOST:$PROGRAM:$PID” context-timeout=”0″ context-scope=”global”> <patterns> <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:usracct.username:@</pattern> </patterns> … </rule>
As you can see a “session” is identified using the triplet ($HOST, $PROGRAM, $PID) and these two rules correllate the login/logout events into the same context, which means that you can create a derived event that contains information from both of them.
Please note that it is fairly common that messages only need to be correllated if they originate from the same host: e.g. this SSH login message needs only to be correllated to its logout counterpart if they both originate from the same host. In the previous example this was achieved using explicit macros in the context-id attribute, however since this is quite often the case, this was worked into a function of its own right: each rule can have a context-scope attribute:
<rule id=”…” context-scope=”process” context-id=”ssh-login” context-timeout=”5″> </rule>
The context-scope tells syslog-ng which messages need to be considered when looking for correllations:
process: only consider messages that have matching $HOST, $PROGRAM and $PID values program: only consider messages that have matching $HOST and $PROGRAM values host: only consider messages that have matching $HOST values global: any kind of message is fine
The default is to use “process”, which means that if it is true that the same process is emitting all the messages that you want to correllate, then you don’t need to use a variable part in your context-id attribute. But it is also important to know that it is way faster to specify the scope this way than it’d be to add all relevant macros to your context-id attribute.
So far so good, we have all the functions that we used to have with the previous versions of the functionality. But I mentioned something about “actions” to be performed. Until now a patterndb rule basically only identified the incoming message, possibly associated tags and name-value pairs, but didn’t perform anything else. This is being changed: one or more actions can be associated with a patterndb rule in order to make it possible to react to more complex situations.
Here’s an example action:
<action rate=”1/86400″> <message> <values> <value name=”MESSAGE”>a patterndb rule matched</value> </values> </message> </action>
Right now the only real response to a message is to generate another message, but this allows us to do a couple of powerful transformations, especially with the following options that you can specify for an action tag:
condition: specifies a syslog-ng filter expression that needs to be matched in order to really perform the action. It is evaluated on the current message that matched the rule. rate: <num>/<period> specifies how much messages are to be generated (num), in the specified time period (period). Excess messages are dropped. For example: “1/60″ allows 1 message / minute. Rates apply to the given scope for the given rule/action. E.g. context-scope=”host”, rate=”1/60″ means that one message gets generated for _each host_ per minute. trigger: specifies when to execute the action, there are two possible triggers right now:
match: execute immediately once the rule matches timeout: execute when the correllation timer expires
I’d like to highlight two things:
it is possible to react to the expiration of a correllation timer (e.g. trigger=”timeout”) it is possible to generate a message only in case a given condition is met (e.g. “$PID” == “”)
Right now new messages are posted to the internal() driver. This is not the way I wanted it to be, but doing my original plan would require an enormous refactorization of the code, and it is too late for that too happen. My original idea was to let the db-parser() emit multiple messages, but since the current state of affairs in syslog-ng assumes that only sources generate messages, that needs a lot of work. But hey, we need something to do for syslog-ng 3.3, right?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Tue, 2010-10-12 at 09:50 -0500, Martin Holste wrote:
Thanks for the examples, this helps. However, I do have a question. The best use I can think of for this is to correlate our email gateway logs, which currently spew about 20 log entries per email. I'd love for all of the data to be printed out in one line like you've demonstrated the action feature can accomplish. The problem that I foresee is that many log entries do not have $PID available, just $HOST and $PROGRAM, and that will not be unique enough. Our mail gateways have message ID's built into the log entry, but it would have to be parsed out with a pattern. Can this be done and still work within the system you've created? If so, can you show an example?
Yes, sure. context-id attribute can contain values parsed outside the message. e.g. if you have parsed out the queue-id from the log that groups the log messages, you can use: context-scope="host" context-id="mail-correllation:${queue_id}" Assuming that even the $PROGRAM value varies between lines. If that stays the same, you could probably use context-scope="program". -- Bazsi
Perfect, thanks! On Fri, Oct 15, 2010 at 3:14 PM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2010-10-12 at 09:50 -0500, Martin Holste wrote:
Thanks for the examples, this helps. However, I do have a question. The best use I can think of for this is to correlate our email gateway logs, which currently spew about 20 log entries per email. I'd love for all of the data to be printed out in one line like you've demonstrated the action feature can accomplish. The problem that I foresee is that many log entries do not have $PID available, just $HOST and $PROGRAM, and that will not be unique enough. Our mail gateways have message ID's built into the log entry, but it would have to be parsed out with a pattern. Can this be done and still work within the system you've created? If so, can you show an example?
Yes, sure. context-id attribute can contain values parsed outside the message.
e.g. if you have parsed out the queue-id from the log that groups the log messages, you can use:
context-scope="host" context-id="mail-correllation:${queue_id}"
Assuming that even the $PROGRAM value varies between lines. If that stays the same, you could probably use context-scope="program".
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (5)
-
Balazs Scheidler
-
Lee, Steve
-
Martin Holste
-
Matthew Hall
-
Patrick H.