Correct Usage of Multiple 'pattern' Databases
Hello, I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases): log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; }; The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match. I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way? Really appreciate any help you might be able to offer. Thanks, -David
Hi, I guess the problem is that patterndb parsers were not thought to be used this way, and each parser sets the .classifier.class value based on its own rules, overwriting any previous values. Consequently, merging the patterndbs into a single file would definitely work. Another option could be a variant of your second idea: you use a single log statement, and embed the parsers into a junction, where each channel of the junction contains a filter (to process only the messages that the parser can parse), and one of the parsers. Something like: log { filter(f_auth); junction{ channel {filter (program(sshd)); parser("sshd");} channel {filter (program(sudo)); parser("sudo");} .... } } Robert On Friday, April 11, 2014 19:18 CEST, David Hauck <davidh@netacquire.com> wrote:
Hello,
I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases):
log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; };
The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match.
I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way?
Really appreciate any help you might be able to offer.
Thanks, -David ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Robert, On Friday, April 11, 2014 12:19 PM, syslog-ng-bounces@lists.balabit.hu wrote:
Hi,
I guess the problem is that patterndb parsers were not thought to be used this way, and each parser sets the .classifier.class value based on its own rules, overwriting any previous values. Consequently, merging the patterndbs into a single file would definitely work.
As it turns out all of the pattern DBs here (these are just the stock files from: https://github.com/balabit/syslog-ng-patterndb) define the same 'class' (so the results of .classifier.class should always be 'system' after running through each of the parsers, no?).
Another option could be a variant of your second idea: you use a single log statement, and embed the parsers into a junction, where each channel of the junction contains a filter (to process only the messages that the parser can parse), and one of the parsers. Something like: log { filter(f_auth); junction{ channel {filter (program(sshd)); parser("sshd");} channel {filter (program(sudo)); parser("sudo");} .... } }
Thanks, I've tried this and it does appear to address this problem (notwithstanding the fact I mention above, where each pattern file specifies the same .classifier.class value). I'm going to follow-up with another question related to boolean filters and embedded log statements since some of the follow-on filtering criteria now appears unexplainable. Thanks, -David
Robert
On Friday, April 11, 2014 19:18 CEST, David Hauck <davidh@netacquire.com> wrote:
Hello,
I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases):
log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; };
The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match.
I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way?
Really appreciate any help you might be able to offer.
Thanks, -David
__________________________________________________________ ____________
________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
__________________________________________________________ ____________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi David! If a log message does not match any pattern for a parser, syslog-ng db-parser sets its .classifier.class to "unknown" regardless of the field's previous state. So if it matched on a previous parser, the next parser will overwrite it if it doesn't match on that. I think it's a bug rather than a feature, so could you please open an issue for that on github? You can merge patterndb .pdb files easily with "pdbtool merge" command, which is shipped with syslog-ng. It's simpler than having junctions :). Best Regards, Viktor On Fri, Apr 11, 2014 at 9:52 PM, David Hauck <davidh@netacquire.com> wrote:
Hi Robert,
On Friday, April 11, 2014 12:19 PM, syslog-ng-bounces@lists.balabit.huwrote:
Hi,
I guess the problem is that patterndb parsers were not thought to be used this way, and each parser sets the .classifier.class value based on its own rules, overwriting any previous values. Consequently, merging the patterndbs into a single file would definitely work.
As it turns out all of the pattern DBs here (these are just the stock files from: https://github.com/balabit/syslog-ng-patterndb) define the same 'class' (so the results of .classifier.class should always be 'system' after running through each of the parsers, no?).
Another option could be a variant of your second idea: you use a single log statement, and embed the parsers into a junction, where each channel of the junction contains a filter (to process only the messages that the parser can parse), and one of the parsers. Something like: log { filter(f_auth); junction{ channel {filter (program(sshd)); parser("sshd");} channel {filter (program(sudo)); parser("sudo");} .... } }
Thanks, I've tried this and it does appear to address this problem (notwithstanding the fact I mention above, where each pattern file specifies the same .classifier.class value).
I'm going to follow-up with another question related to boolean filters and embedded log statements since some of the follow-on filtering criteria now appears unexplainable.
Thanks, -David
Robert
On Friday, April 11, 2014 19:18 CEST, David Hauck <davidh@netacquire.com> wrote:
Hello,
I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases):
log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; };
The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match.
I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way?
Really appreciate any help you might be able to offer.
Thanks, -David
__________________________________________________________ ____________
________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
__________________________________________________________ ____________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Viktor, On Friday, April 11, 2014 1:01 PM, syslog-ng-bounces@lists.balabit.hu wrote:
Hi David!
If a log message does not match any pattern for a parser, syslog-ng db-parser sets its .classifier.class to "unknown" regardless of the field's previous state. So if it matched on a previous parser, the next parser will overwrite it if it doesn't match on that. I think it's a bug rather than a feature, so could you please open an issue for that on github?
Sure, I can do that (although I can imagine a potential valid semantic for wanting this to behave either way).
You can merge patterndb .pdb files easily with "pdbtool merge" command, which is shipped with syslog-ng. It's simpler than having junctions :).
:) OK, that's an option too (although I also like splitting these out into individual files and not having to run the merge whenever an individual file is modified). Cheers, -David
On Fri, Apr 11, 2014 at 10:10 PM, David Hauck <davidh@netacquire.com> wrote:
Hi Viktor,
On Friday, April 11, 2014 1:01 PM, syslog-ng-bounces@lists.balabit.huwrote:
Hi David!
If a log message does not match any pattern for a parser, syslog-ng db-parser sets its .classifier.class to "unknown" regardless of the field's previous state. So if it matched on a previous parser, the next parser will overwrite it if it doesn't match on that. I think it's a bug rather than a feature, so could you please open an issue for that on github?
Sure, I can do that (although I can imagine a potential valid semantic for wanting this to behave either way).
Perhaps then we should make this switchable and the default should be the current behaviour. You're right, I forgot that changing behaviour could break existing configs :(.
You can merge patterndb .pdb files easily with "pdbtool merge" command, which is shipped with syslog-ng. It's simpler than having junctions :).
:) OK, that's an option too (although I also like splitting these out into individual files and not having to run the merge whenever an individual file is modified).
Cheers, -David
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Friday, April 11, 2014 12:53 PM, syslog-ng-bounces@lists.balabit.hu wrote:
Hi Robert,
On Friday, April 11, 2014 12:19 PM, syslog-ng-bounces@lists.balabit.hu wrote:
Hi,
I guess the problem is that patterndb parsers were not thought to be used this way, and each parser sets the .classifier.class value based on its own rules, overwriting any previous values. Consequently, merging the patterndbs into a single file would definitely work.
As it turns out all of the pattern DBs here (these are just the stock files from: https://github.com/balabit/syslog-ng-patterndb) define the same 'class' (so the results of .classifier.class should always be 'system' after running through each of the parsers, no?).
Another option could be a variant of your second idea: you use a single log statement, and embed the parsers into a junction, where each channel of the junction contains a filter (to process only the messages that the parser can parse), and one of the parsers. Something like: log { filter(f_auth); junction{ channel {filter (program(sshd)); parser("sshd");} channel {filter (program(sudo)); parser("sudo");} .... } }
Thanks, I've tried this and it does appear to address this problem (notwithstanding the fact I mention above, where each pattern file specifies the same .classifier.class value).
I'm going to follow-up with another question related to boolean filters and embedded log statements since some of the follow-on filtering criteria now appears unexplainable.
Just to close this part out: the behaviour I was observing wasn't related to any particular quirk of syslog-ng (and Boolean condition filters). I've moved onto the next part of the configuration now that the 'junction()' variant appears to be working...
Thanks, -David
Robert
On Friday, April 11, 2014 19:18 CEST, David Hauck <davidh@netacquire.com> wrote:
Hello,
I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases):
log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; };
The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match.
I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way?
Really appreciate any help you might be able to offer.
Thanks, -David
__________________________________________________________ ____________
________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
__________________________________________________________ ____________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I am having two problems with the multi-line-mode (I am using a pipe source for testing). 1. The last message written to the pipe will not be processed until the start (multi-line-prefix matches) of the next message is received. This makes messages delay until the next message arrives. 2. The last message the is written to the pipe is lost if syslog-ng is reloaded or restarted. Is there an option for how long to wait for the "full" message to arrive? Is there an option to specify what the "LAST" line of the mutli-line message should match? That way the message is known to be completed. Thanks for your time. Evan.
Hi Evan! Unfortunately only PE supports multiline timeout as of now, but we hope that this feature will make it to 3.6 OSE. As for the second question, I submitted a pull request which introduces regexp-suffix multiline mode, it does roughly the same that you mentioned. You can check it here: https://github.com/balabit/syslog-ng/pull/120. Hope I could help, Viktor On Tue, Apr 22, 2014 at 9:30 PM, Evan Rempel <erempel@uvic.ca> wrote:
I am having two problems with the multi-line-mode (I am using a pipe source for testing).
1. The last message written to the pipe will not be processed until the start (multi-line-prefix matches) of the next message is received. This makes messages delay until the next message arrives.
2. The last message the is written to the pipe is lost if syslog-ng is reloaded or restarted.
Is there an option for how long to wait for the "full" message to arrive? Is there an option to specify what the "LAST" line of the mutli-line message should match? That way the message is known to be completed.
Thanks for your time.
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key. Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common) Essentially I want to pull the MID out of the line marked marked: "Start MID (\d+) <other stuff>" and then process every line that matches that specific MID value as part of the message. Note: they all have this string included somewhere: "MID (\d+) " Up to a reasonable timeout - or ended by: "Message finished mid (\d+) done" with the matching ID. Is this possible with syslog-ng? (OSE or PE?) I thought I had seen something using patterndb but I cannot seem to find the reference Clearly there will be interleaved lines with *different* MIDs that need to be processed independently. Thanks in advance! Jim
Hmmm - crickets :-) I have some examples like this: <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> <date> <host> <program>: Info: ICID [0-9]{9} close this is only an example to illustrate the different message elements that contain different kinds of IDs. The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays). I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message. I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately. Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry. Has anyone addressed this? Thanks for any working-examples/guidance/sympathy (in roughly that order :-) Jim ---- jrhendri@roadrunner.com wrote:
Hi,
I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.
Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)
Essentially I want to pull the MID out of the line marked marked:
"Start MID (\d+) <other stuff>"
and then process every line that matches that specific MID value as part of the message.
Note: they all have this string included somewhere:
"MID (\d+) "
Up to a reasonable timeout - or ended by:
"Message finished mid (\d+) done" with the matching ID.
Is this possible with syslog-ng? (OSE or PE?)
I thought I had seen something using patterndb but I cannot seem to find the reference
Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.
Thanks in advance! Jim
I would use logstash to do your multiline stuff http://logstash.net/docs/1.4.0/filters/multiline On Mon, Apr 28, 2014 at 7:44 AM, <jrhendri@roadrunner.com> wrote:
Hmmm - crickets :-)
I have some examples like this: <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> <date> <host> <program>: Info: ICID [0-9]{9} close
this is only an example to illustrate the different message elements that contain different kinds of IDs.
The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).
I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.
I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.
Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.
Has anyone addressed this?
Thanks for any working-examples/guidance/sympathy (in roughly that order :-)
Jim
---- jrhendri@roadrunner.com wrote:
Hi,
I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.
Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)
Essentially I want to pull the MID out of the line marked marked:
"Start MID (\d+) <other stuff>"
and then process every line that matches that specific MID value as part of the message.
Note: they all have this string included somewhere:
"MID (\d+) "
Up to a reasonable timeout - or ended by:
"Message finished mid (\d+) done" with the matching ID.
Is this possible with syslog-ng? (OSE or PE?)
I thought I had seen something using patterndb but I cannot seem to find the reference
Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.
Thanks in advance! Jim
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Jim, Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ... The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl. But, as a Orangepeel says, logstash can be an option. Bye. On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com> wrote:
Hmmm - crickets :-)
I have some examples like this: <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> <date> <host> <program>: Info: ICID [0-9]{9} close
this is only an example to illustrate the different message elements that contain different kinds of IDs.
The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).
I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.
I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.
Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.
Has anyone addressed this?
Thanks for any working-examples/guidance/sympathy (in roughly that order :-)
Jim
---- jrhendri@roadrunner.com wrote:
Hi,
I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.
Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)
Essentially I want to pull the MID out of the line marked marked:
"Start MID (\d+) <other stuff>"
and then process every line that matches that specific MID value as part of the message.
Note: they all have this string included somewhere:
"MID (\d+) "
Up to a reasonable timeout - or ended by:
"Message finished mid (\d+) done" with the matching ID.
Is this possible with syslog-ng? (OSE or PE?)
I thought I had seen something using patterndb but I cannot seem to find the reference
Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.
Thanks in advance! Jim
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi! If you know the format of all the messages which possibly contains a MID, you can write patterns for them and then you can use correlation to extract information from these messages. But it only works with special conditions, I think it wouldn't work in your case. But it wouldn't be so hard to create such functionality in syslog-ng, so if you open a github issue in http://github.com/balabit/syslog-ng, some of us will try to make it work. Best Regards, Viktor On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart@gmail.com>wrote:
Hi Jim,
Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ...
The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl. But, as a Orangepeel says, logstash can be an option.
Bye.
On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com> wrote:
Hmmm - crickets :-)
I have some examples like this: <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> <date> <host> <program>: Info: ICID [0-9]{9} close
this is only an example to illustrate the different message elements that contain different kinds of IDs.
The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).
I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.
I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.
Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.
Has anyone addressed this?
Thanks for any working-examples/guidance/sympathy (in roughly that order :-)
Jim
---- jrhendri@roadrunner.com wrote:
Hi,
I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.
Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)
Essentially I want to pull the MID out of the line marked marked:
"Start MID (\d+) <other stuff>"
and then process every line that matches that specific MID value as part of the message.
Note: they all have this string included somewhere:
"MID (\d+) "
Up to a reasonable timeout - or ended by:
"Message finished mid (\d+) done" with the matching ID.
Is this possible with syslog-ng? (OSE or PE?)
I thought I had seen something using patterndb but I cannot seem to find the reference
Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.
Thanks in advance! Jim
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks all for the thoughts - I will try to write up some of the patterns and correlations, starting with the most simple. This would (I think) be a valuable addition to track different logs that have some dynamic id as a key. (ultimately I am hoping to parse specific data out of these multi-line beasties and be able to populate a database directly from syslog-ng) I will work on writing this up this week. Thanks again! Jim On 04/29/2014 04:53 AM, Tusa Viktor wrote:
Hi!
If you know the format of all the messages which possibly contains a MID, you can write patterns for them and then you can use correlation to extract information from these messages. But it only works with special conditions, I think it wouldn't work in your case. But it wouldn't be so hard to create such functionality in syslog-ng, so if you open a github issue in http://github.com/balabit/syslog-ng, some of us will try to make it work.
Best Regards, Viktor
On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart@gmail.com <mailto:carlopmart@gmail.com>> wrote:
Hi Jim,
Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ...
The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, logstash can be an option.
Bye.
On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com>> wrote: > Hmmm - crickets :-) > > I have some examples like this: > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> > <date> <host> <program>: Info: ICID [0-9]{9} close > > this is only an example to illustrate the different message elements that contain different kinds of IDs. > > The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays). > > I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message. > > I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately. > > Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry. > > Has anyone addressed this? > > Thanks for any working-examples/guidance/sympathy (in roughly that order :-) > > Jim > > > > > ---- jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: >> Hi, >> >> I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key. >> >> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common) >> >> >> Essentially I want to pull the MID out of the line marked marked: >> >> "Start MID (\d+) <other stuff>" >> >> and then process every line that matches that specific MID value as part of the message. >> >> Note: they all have this string included somewhere: >> >> "MID (\d+) " >> >> Up to a reasonable timeout - or ended by: >> >> "Message finished mid (\d+) done" with the matching ID. >> >> Is this possible with syslog-ng? (OSE or PE?) >> >> I thought I had seen something using patterndb but I cannot seem to find the reference >> >> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently. >> >> Thanks in advance! >> Jim > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I have a use case where I want to preface every message from a source with some kind of instance name. It would be easy to create a static template, but I would like to be able to use the same configuration include on all of the hosts and provide the "instance" name from a local configuration space. Is there any way to do this kind of thing @module confgen context(root) name(myinstance) exec("/bin/cat /path/to/instance/file") @define instance "myinstance()" template t_mytemplate { template("<$pri>$parsedate $parsehost ldap: `instance`: $parsemessage\n"); template_escape(no); }; or if the confgen could have a template context @module confgen context(template) name(myinstance) exec("/bin/cat /path/to/instance/file") template t_mytemplate { template("<$pri>$parsedate $parsehost ldap: $(myinstance): $parsemessage\n"); template_escape(no); }; Evan.
On 24.04.2014 00:57, Evan Rempel wrote:
I have a use case where I want to preface every message from a source with some kind of instance name. It would be easy to create a static template, but I would like to be able to use the same configuration include on all of the hosts and provide the "instance" name from a local configuration space.
Is there any way to do this kind of thing
@module confgen context(root) name(myinstance) exec("/bin/cat /path/to/instance/file") @define instance "myinstance()" template t_mytemplate { template("<$pri>$parsedate $parsehost ldap: `instance`: $parsemessage\n"); template_escape(no); };
or if the confgen could have a template context
@module confgen context(template) name(myinstance) exec("/bin/cat /path/to/instance/file") template t_mytemplate { template("<$pri>$parsedate $parsehost ldap: $(myinstance): $parsemessage\n"); template_escape(no); };
Hm, why don't you just include a syslog-ng configuration snippet (that sets what you want) from a file that is local to your instance? Jakub -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
I thought about a config snippet but in my case, I am reading log files from an application that is administered by another group in our organization. It becomes the responsibility of this other group to define the instance name. If it is an actual configuration snippet then if they get it wrong, syslog-ng won't start. I have come up with a working solution, although it is a little indirect. 1. Make a small script that takes as its arguments a name and a path-to-file and produces the code snippet @define name "first-word-of-file" If the file does not exist, or as multiple words or other things, then return nothing. 2. Add these two lines to syslog-ng.conf @module confgen context(root) name(myinstance) exec("/path/to/script instance /path/to/instance/file") myinstance() 3. Use the defined value of `instance` in the template template t_my_template { template("<$pri>$parsedate $parsehost ldap: `instance`: $parsemessage\n"); template_escape(no); }; All of the validation/checking is done in the script to ensure that syslog-ng can restart, and it only needs to be invoked at startup/reload. Evan. On 04/23/2014 05:00 PM, Jakub Jankowski wrote:
On 24.04.2014 00:57, Evan Rempel wrote:
I have a use case where I want to preface every message from a source with some kind of instance name. It would be easy to create a static template, but I would like to be able to use the same configuration include on all of the hosts and provide the "instance" name from a local configuration space.
Is there any way to do this kind of thing
@module confgen context(root) name(myinstance) exec("/bin/cat /path/to/instance/file") @define instance "myinstance()" template t_mytemplate { template("<$pri>$parsedate $parsehost ldap: `instance`: $parsemessage\n"); template_escape(no); };
or if the confgen could have a template context
@module confgen context(template) name(myinstance) exec("/bin/cat /path/to/instance/file") template t_mytemplate { template("<$pri>$parsedate $parsehost ldap: $(myinstance): $parsemessage\n"); template_escape(no); };
Hm, why don't you just include a syslog-ng configuration snippet (that sets what you want) from a file that is local to your instance?
Jakub
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
participants (9)
-
C. L. Martinez
-
David Hauck
-
Evan Rempel
-
Fekete Róbert
-
Jakub Jankowski
-
Jim Hendrick
-
jrhendri＠roadrunner.com
-
Orangepeel Beef
-
Tusa Viktor