Hi Jim, Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ... The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl. But, as a Orangepeel says, logstash can be an option. Bye. On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com> wrote:
Hmmm - crickets :-)
I have some examples like this: <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> <date> <host> <program>: Info: ICID [0-9]{9} close
this is only an example to illustrate the different message elements that contain different kinds of IDs.
The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).
I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.
I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.
Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.
Has anyone addressed this?
Thanks for any working-examples/guidance/sympathy (in roughly that order :-)
Jim
---- jrhendri@roadrunner.com wrote:
Hi,
I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.
Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)
Essentially I want to pull the MID out of the line marked marked:
"Start MID (\d+) <other stuff>"
and then process every line that matches that specific MID value as part of the message.
Note: they all have this string included somewhere:
"MID (\d+) "
Up to a reasonable timeout - or ended by:
"Message finished mid (\d+) done" with the matching ID.
Is this possible with syslog-ng? (OSE or PE?)
I thought I had seen something using patterndb but I cannot seem to find the reference
Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.
Thanks in advance! Jim
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq