Thanks all for the thoughts - I will try to write up some of the patterns and correlations, starting with the most simple. This would (I think) be a valuable addition to track different logs that have some dynamic id as a key. (ultimately I am hoping to parse specific data out of these multi-line beasties and be able to populate a database directly from syslog-ng) I will work on writing this up this week. Thanks again! Jim On 04/29/2014 04:53 AM, Tusa Viktor wrote:
Hi!
If you know the format of all the messages which possibly contains a MID, you can write patterns for them and then you can use correlation to extract information from these messages. But it only works with special conditions, I think it wouldn't work in your case. But it wouldn't be so hard to create such functionality in syslog-ng, so if you open a github issue in http://github.com/balabit/syslog-ng, some of us will try to make it work.
Best Regards, Viktor
On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart@gmail.com <mailto:carlopmart@gmail.com>> wrote:
Hi Jim,
Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ...
The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, logstash can be an option.
Bye.
On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com>> wrote: > Hmmm - crickets :-) > > I have some examples like this: > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> > <date> <host> <program>: Info: ICID [0-9]{9} close > > this is only an example to illustrate the different message elements that contain different kinds of IDs. > > The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays). > > I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message. > > I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately. > > Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry. > > Has anyone addressed this? > > Thanks for any working-examples/guidance/sympathy (in roughly that order :-) > > Jim > > > > > ---- jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: >> Hi, >> >> I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key. >> >> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common) >> >> >> Essentially I want to pull the MID out of the line marked marked: >> >> "Start MID (\d+) <other stuff>" >> >> and then process every line that matches that specific MID value as part of the message. >> >> Note: they all have this string included somewhere: >> >> "MID (\d+) " >> >> Up to a reasonable timeout - or ended by: >> >> "Message finished mid (\d+) done" with the matching ID. >> >> Is this possible with syslog-ng? (OSE or PE?) >> >> I thought I had seen something using patterndb but I cannot seem to find the reference >> >> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently. >> >> Thanks in advance! >> Jim > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq