[syslog-ng] Basic (?) multi line question

[Jim Hendrick]

Thanks all for the thoughts -

I will try to write up some of the patterns and correlations, starting
with the most simple.

This would (I think) be a valuable addition to track different logs that
have some dynamic id as a key.

(ultimately I am hoping to parse specific data out of these multi-line
beasties and be able to populate a database directly from syslog-ng)

I will work on writing this up this week.

Thanks again!
Jim


On 04/29/2014 04:53 AM, Tusa Viktor wrote:
> Hi!
>
> If you know the format of all the messages which possibly contains a
> MID, you can write patterns for them and then you can use correlation
> to extract information from these messages. But it only works with
> special conditions, I think it wouldn't work in your case. But it
> wouldn't be so hard to create such functionality in syslog-ng, so if
> you open a github issue in http://github.com/balabit/syslog-ng, some
> of us will try to make it work.
>
> Best Regards,
> Viktor
>
>
> On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart at gmail.com
> <mailto:carlopmart at gmail.com>> wrote:
>
>     Hi Jim,
>
>      Some time ago, I have tried the same: correlate logs for Ironport
>     devices. And my conclusion was: impossible. I loose a lot info and
>     some correlated logs are wrong ...
>
>      The only approach that maybe should work with opensource tools, IMO,
>     is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says,
>     logstash can be an
>     option.
>
>     Bye.
>
>     On Mon, Apr 28, 2014 at 2:44 PM,  <jrhendri at roadrunner.com
>     <mailto:jrhendri at roadrunner.com>> wrote:
>     > Hmmm - crickets :-)
>     >
>     > I have some examples like this:
>     > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of
>     message>
>     > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
>     <rest of message>
>     > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
>     <rest of message>
>     > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
>     <rest of message>
>     > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
>     <rest of message>
>     > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of
>     message>
>     > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID
>     [0-9]{9} <rest of message>
>     > <date> <host> <program>: Info: ICID [0-9]{9} close
>     >
>     > this is only an example to illustrate the different message
>     elements that contain different kinds of IDs.
>     >
>     > The issue is there will be interleaving with *different* ICID
>     (inbound connections from different SMTP servers) each sending
>     multiple MIDs (message IDs) and also different DCID (destination
>     connections *to* different mail relays).
>     >
>     > I have been looking at multi-line-mode(regexp) but that seems to
>     imply all consecutive lines until the next regex match are assumed
>     to be part of the same message.
>     >
>     > I hope I can do something where all matching ICIDs are treated
>     as part of one line, that can be parsed separately.
>     >
>     > Not sure if this is possible with multi-line-mode *or* with some
>     patterndb wizardry.
>     >
>     > Has anyone addressed this?
>     >
>     > Thanks for any working-examples/guidance/sympathy (in roughly
>     that order :-)
>     >
>     > Jim
>     >
>     >
>     >
>     >
>     > ---- jrhendri at roadrunner.com <mailto:jrhendri at roadrunner.com> wrote:
>     >> Hi,
>     >>
>     >>   I am trying to parse data elements out of a variable number
>     of log lines that all are associated by a single unique key.
>     >>
>     >> Specifically - they are Cisco IronPort email logs that have
>     various "ID" fields (MID - message ID is the most common)
>     >>
>     >>
>     >> Essentially I want to pull the MID out of the line marked marked:
>     >>
>     >> "Start MID (\d+) <other stuff>"
>     >>
>     >>  and then process every line that matches that specific MID
>     value as part of the message.
>     >>
>     >> Note: they all have this string included somewhere:
>     >>
>     >> "MID (\d+) "
>     >>
>     >> Up to a reasonable timeout - or ended by:
>     >>
>     >>  "Message finished mid (\d+) done" with the matching ID.
>     >>
>     >> Is this possible with syslog-ng? (OSE or PE?)
>     >>
>     >> I thought I had seen something using patterndb but I cannot
>     seem to find the reference
>     >>
>     >> Clearly there will be interleaved lines with *different* MIDs
>     that need to be processed independently.
>     >>
>     >> Thanks in advance!
>     >> Jim
>     >
>     >
>     ______________________________________________________________________________
>     > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     > Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>     >
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140429/ebda655f/attachment-0001.htm