[syslog-ng] Basic (?) multi line question

[Orangepeel Beef]

I would use logstash to do your multiline stuff

http://logstash.net/docs/1.4.0/filters/multiline


On Mon, Apr 28, 2014 at 7:44 AM, <jrhendri at roadrunner.com> wrote:

> Hmmm - crickets :-)
>
> I have some examples like this:
> <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message>
> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9}
> <rest of message>
> <date> <host> <program>: Info: ICID [0-9]{9} close
>
> this is only an example to illustrate the different message elements that
> contain different kinds of IDs.
>
> The issue is there will be interleaving with *different* ICID (inbound
> connections from different SMTP servers) each sending multiple MIDs
> (message IDs) and also different DCID (destination connections *to*
> different mail relays).
>
> I have been looking at multi-line-mode(regexp) but that seems to imply all
> consecutive lines until the next regex match are assumed to be part of the
> same message.
>
> I hope I can do something where all matching ICIDs are treated as part of
> one line, that can be parsed separately.
>
> Not sure if this is possible with multi-line-mode *or* with some patterndb
> wizardry.
>
> Has anyone addressed this?
>
> Thanks for any working-examples/guidance/sympathy (in roughly that order
> :-)
>
> Jim
>
>
>
>
> ---- jrhendri at roadrunner.com wrote:
> > Hi,
> >
> >   I am trying to parse data elements out of a variable number of log
> lines that all are associated by a single unique key.
> >
> > Specifically - they are Cisco IronPort email logs that have various "ID"
> fields (MID - message ID is the most common)
> >
> >
> > Essentially I want to pull the MID out of the line marked marked:
> >
> > "Start MID (\d+) <other stuff>"
> >
> >  and then process every line that matches that specific MID value as
> part of the message.
> >
> > Note: they all have this string included somewhere:
> >
> > "MID (\d+) "
> >
> > Up to a reasonable timeout - or ended by:
> >
> >  "Message finished mid (\d+) done" with the matching ID.
> >
> > Is this possible with syslog-ng? (OSE or PE?)
> >
> > I thought I had seen something using patterndb but I cannot seem to find
> the reference
> >
> > Clearly there will be interleaved lines with *different* MIDs that need
> to be processed independently.
> >
> > Thanks in advance!
> > Jim
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140428/1c58d411/attachment.htm