[syslog-ng] Basic (?) multi line question

[Tusa Viktor]

Hi!

If you know the format of all the messages which possibly contains a MID,
you can write patterns for them and then you can use correlation to extract
information from these messages. But it only works with special conditions,
I think it wouldn't work in your case. But it wouldn't be so hard to create
such functionality in syslog-ng, so if you open a github issue in
http://github.com/balabit/syslog-ng, some of us will try to make it work.

Best Regards,
Viktor


On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart at gmail.com>wrote:

> Hi Jim,
>
>  Some time ago, I have tried the same: correlate logs for Ironport
> devices. And my conclusion was: impossible. I loose a lot info and
> some correlated logs are wrong ...
>
>  The only approach that maybe should work with opensource tools, IMO,
> is rsyslog+sec.pl. But, as a Orangepeel says, logstash can be an
> option.
>
> Bye.
>
> On Mon, Apr 28, 2014 at 2:44 PM,  <jrhendri at roadrunner.com> wrote:
> > Hmmm - crickets :-)
> >
> > I have some examples like this:
> > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message>
> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of
> message>
> > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message>
> > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9}
> <rest of message>
> > <date> <host> <program>: Info: ICID [0-9]{9} close
> >
> > this is only an example to illustrate the different message elements
> that contain different kinds of IDs.
> >
> > The issue is there will be interleaving with *different* ICID (inbound
> connections from different SMTP servers) each sending multiple MIDs
> (message IDs) and also different DCID (destination connections *to*
> different mail relays).
> >
> > I have been looking at multi-line-mode(regexp) but that seems to imply
> all consecutive lines until the next regex match are assumed to be part of
> the same message.
> >
> > I hope I can do something where all matching ICIDs are treated as part
> of one line, that can be parsed separately.
> >
> > Not sure if this is possible with multi-line-mode *or* with some
> patterndb wizardry.
> >
> > Has anyone addressed this?
> >
> > Thanks for any working-examples/guidance/sympathy (in roughly that order
> :-)
> >
> > Jim
> >
> >
> >
> >
> > ---- jrhendri at roadrunner.com wrote:
> >> Hi,
> >>
> >>   I am trying to parse data elements out of a variable number of log
> lines that all are associated by a single unique key.
> >>
> >> Specifically - they are Cisco IronPort email logs that have various
> "ID" fields (MID - message ID is the most common)
> >>
> >>
> >> Essentially I want to pull the MID out of the line marked marked:
> >>
> >> "Start MID (\d+) <other stuff>"
> >>
> >>  and then process every line that matches that specific MID value as
> part of the message.
> >>
> >> Note: they all have this string included somewhere:
> >>
> >> "MID (\d+) "
> >>
> >> Up to a reasonable timeout - or ended by:
> >>
> >>  "Message finished mid (\d+) done" with the matching ID.
> >>
> >> Is this possible with syslog-ng? (OSE or PE?)
> >>
> >> I thought I had seen something using patterndb but I cannot seem to
> find the reference
> >>
> >> Clearly there will be interleaved lines with *different* MIDs that need
> to be processed independently.
> >>
> >> Thanks in advance!
> >> Jim
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140429/f01ad16d/attachment-0001.htm