[syslog-ng] Basic (?) multi line question

[jrhendri at roadrunner.com]

Hmmm - crickets :-)

I have some examples like this:
<date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message>
<date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message>
<date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message>
<date> <host> <program>: Info: ICID [0-9]{9} close

this is only an example to illustrate the different message elements that contain different kinds of IDs.

The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).

I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.

I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.

Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.

Has anyone addressed this?

Thanks for any working-examples/guidance/sympathy (in roughly that order :-)

Jim




---- jrhendri at roadrunner.com wrote: 
> Hi,
> 
>   I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.
> 
> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)
> 
> 
> Essentially I want to pull the MID out of the line marked marked:
> 
> "Start MID (\d+) <other stuff>"
> 
>  and then process every line that matches that specific MID value as part of the message.
> 
> Note: they all have this string included somewhere:
> 
> "MID (\d+) "
> 
> Up to a reasonable timeout - or ended by:
> 
>  "Message finished mid (\d+) done" with the matching ID.
> 
> Is this possible with syslog-ng? (OSE or PE?)
> 
> I thought I had seen something using patterndb but I cannot seem to find the reference
> 
> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.
> 
> Thanks in advance!
> Jim