<div dir="ltr">I would use logstash to do your multiline stuff <div><br></div><div><a href="http://logstash.net/docs/1.4.0/filters/multiline">http://logstash.net/docs/1.4.0/filters/multiline</a><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Apr 28, 2014 at 7:44 AM, <span dir="ltr"><<a href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hmmm - crickets :-)<br>
<br>
I have some examples like this:<br>
<date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message><br>
<date> <host> <program>: Info: ICID [0-9]{9} close<br>
<br>
this is only an example to illustrate the different message elements that contain different kinds of IDs.<br>
<br>
The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).<br>
<br>
I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.<br>
<br>
I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.<br>
<br>
Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.<br>
<br>
Has anyone addressed this?<br>
<br>
Thanks for any working-examples/guidance/sympathy (in roughly that order :-)<br>
<br>
Jim<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
---- <a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a> wrote:<br>
> Hi,<br>
><br>
> I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.<br>
><br>
> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)<br>
><br>
><br>
> Essentially I want to pull the MID out of the line marked marked:<br>
><br>
> "Start MID (\d+) <other stuff>"<br>
><br>
> and then process every line that matches that specific MID value as part of the message.<br>
><br>
> Note: they all have this string included somewhere:<br>
><br>
> "MID (\d+) "<br>
><br>
> Up to a reasonable timeout - or ended by:<br>
><br>
> "Message finished mid (\d+) done" with the matching ID.<br>
><br>
> Is this possible with syslog-ng? (OSE or PE?)<br>
><br>
> I thought I had seen something using patterndb but I cannot seem to find the reference<br>
><br>
> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.<br>
><br>
> Thanks in advance!<br>
> Jim<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br></div>