<div dir="ltr"><div><div><div>Hi!<br><br></div>If you know the format of all the messages which possibly contains a MID, you can write patterns for them and then you can use correlation to extract information from these messages. But it only works with special conditions, I think it wouldn't work in your case. But it wouldn't be so hard to create such functionality in syslog-ng, so if you open a github issue in <a href="http://github.com/balabit/syslog-ng">http://github.com/balabit/syslog-ng</a>, some of us will try to make it work.<br>
<br></div>Best Regards,<br></div>Viktor<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <span dir="ltr"><<a href="mailto:carlopmart@gmail.com" target="_blank">carlopmart@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jim,<br>
<br>
Some time ago, I have tried the same: correlate logs for Ironport<br>
devices. And my conclusion was: impossible. I loose a lot info and<br>
some correlated logs are wrong ...<br>
<br>
The only approach that maybe should work with opensource tools, IMO,<br>
is rsyslog+<a href="http://sec.pl" target="_blank">sec.pl</a>. But, as a Orangepeel says, logstash can be an<br>
option.<br>
<br>
Bye.<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Apr 28, 2014 at 2:44 PM, <<a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>> wrote:<br>
> Hmmm - crickets :-)<br>
><br>
> I have some examples like this:<br>
> <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info: ICID [0-9]{9} close<br>
><br>
> this is only an example to illustrate the different message elements that contain different kinds of IDs.<br>
><br>
> The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays).<br>
><br>
> I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message.<br>
><br>
> I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately.<br>
><br>
> Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry.<br>
><br>
> Has anyone addressed this?<br>
><br>
> Thanks for any working-examples/guidance/sympathy (in roughly that order :-)<br>
><br>
> Jim<br>
><br>
><br>
><br>
><br>
> ---- <a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a> wrote:<br>
>> Hi,<br>
>><br>
>> I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key.<br>
>><br>
>> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common)<br>
>><br>
>><br>
>> Essentially I want to pull the MID out of the line marked marked:<br>
>><br>
>> "Start MID (\d+) <other stuff>"<br>
>><br>
>> and then process every line that matches that specific MID value as part of the message.<br>
>><br>
>> Note: they all have this string included somewhere:<br>
>><br>
>> "MID (\d+) "<br>
>><br>
>> Up to a reasonable timeout - or ended by:<br>
>><br>
>> "Message finished mid (\d+) done" with the matching ID.<br>
>><br>
>> Is this possible with syslog-ng? (OSE or PE?)<br>
>><br>
>> I thought I had seen something using patterndb but I cannot seem to find the reference<br>
>><br>
>> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently.<br>
>><br>
>> Thanks in advance!<br>
>> Jim<br>
><br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br></div>