<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Thanks all for the thoughts -<br>
<br>
I will try to write up some of the patterns and correlations,
starting with the most simple.<br>
<br>
This would (I think) be a valuable addition to track different logs
that have some dynamic id as a key.<br>
<br>
(ultimately I am hoping to parse specific data out of these
multi-line beasties and be able to populate a database directly from
syslog-ng)<br>
<br>
I will work on writing this up this week.<br>
<br>
Thanks again!<br>
Jim<br>
<br>
<br>
<div class="moz-cite-prefix">On 04/29/2014 04:53 AM, Tusa Viktor
wrote:<br>
</div>
<blockquote
cite="mid:CAB3gDDk1wdgLSX3hvKVChoZxB6Xfx76b_PJnjSHxEc5VPnEtgQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi!<br>
<br>
</div>
If you know the format of all the messages which possibly
contains a MID, you can write patterns for them and then you
can use correlation to extract information from these
messages. But it only works with special conditions, I think
it wouldn't work in your case. But it wouldn't be so hard to
create such functionality in syslog-ng, so if you open a
github issue in <a moz-do-not-send="true"
href="http://github.com/balabit/syslog-ng">http://github.com/balabit/syslog-ng</a>,
some of us will try to make it work.<br>
<br>
</div>
Best Regards,<br>
</div>
Viktor<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Apr 29, 2014 at 8:14 AM, C. L.
Martinez <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:carlopmart@gmail.com" target="_blank">carlopmart@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jim,<br>
<br>
Some time ago, I have tried the same: correlate logs for
Ironport<br>
devices. And my conclusion was: impossible. I loose a lot
info and<br>
some correlated logs are wrong ...<br>
<br>
The only approach that maybe should work with opensource
tools, IMO,<br>
is rsyslog+<a moz-do-not-send="true" href="http://sec.pl"
target="_blank">sec.pl</a>. But, as a Orangepeel says,
logstash can be an<br>
option.<br>
<br>
Bye.<br>
<div class="HOEnZb">
<div class="h5"><br>
On Mon, Apr 28, 2014 at 2:44 PM, <<a
moz-do-not-send="true"
href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>>
wrote:<br>
> Hmmm - crickets :-)<br>
><br>
> I have some examples like this:<br>
> <date> <host> <program>: Info:
New SMTP ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info:
Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info:
Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info:
Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info:
Start MID [0-9]{9} ICID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info:
New SMTP DCID [0-9]{9} <rest of message><br>
> <date> <host> <program>: Info:
Message done DCID [0-9]{9} MID [0-9]{9} <rest of
message><br>
> <date> <host> <program>: Info:
ICID [0-9]{9} close<br>
><br>
> this is only an example to illustrate the different
message elements that contain different kinds of IDs.<br>
><br>
> The issue is there will be interleaving with
*different* ICID (inbound connections from different
SMTP servers) each sending multiple MIDs (message IDs)
and also different DCID (destination connections *to*
different mail relays).<br>
><br>
> I have been looking at multi-line-mode(regexp) but
that seems to imply all consecutive lines until the next
regex match are assumed to be part of the same message.<br>
><br>
> I hope I can do something where all matching ICIDs
are treated as part of one line, that can be parsed
separately.<br>
><br>
> Not sure if this is possible with multi-line-mode
*or* with some patterndb wizardry.<br>
><br>
> Has anyone addressed this?<br>
><br>
> Thanks for any working-examples/guidance/sympathy
(in roughly that order :-)<br>
><br>
> Jim<br>
><br>
><br>
><br>
><br>
> ---- <a moz-do-not-send="true"
href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>
wrote:<br>
>> Hi,<br>
>><br>
>> I am trying to parse data elements out of a
variable number of log lines that all are associated by
a single unique key.<br>
>><br>
>> Specifically - they are Cisco IronPort email
logs that have various "ID" fields (MID - message ID is
the most common)<br>
>><br>
>><br>
>> Essentially I want to pull the MID out of the
line marked marked:<br>
>><br>
>> "Start MID (\d+) <other stuff>"<br>
>><br>
>> and then process every line that matches that
specific MID value as part of the message.<br>
>><br>
>> Note: they all have this string included
somewhere:<br>
>><br>
>> "MID (\d+) "<br>
>><br>
>> Up to a reasonable timeout - or ended by:<br>
>><br>
>> "Message finished mid (\d+) done" with the
matching ID.<br>
>><br>
>> Is this possible with syslog-ng? (OSE or PE?)<br>
>><br>
>> I thought I had seen something using patterndb
but I cannot seem to find the reference<br>
>><br>
>> Clearly there will be interleaved lines with
*different* MIDs that need to be processed
independently.<br>
>><br>
>> Thanks in advance!<br>
>> Jim<br>
><br>
>
______________________________________________________________________________<br>
> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>