[syslog-ng] using correlation to filter out some messages ?

[Martin Holste]

Ok, got it.  So here's the patterndb correlation challenge I'm working
on:  Cisco's Ironport is an email/spam filter that produces verbose
logging.  The logs have two ID's of interest, the ICID which goes with
the connection and the MID which goes with each individual email.
Usually, a single email will produce between 15 and 30 log messages.
I'm looking to use correlation to produce one "meta" message which has
the most important details from all messages so that when searching
logs, you don't have to do follow-up searches to find all of the info
you're looking for.  Here's what I've got so far:

<ruleset>
		<rules>
			<rule class="10" id="10" context-id="ironport-icid"
context-timeout="10" context-scope="program">
				<patterns>
					<pattern>Info: New SMTP ICID @NUMBER:icid:@ interface
@ESTRING:interface_name: @(@IPv4:interface_ip:@) address
@IPv4:sender_ip:@ reverse dns host @ESTRING:sender_dns: @verified
yes</pattern>
				</patterns>
				<examples>
					<example>
						<test_message program="ironport_mail_logs">Info: New SMTP ICID
696117306 interface InternalNet (x.x.88.227) address 10.x.x.x reverse
dns host xx verified yes</test_message>
						<test_value name="icid">696117306</test_value>
						<test_value name="interface_name">InternalNet</test_value>
						<test_value name="interface_ip">x.x.88.227</test_value>
						<test_value name="sender_ip">10.x.x.x</test_value>
						<test_value name="sender_dns">xx</test_value>
					</example>
				</examples>
			</rule>
			<rule class="10" id="10" context-id="ironport-icid"
context-timeout="10" context-scope="program">
				<patterns>
					<pattern>Info: ICID @NUMBER:icid:@ close</pattern>
				</patterns>
				<actions>
					<action>
						<message>
							<values>
								<value name="MESSAGE">IronPort message complete: icid: $icid,
mid: $mid, interface_name: $interface_name, interface_ip:
$interface_ip, sender_ip: $sender_ip, sender_dns: $sender_dns</value>
							</values>
						</message>
					</action>
				</actions>
				<examples>
					<example>
						<test_message program="ironport_mail_logs">Info: ICID 696117306
close</test_message>
					</example>
				</examples>
			</rule>
			<rule class="10" id="10" context-id="ironport-mid"
context-timeout="10" context-scope="program">
				<patterns>
					<pattern>Info: Start MID @NUMBER:mid:@ ICID @NUMBER:icid:@</pattern>
				</patterns>
				<values>
					<value name="icid">$icid</value>
				</values>
				<examples>
					<example>
						<test_message program="ironport_mail_logs">Info: Start MID
144753300 ICID 696117306</test_message>
					</example>
				</examples>
			</rule>
			<rule class="10" id="10" context-id="ironport-mid"
context-timeout="10" context-scope="program">
				<patterns>
					<pattern>Info: Message finished MID @NUMBER:mid:@ done</pattern>
				</patterns>
				<actions>
					<action>
						<message>
							<values>
								<value name="MESSAGE">IronPort message complete: icid:
$icid at 2, mid: $mid</value>
							</values>
						</message>
					</action>
				</actions>
				<examples>
					<example>
						<test_message program="ironport_mail_logs">Info: Message
finished MID 144753300 done</test_message>
					</example>
				</examples>
			</rule>
		</rules>
	</ruleset>

So how do I use tagging to propagate the information contained from
previous messages?  Previously, you had suggested I use grep, but
that's proving to be tough for me to implement.

On Sun, Jan 16, 2011 at 9:06 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> On Sat, 2011-01-15 at 11:16 -0600, Martin Holste wrote:
>> Ok, so the first time the raw message is seen, it is not forwarded
>> through normally then and has effectively been put in limbo, not
>> duplicated, right?
>
> yes.
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>