Ok, got it. So here's the patterndb correlation challenge I'm working on: Cisco's Ironport is an email/spam filter that produces verbose logging. The logs have two ID's of interest, the ICID which goes with the connection and the MID which goes with each individual email. Usually, a single email will produce between 15 and 30 log messages. I'm looking to use correlation to produce one "meta" message which has the most important details from all messages so that when searching logs, you don't have to do follow-up searches to find all of the info you're looking for. Here's what I've got so far: <ruleset> <rules> <rule class="10" id="10" context-id="ironport-icid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: New SMTP ICID @NUMBER:icid:@ interface @ESTRING:interface_name: @(@IPv4:interface_ip:@) address @IPv4:sender_ip:@ reverse dns host @ESTRING:sender_dns: @verified yes</pattern> </patterns> <examples> <example> <test_message program="ironport_mail_logs">Info: New SMTP ICID 696117306 interface InternalNet (x.x.88.227) address 10.x.x.x reverse dns host xx verified yes</test_message> <test_value name="icid">696117306</test_value> <test_value name="interface_name">InternalNet</test_value> <test_value name="interface_ip">x.x.88.227</test_value> <test_value name="sender_ip">10.x.x.x</test_value> <test_value name="sender_dns">xx</test_value> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-icid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: ICID @NUMBER:icid:@ close</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">IronPort message complete: icid: $icid, mid: $mid, interface_name: $interface_name, interface_ip: $interface_ip, sender_ip: $sender_ip, sender_dns: $sender_dns</value> </values> </message> </action> </actions> <examples> <example> <test_message program="ironport_mail_logs">Info: ICID 696117306 close</test_message> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Start MID @NUMBER:mid:@ ICID @NUMBER:icid:@</pattern> </patterns> <values> <value name="icid">$icid</value> </values> <examples> <example> <test_message program="ironport_mail_logs">Info: Start MID 144753300 ICID 696117306</test_message> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Message finished MID @NUMBER:mid:@ done</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">IronPort message complete: icid: $icid@2, mid: $mid</value> </values> </message> </action> </actions> <examples> <example> <test_message program="ironport_mail_logs">Info: Message finished MID 144753300 done</test_message> </example> </examples> </rule> </rules> </ruleset> So how do I use tagging to propagate the information contained from previous messages? Previously, you had suggested I use grep, but that's proving to be tough for me to implement. On Sun, Jan 16, 2011 at 9:06 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Sat, 2011-01-15 at 11:16 -0600, Martin Holste wrote:
Ok, so the first time the raw message is seen, it is not forwarded through normally then and has effectively been put in limbo, not duplicated, right?
yes.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html