Not able to match on a field in syslog messages from a DataPower Appliance
I am using syslog-ng 1.6.12 to collect logs from an IBM DataPower appliance. The DataPower appliance allows a certain custom string to be added to the outgoing syslogs called a Local Identifier. I am trying to use a regex to match on this Local Identifier. I have used tcpdump to capture actual packets and can see that the Local Identifier is in the Application layer Syslog message field. My question is why is match not working when I try to use a filter to grab just a particular Local ID? No logs are caught by my filter. I have tried: match ("DP\d+Syslog") match ("DPRASSyslogAudit3") match ("DPRASSyslogAudit") match ("DP") I have been able to catch other strings in the messages, such as system, debug, dpHandler, etc..." These example packets below are copy and pasted from wireshark by clicking on the Syslog message field in the packet details from and right clicking -> Copy -> Description. * Three Syslog messages from the DataPower appliance that show the Local identifier that I want to match against in the fourth field (DPRASSyslogAudit3, DP472Syslog). Syslog message: USER.DEBUG: Jul 15 18:23:44 DPRASSyslogAudit3 [system][debug] trans(383): cpu usage: 57%(10 sec) 57%(1 min) 57%(10 min) 56%(1 hour) 58%(1 day)\n Syslog message: USER.ERR: Jul 15 18:23:43 DP472Syslog [mpgw][error] trans(8152497)[19.x.x.x]: Request processing failed: Connection terminated before request headers read\n Syslog message: USER.ERR: Jul 15 18:25:11 DP474Syslog [dpHandler_prod_host474][ssl][error] valcred(pubcert): trans(8043025)[19.x.x.x]: SSL Proxy Profile 'xgtws': connection error: peer did not send a certificate\n * One syslog messsage from a Cisco device which shows the hostname in the fourth field Syslog message: LOCAL7.NOTICE: Jul 15 22:23:48 hostname 2008 Jul 15 22:23:46 %SNMP-5-SNMPAUTHFAIL:Authentication failed for message from 19.x.x.x\n Does anyone have any idea why this field might be getting missed or can anyone give me some guidance as to where in the source the incoming messages are parsed? I have tried all the macros listed in the code and never get the local ID. #DataPower Appliances Extreme Test destination syncDPextest { file("/logs/syncDPextest/$YEAR$MONTH$DAY.txt" template("$FACILITY $FACILITY_NUM $PRIORITY $LEVEL $LEVEL_NUM $TAG $PRI $HOST $DATE $FULLHOST_FROM $MESSAGE\n") ); }; user 1 err err 3 0b 11 hostname Jul 16 16:10:59 hostname [dpHandler_prod_host472][ssl][error] valcred(pubcert): trans(8320177)[19.x.x.x]: SSL Proxy Profile \'xgtws\': connection error: peer did not send a certificate Any help or suggestions whould be appreciated. Thanks, Adam
USER.DEBUG: Jul 15 18:23:44 DPRASSyslogAudit3 [system][debug] trans(383): cpu usage: 57%(10 sec) 57%(1 min) 57%(10 min) 56%(1 hour) 58%(1 day) In syslog wire protocol the field immediately after the timestamp is the hostname, so this message would be interpreted as being from a host called DPRASSyslogAudit3. Your filters are not matching these messages because match() only matches elements of the message body and the text you're looking for does not appear in the message body. Joe. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Salowitz, Adam (AS.) Sent: 16 July 2008 21:14 To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] Not able to match on a field in syslog messages from a DataPower Appliance I am using syslog-ng 1.6.12 to collect logs from an IBM DataPower appliance. The DataPower appliance allows a certain custom string to be added to the outgoing syslogs called a Local Identifier. I am trying to use a regex to match on this Local Identifier. I have used tcpdump to capture actual packets and can see that the Local Identifier is in the Application layer Syslog message field. My question is why is match not working when I try to use a filter to grab just a particular Local ID? No logs are caught by my filter. I have tried: match ("DP\d+Syslog") match ("DPRASSyslogAudit3") match ("DPRASSyslogAudit") match ("DP") I have been able to catch other strings in the messages, such as system, debug, dpHandler, etc..." These example packets below are copy and pasted from wireshark by clicking on the Syslog message field in the packet details from and right clicking -> Copy -> Description. * Three Syslog messages from the DataPower appliance that show the Local identifier that I want to match against in the fourth field (DPRASSyslogAudit3, DP472Syslog). Syslog message: USER.DEBUG: Jul 15 18:23:44 DPRASSyslogAudit3 [system][debug] trans(383): cpu usage: 57%(10 sec) 57%(1 min) 57%(10 min) 56%(1 hour) 58%(1 day)\n Syslog message: USER.ERR: Jul 15 18:23:43 DP472Syslog [mpgw][error] trans(8152497)[19.x.x.x]: Request processing failed: Connection terminated before request headers read\n Syslog message: USER.ERR: Jul 15 18:25:11 DP474Syslog [dpHandler_prod_host474][ssl][error] valcred(pubcert): trans(8043025)[19.x.x.x]: SSL Proxy Profile 'xgtws': connection error: peer did not send a certificate\n * One syslog messsage from a Cisco device which shows the hostname in the fourth field Syslog message: LOCAL7.NOTICE: Jul 15 22:23:48 hostname 2008 Jul 15 22:23:46 %SNMP-5-SNMPAUTHFAIL:Authentication failed for message from 19.x.x.x\n Does anyone have any idea why this field might be getting missed or can anyone give me some guidance as to where in the source the incoming messages are parsed? I have tried all the macros listed in the code and never get the local ID. #DataPower Appliances Extreme Test destination syncDPextest { file("/logs/syncDPextest/$YEAR$MONTH$DAY.txt" template("$FACILITY $FACILITY_NUM $PRIORITY $LEVEL $LEVEL_NUM $TAG $PRI $HOST $DATE $FULLHOST_FROM $MESSAGE\n") ); }; user 1 err err 3 0b 11 hostname Jul 16 16:10:59 hostname [dpHandler_prod_host472][ssl][error] valcred(pubcert): trans(8320177)[19.x.x.x]: SSL Proxy Profile \'xgtws\': connection error: peer did not send a certificate Any help or suggestions whould be appreciated. Thanks, Adam ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Salowitz, Adam (AS.) wrote:
I am using syslog-ng 1.6.12 to collect logs from an IBM DataPower appliance. The DataPower appliance allows a certain custom string to be added to the outgoing syslogs called a Local Identifier. I am trying to use a regex to match on this Local Identifier. I have used tcpdump to capture actual packets and can see that the Local Identifier is in the Application layer Syslog message field.
My question is why is match not working when I try to use a filter to grab just a particular Local ID? No logs are caught by my filter. I have tried:
match ("DP\d+Syslog") match ("DPRASSyslogAudit3") match ("DPRASSyslogAudit") match ("DP")
I have been able to catch other strings in the messages, such as system, debug, dpHandler, etc..."
These example packets below are copy and pasted from wireshark by clicking on the Syslog message field in the packet details from and right clicking -> Copy -> Description.
* Three Syslog messages from the DataPower appliance that show the Local identifier that I want to match against in the fourth field (DPRASSyslogAudit3, DP472Syslog).
Syslog message: USER.DEBUG: Jul 15 18:23:44 DPRASSyslogAudit3 [system][debug] trans(383): cpu usage: 57%(10 sec) 57%(1 min) 57%(10 min) 56%(1 hour) 58%(1 day)\n
Syslog message: USER.ERR: Jul 15 18:23:43 DP472Syslog [mpgw][error] trans(8152497)[19.x.x.x]: Request processing failed: Connection terminated before request headers read\n
Syslog message: USER.ERR: Jul 15 18:25:11 DP474Syslog [dpHandler_prod_host474][ssl][error] valcred(pubcert): trans(8043025)[19.x.x.x]: SSL Proxy Profile 'xgtws': connection error: peer did not send a certificate\n
* One syslog messsage from a Cisco device which shows the hostname in the fourth field
Syslog message: LOCAL7.NOTICE: Jul 15 22:23:48 hostname 2008 Jul 15 22:23:46 %SNMP-5-SNMPAUTHFAIL:Authentication failed for message from 19.x.x.x\n
Does anyone have any idea why this field might be getting missed or can anyone give me some guidance as to where in the source the incoming messages are parsed? I have tried all the macros listed in the code and never get the local ID.
#DataPower Appliances Extreme Test destination syncDPextest { file("/logs/syncDPextest/$YEAR$MONTH$DAY.txt" template("$FACILITY $FACILITY_NUM $PRIORITY $LEVEL $LEVEL_NUM $TAG $PRI $HOST $DATE $FULLHOST_FROM $MESSAGE\n") ); };
user 1 err err 3 0b 11 hostname Jul 16 16:10:59 hostname [dpHandler_prod_host472][ssl][error] valcred(pubcert): trans(8320177)[19.x.x.x]: SSL Proxy Profile \'xgtws\': connection error: peer did not send a certificate
Any help or suggestions whould be appreciated.
Thanks,
Adam ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
This filter: match ("DP\d+Syslog") Will not work because \d is a PCRE metacharacter and IIRC syslog-nx uses POSIX-extended regular expressions. The POSIX metastring for \d is [:digit:], so your filter would look like this: match ("DP[:digit:]+Syslog")
participants (3)
-
Enigma
-
Fegan, Joe
-
Salowitz, Adam (AS.)