July 2008 - syslog-ng - lists.balabit.hu

Re: [syslog-ng] Not able to match on a field in syslog messages from a DataPower Appliance
by Salowitz, Adam (AS.) 17 Jul '08

17 Jul '08

>USER.DEBUG: Jul 15 18:23:44 DPRASSyslogAudit3 [system][debug] >trans(383): cpu usage: 57%(10 sec) 57%(1 min) 57%(10 >min) 56%(1 hour) 58%(1 day) > >In syslog wire protocol the field immediately after the >timestamp is the hostname, so this message would be >interpreted as being from a host called DPRASSyslogAudit3. >Your filters are not matching these messages because match() >only matches elements of the message body and the text you're >looking for does not appear in the message body. > >Joe. That's actually what thought (but forgot to mention in my earlier email), but when I try to match on host it doesn't catch anything either. Neither of these catch anything: filter group_syncDPextest { host("DP[:digit:]+Syslog") or host("DPRASSyslogAudit[:digit:]+") ;}; filter group_syncDPextest { host("DP") ;}; Is this because the host filter actually comes from the source host of the packet and not the message itself? Are DataPower's syslog messages just broken or non-rfc complient? Should I just give up and try to match on other strings in the message? Thanks again for all your help on this weird problem. Adam

2 2

Not able to match on a field in syslog messages from a DataPower Appliance
by Salowitz, Adam (AS.) 17 Jul '08

17 Jul '08

I am using syslog-ng 1.6.12 to collect logs from an IBM DataPower appliance. The DataPower appliance allows a certain custom string to be added to the outgoing syslogs called a Local Identifier. I am trying to use a regex to match on this Local Identifier. I have used tcpdump to capture actual packets and can see that the Local Identifier is in the Application layer Syslog message field. My question is why is match not working when I try to use a filter to grab just a particular Local ID? No logs are caught by my filter. I have tried: match ("DP\d+Syslog") match ("DPRASSyslogAudit3") match ("DPRASSyslogAudit") match ("DP") I have been able to catch other strings in the messages, such as system, debug, dpHandler, etc..." These example packets below are copy and pasted from wireshark by clicking on the Syslog message field in the packet details from and right clicking -> Copy -> Description. * Three Syslog messages from the DataPower appliance that show the Local identifier that I want to match against in the fourth field (DPRASSyslogAudit3, DP472Syslog). Syslog message: USER.DEBUG: Jul 15 18:23:44 DPRASSyslogAudit3 [system][debug] trans(383): cpu usage: 57%(10 sec) 57%(1 min) 57%(10 min) 56%(1 hour) 58%(1 day)\n Syslog message: USER.ERR: Jul 15 18:23:43 DP472Syslog [mpgw][error] trans(8152497)[19.x.x.x]: Request processing failed: Connection terminated before request headers read\n Syslog message: USER.ERR: Jul 15 18:25:11 DP474Syslog [dpHandler_prod_host474][ssl][error] valcred(pubcert): trans(8043025)[19.x.x.x]: SSL Proxy Profile 'xgtws': connection error: peer did not send a certificate\n * One syslog messsage from a Cisco device which shows the hostname in the fourth field Syslog message: LOCAL7.NOTICE: Jul 15 22:23:48 hostname 2008 Jul 15 22:23:46 %SNMP-5-SNMPAUTHFAIL:Authentication failed for message from 19.x.x.x\n Does anyone have any idea why this field might be getting missed or can anyone give me some guidance as to where in the source the incoming messages are parsed? I have tried all the macros listed in the code and never get the local ID. #DataPower Appliances Extreme Test destination syncDPextest { file("/logs/syncDPextest/$YEAR$MONTH$DAY.txt" template("$FACILITY $FACILITY_NUM $PRIORITY $LEVEL $LEVEL_NUM $TAG $PRI $HOST $DATE $FULLHOST_FROM $MESSAGE\n") ); }; user 1 err err 3 0b 11 hostname Jul 16 16:10:59 hostname [dpHandler_prod_host472][ssl][error] valcred(pubcert): trans(8320177)[19.x.x.x]: SSL Proxy Profile \'xgtws\': connection error: peer did not send a certificate Any help or suggestions whould be appreciated. Thanks, Adam

3 2

Using Regex information for destination
by James Whitt 11 Jul '08

11 Jul '08

I had asked this question about a year ago, but was told to wait for the new 2.0 version. I'm currently running the latest syslog-ng 2.0.7. What I would like to do is put the line in a specific file based on part of the hostname. For example: Hostname: subdomain.domain.tld destination customer_firewall_logs { file("/logs/$TLD/$DOMAIN/$SUBDOMAIN/$R_YEAR-$R_MONTH-$R_DAY.log"); }; WHERE $TLD is the tld of the hostname, $DOMAIN is the domain from the hostname, and $SUBDOMAIN is teh subdomain from the hostname. I would like to be able to regex this information out of the hostname to use in the destination. Is this possible and any tips on getting it to work correctly would be greatly appreciated.

3 6

TCP Wrappers
by Mike 11 Jul '08

11 Jul '08

hello all, it seems that TCP Wrappers can be enabled by default when compiling syslog-ng 2.0.9, despite what the documentation says. >From what I can see, the configure script looks for the existance of the libwrap libraries, and if they exist it will enable support (added in syslog-ng 2.0.3). would it be possible to either update the documents to mention that libwrap is not disabled by default, or maybe make it so you do actually have to manually enable libwrap? (compiling on RedHat ES4, which has the libwrap devel libraries) thanks! Mike

2 3

Using regexp in match()
by Jan Kreps 09 Jul '08

09 Jul '08

Hi All. I'm trying to setup central syslog-ng server for my Exchange servers. On windows servers I use Epilog agent (brother of Snare) forwarding tracking logs to central syslog. And I want to feed the logs into MySQL. Problem is that event sended by Epilog consists from four fields: Date, System, Type and Strings. But I don't need them. Actual content of tracking log -- some 20 fileds -- is hidden in the String. So I need to parse the String to obtain the fields I'm interested in. According to syslog-ng admin guide: The host(), match(), and program() filter functions accept extended regular expressions (also called POSIX modern regular expressions) as parameters. The regular expressions can use up to 255 regexp matches ($1 ... $255). For case-insensitive searches, start the expression with the (?i) string. So I tried regexps to do this, but without success. My idea was to use grouping regexp, which would remmember matches in $1 .. $20 and use these in template section. Say the format is: filed1\011field2\011, so fields in String are separated by '\011'. I tried: filter f_parse { match("([^\\011]*)\\011([^\\011]*)\\011"); }; but when i try use $1 and $2 afterwards in template they seem empty Am I completely wrong with this approach, or I missed something? Thanks in advance Jan

2 1

Re: [syslog-ng] loop caused by syslog-ng filter
by JUNG, Christian 07 Jul '08

07 Jul '08

Okay. Don't know if I get the things right, but: All scripts you've mailed to the list write something to a file or send a mail if they're called and terminate directly after that action. syslog-ng will respawn (neu starten) them directly. The program started by the program destination should not terminate itself. It has to listen on STDIN for a log message given from syslog-ng. Afterwards it can do something useful and then it has to listen for the next message. In a shell script you can do this with a "while read LINE; do ...; done". Try this: ---8<--- #!/bin/bash while read LINE; do echo $(date) $LINE >> /tmp/schrott done ---8<--- Execute this script on the command line, enter some random stuff and look into /tmp/schrott: user@box:~> ./test-script bla bla bla bla Terminate this script by pressing CTRL-D. You should see something like this in /tmp/schrott: Thu Mar 20 15:13:16 CET 2008 bla bla bla Thu Mar 20 15:13:18 CET 2008 bla If not your script doesn't work :-) See for typos. If this works, put it in your syslog-ng conf. It should do the same (only prio 1 messages should be visible with the date prepended). If this works well, try this script first on the command line: ---8<--- #!/bin/bash while read LINE; do mail -s "High Priority Snort Alert" Sub-Zero(a)xxx.de <<-EOF Alert, Priority 1 $LINE EOF done ---8<--- You should receive for every given input line exactly one mail. If this works put it in your syslog-ng.conf. Now you should be done. The thread you mentioned is about two running scripts where only one should run. This shouldn't be the case here. But you can have a look at the output of "ps fax". If you see multiple processes under syslog-ng then you might have the problem. bye Chris > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of Adam Richter > Sent: Thursday, March 20, 2008 2:46 PM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] loop caused by syslog-ng filter > > > Hi! > > Not working! Syslog-ng filters for exactly the string: [Priority: 1] > and not as it is piped by the mail script: #priority 1# note the ":" ! > Anyway I used your script --> same fault!!! > I have also used following script: > > > #!/bin/sh > > echo AAA >> /tmp/schrott > date >> /tmp/schrott > > There is no output like [Priority: 1]!!! > Then I did following: tail -f /tmp/schrott and got a loop > too! The loop starts when syslog-ng recognices the first > machting string [Priority: 1] and loops till I stop syslog-ng!!! > > > Sensor1:~# /etc/init.d/syslog-ng stop > Stopping system logging: syslog-ng. > Sensor1:~# tail -f /tmp/schrott > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > > > I think it has something in common with this thread: > > https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009454.html > > Any other ideas? It´s very important! > > German: Es handelt sich hier um meine Abschlussprüfung, und > dies ist der letzte Fehler der Auftritt, ansonsten läuft das Projekt. > > bye, Adam / Sub-Zero ! > > > -------- Original-Nachricht -------- > > Datum: Thu, 20 Mar 2008 10:32:31 +0100 > > Von: "JUNG, Christian" <christian.jung(a)saarstahl.com> > > An: "Syslog-ng users\' and developers\' mailing list" > <syslog-ng(a)lists.balabit.hu> > > Betreff: Re: [syslog-ng] loop caused by syslog-ng filter > > > Hi Adam, > > > > syslog-ng does the right thing :-). > > > > It starts the program/script once and pipes on STDIN every > log-message > > which matches the filter. > > > > If your script is started, it will call mail and pipe > "Alert, priority 1" > > to its STDIN and then exits. syslog-ng sees this and > restarts it (version > > 2.0 or higher behave that way, see > > > <http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ > ch09s02.html#reference_destination_program>). > > > > For you purpose this would be better: > > > > ---8<--- > > #!/bin/bash > > > > while read LINE; do > > cat <<-EOF | mail -s "High Priority Snort Alert" > Sub-Zero(a)xxx.de > > Alert, Priority 1 > > $LINE > > EOF > > done > > ---8<--- > > > > > > bye > > Chris > > > > > > > > -----Original Message----- > > > From: syslog-ng-bounces(a)lists.balabit.hu > > > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of > Adam Richter > > > Sent: Thursday, March 20, 2008 9:23 AM > > > To: syslog-ng(a)lists.balabit.hu > > > Subject: [syslog-ng] loop caused by syslog-ng filter > > > > > > > > > Hi! > > > > > > First off, sorry for my poor english! > > > I have a problem with a loop caused by syslog-ng v. 2.0.8. I > > > have set up Snort as an IDS System. Snort writes its messages > > > in unified-format to /var/log/snort/snort.alert and > > > /var/log/snort/snort.log. There are two Barnyard processes > > > which read the unified files and convert it to messages that > > > syslog and MySQL understand. Syslog-ng writes the messages > > > to /var/log/auth.log. All this is working fine. Now, I want > > > to set up a filter for Priority 1 alerts. This alert should > > > be send to the Administrator. > > > > > > I used following filter for syslog-ng: > > > > > > source src {unix-stream("/dev/log"); internal();}; > > > destination email{program("/usr/local/bin/alert_mail.sh");}; > > > filter high {match("[Priority: 1]");}; > > > log {source(src);filter(high); destination(email);}; > > > > > > > > > The alert_mail.sh: > > > > > > #!/bin/sh > > > cat << EOF | mail -s "High Priority Snort Alert" Sub-Zero(a)xxx.de > > > Alert, Priority 1 > > > EOF > > > > > > > > > Then I use Nessus to cause some alerts with Priority 1. I can > > > see 4 alerts with the Priority 1 with BASE and in > /var/log/auth.log. > > > > > > Syslog-ng recognises the alert with Priority 1 and activates > > > the script /usr/local/bin/alert_mail.sh > > > > > > All this is working, but the script is restarted by syslog-ng > > > again an again. > > > > > > Extract from /var/log/messages: > > > > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Closing log writer > > > fd; fd='11' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > … > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='256' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Starting > > > destination program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='256' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Starting > > > destination program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; > > > > > > … > > > > > > > > > I get thousands of mails per minute till I stop syslog-ng. > > > > > > Output of /var/log/auth.log(so y see that syslog-ng writes > > > snort/barnyard messages correctly to auth.log): > > > > > > Mar 19 13:56:54 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:4758 -> 172.28.100.10:137 > > > Mar 19 13:57:13 src@Sensor1 barnyard: [1:1446:8] SMTP vrfy > > > root [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:1085 -> 172.28.100.10:25 > > > Mar 19 13:57:13 src@Sensor1 barnyard: [1:660:11] SMTP expn > > > root [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:1085 -> 172.28.100.10:25 > > > Mar 19 13:57:22 src@Sensor1 barnyard: [1:12626:2] Snort Alert > > > [1:12626:0] [Classification: Decode of an RPC Query] > > > [Priority: 2] {UDP} 172.25.1.152:1146 -> 172.28.100.10:111 > > > Mar 19 13:57:22 src@Sensor1 barnyard: [1:585:9] RPC portmap > > > sadmind request UDP [Classification: Decode of an RPC Query] > > > [Priority: 2] {UDP} 172.25.1.152:1146 -> 172.28.100.10:111 > > > Mar 19 13:57:24 src@Sensor1 barnyard: [1:566:6] POLICY > > > PCAnywhere server response [Classification: Misc activity] > > > [Priority: 3] {UDP} 172.25.1.152:1155 -> 172.28.100.10:5632 > > > Mar 19 15:11:27 src@Sensor1 barnyard: [122:1:0] portscan: TCP > > > Portscan [Classification: Unknown] [Priority: 3] {PROTO255} > > > 172.25.1.152 -> 172.28.100.10 > > > Mar 19 15:11:58 src@Sensor1 barnyard: [1:1420:13] SNMP trap > > > tcp [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:162 > > > Mar 19 15:11:58 src@Sensor1 barnyard: [1:1418:13] SNMP > > > request tcp [Classification: Attempted Information Leak] > > > [Priority: 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:161 > > > Mar 19 15:12:05 src@Sensor1 barnyard: [1:1421:13] SNMP > > > AgentX/tcp request [Classification: Attempted Information > > > Leak] [Priority: 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:705 > > > Mar 19 15:12:16 src@Sensor1 barnyard: [122:1:0] portscan: TCP > > > Portscan [Classification: Unknown] [Priority: 3] {PROTO255} > > > 172.25.1.152 -> 172.28.100.10 > > > Mar 19 15:12:19 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > Mar 19 15:12:20 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > Mar 19 15:12:22 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > > > > > > > I think it has something in common with this topic: > > > > > > https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009454.html > > > > Thanks in advance! > > > > Sub-Zero > > > > > > -- > > GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! > > Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx > > ______________________________________________________________ > > ________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > -- Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! http://games.entertainment.gmx.net/de/entertainment/games/free ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

4 10

migrate over to PCRE?
by Balazs Scheidler 04 Jul '08

04 Jul '08

Dear syslog-ng users, The following text was written on my blog, but I'm reposting it here for a wider audience to read: As of now the development of the generic rewrite feature has been completed in one of my private git repositories. The new code uses PCRE and I'm somewhat undecided how to move forward with PCRE. For those who might not know PCRE is an implementation of regular expressions and is an acronym for "Perl Compatible Regular Expressions". PCRE adds a lot more features and seems to perform better than its POSIX equivalent. So the situation is as follows: * various filters use POSIX regexps * rewrite uses PCRE This is not a very consistent combination, thus I'm planning to add PCRE support for filters too. The only question is whether it is needed to have two independent regexp styles in syslog-ng in the long run. If I decide that one of them is enough, then I'd deprecate POSIX style regexps in filters and wouldn't implement POSIX in rewrite rules. This combination would yield a syslog-ng that would give warnings when POSIX-style regular expressions are in use and in a forthcoming release I'd change the default regexp style to PCRE, and yet another syslog-ng release later, I'd phase out POSIX completely. If the decision is to keep them both in the long run, it would mean that I'd need to implement POSIX style regexps for rewrite rules as well. This would probably the least intrusive for users, but also a lot more work. Also, this would allow adding other filtering options like globbing or prefix search. What do you think? Is the addition of modular search algorithms worth it? -- Bazsi

3 3

NetScreen Statistics
by lists＠grounded.net 04 Jul '08

04 Jul '08

Anyone using syslog-ng to gather NetScreen log information? Specifically, I'd like to have real time information along with statistical information, analysis of the logs. I would guess some others are doing this. So far, I've not been able to send the logs to syslog-ng, perhaps it's using udp, I need to find more information. Mike

3 9

Re: [syslog-ng] Multi-line Logs
by jrhendri＠maine.rr.com 03 Jul '08

03 Jul '08

Hi, I have been following this thread somewhat. I am facing a similar problem with how to handle logs from OpenVMS OPCOM messages. They are also multi-line. We are having a fowarder written for the VMS side that will send these messages across (still looking at TVP v. UDP and whether or not to place syslog headers in the messages). The goal is to be able to (at minimum) retain them on the NG box (for log retention) and also forward them as-is to a remote system (for processing by a security operations center). Currently, one possibility we are considering is to send the messages as UDP without stripping out the newlines (essentially defining a terminal that will receive the messages, and tying this to an outbound UDP socket pointed at the NG logserver. If you have any thoughts/suggestions, I would appreciate hearing them. If not, I apologize for wasting bandwidth :-) Thanks!! Jim Hendrick GCFW, GCIA, GCIH, GCWN, GCUX jrhendri(a)maine.rr.com

2 1

Multi-line Logs
by Enigma 03 Jul '08

03 Jul '08

I have a host that sends mult-line messages (embedded newlines, not separate syslog messages) from what I can tell syslog-ng strips out all the newlines and replaces them with a space. Is there anyway to disable or modify (replace them with something else that can be easily s/// in post-processing) this functionality without changing the code and compiling from source? I have been through the syslog-ng manual and asked Mr. Google and I cannot find anything on this topic. Thanks...

2 4