March 2006 - syslog-ng - lists.balabit.hu

Multiple IP filters
by R King 18 Mar '06

18 Mar '06

I am trying to setup filters for multiple network devices. Below are excerts to my syslog-ng.conf file. filter f_netdev { host("192.168.23.1") or host("192.168.23.2") or host("192.168.23.4") or host("192.168.23.8") or host("192.168.23.40") or host("192.168.23.75"); }; destination d_host-date { file("/var/log/network/$HOST/$MONTH$DAY" group(watch) perm(0640)); }; log { source(s_udp); filter(f_netdev); destination(d_host-date); }; The problem is when I log host IPs, the filter for "192.168.23.1" includes "192.168.23.11", "192.168.23.12", "192.168.23.135", etc... How can I specify 192.168.23.1 only? --------------------------------- Yahoo! Mail Use Photomail to share photos without annoying attachments.

2 1

Running ON INSERT UPDATE (MySQL)
by Dukes Clayton 17 Mar '06

17 Mar '06

Hii, I'm trying to update a table when I do an insert, do I have the syntax correct here? destination db_mysql { pipe("/var/log/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, datetime, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC','$MSG' ) ON INSERT UPDATE NumRows SET NumRowsLogs=NumRowsLogs+1;\n") template-escape(yes)); }; TIA! -Clayton

2 1

Solaris syslog-ng tuning
by Cary, Kim 17 Mar '06

17 Mar '06

In summary, tuning the solaris udp hiwat and max_buf had no effect on the continuous udpInOverflows loss. At present, changing some parameters in syslog-ng.conf has helped, it seems. We have now gone from log_fifo_size(100000); gc_busy_threshold(3000000); gc_idle_threshold(300); and having A1) udpInOverFlows increment by 3000 every 4 sec A2) Garbage collecting while busy for 3 sec every half hour at peak to log_fifo_size(1000000); # up an order of magnitude gc_busy_threshold(300000); # down an order of magnitude gc_idle_threshold(500); # log was reporting objects 310-322 range, so some headroom and having B1) udpInOverflows increment by 50 every half hour B2) Garbage collecting while busy for a fraction of a second every 4-8 minutes This seems better. Hope that helps someone out there. My first question - we are losing messages during GC while busy - do we only lose the messages that arrive between these two log entries: Mar 16 20:00:30 syslog-ng[9688]: Garbage collecting while busy... Mar 16 20:00:31 syslog-ng[9688]: Objects alive: 310, garbage collected: 299695 My second question is, what can we do better to tune this further? Thanks to all (& Balasz for the wonderful software).

2 1

Re: syslog-ng Digest, Vol 11, Issue 23
by dwarnock＠harbrook.net 16 Mar '06

16 Mar '06

I am on holiday on Wednesday 15th March returning on Thursday 16th. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734. Regards David Warnock

1 0

sylog-ng hangs after upgrade to 1.9.9
by Cristian Livadaru 16 Mar '06

16 Mar '06

Hi, just did an apt-get upgrade today and installed syslog-ng 1.9.9 after this I was fighting some hours with my system. syslog-ng logged a few minutes then it hanged, all logfiles showing the same timestamp. This also caused my mail system ( postfix ) to hang and also caused other problems. After a restart of syslog-ng the problem was gone, also 2 times sylog-ng started logging again after about 15-20 minutes for a short while. I found something on a old mailinglist about 2 sources from the same unix-stream which was the case in my logfile. I removed the second one but the problem still occured. I have also read something about klog runing, which isn't the case on my machine. I have now downgraded syslog-ng to 1.6.9 and it's working fine again. My system is a Debian linux Linux 2.4.31-grsec #21 SMP Fri Dec 2 08:49:48 CET 2005 i686 athlon-4 i386 GNU/Linux Best regards, Cristian Livadaru

3 6

snmptrapd to syslog-ng
by Mike Tremaine 15 Mar '06

15 Mar '06

Does anyone have a simple snmptrapd.conf that can push the traps to syslog-ng (via shell script, logger, pipe or whatever?) Sorry if this has been asked before but I didn't see in the FAQ -Mike

1 1

Regex subpattern macros [1.9.9]
by Mel Collins 15 Mar '06

15 Mar '06

Hi, I'm using syslog-ng's match filter to extract parts of messages for use in parts of the destination; specifically in the log file name and template sections. The messages sent to syslog-ng for this purpose are all of the format: "<filename> <message>" (plus whatever sl-ng prepends). .conf extract: filter f_logsplitter { match ("^.*?: ([^ ]*) (.*)$"); }; destination d_logfiles { file ("/our/logs/$1" template ("$2\n") ... etc. Eg. For the message "root: foo/bar.log some text", the log file /our/logs/foo/bar.log gets "some text" appended to it. Now this seems to work fine for low traffic, but if I send 10k messages in a tight loop - either locally or remotely via TCP - $2 contains the wrong string for many/most of them, eg: 07/03/2006 16:56:34 SEEK #3 07/03/2006 16:56:35 SEEK #3 07/03/2006 16:56:35 SEEK #3 07/03/2006 16:56:35 SEEK #7 07/03/2006 16:56:35 SEEK #7 07/03/2006 16:56:35 SEEK #7 07/03/2006 16:56:35 SEEK #7 07/03/2006 16:56:35 SEEK #8 07/03/2006 16:56:35 SEEK #11 07/03/2006 16:56:35 SEEK #11 07/03/2006 16:56:35 SEEK #11 ...where the #number should be consecutive. If I use $MSGONLY, or such, the correct message line is used. It seems to me like the subpattern macros - $1, $2, etc - aren't maintained on a per-message basis, but are global..? Given that they aren't documented (that I can find), are the presence of $1, etc. an unintended side-effect of using a regex? Am I doing something I shouldn't by using them? Or did I find me a bug? ;o) Takk, - Mel C

3 6

Re: syslog-ng Digest, Vol 11, Issue 22
by dwarnock＠harbrook.net 15 Mar '06

15 Mar '06

I am on holiday on Wednesday 15th March returning on Thursday 16th. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734. Regards David Warnock

1 0

DOMAIN Name Variable
by Breitman, Jason 15 Mar '06

15 Mar '06

I am using the below line in my syslog-ng configuration file. destination maillog { file("/var/log/clients/$YEAR/$MONTH/$HOST/mail.log"); }; I am looking to add a variable for the domain name and have been unable to find one. Can someone help me out? Thank you. Jason Breitman jason.breitman(a)blackrock.com THE INFORMATION CONTAINED IN THIS MESSAGE AND ANY ATTACHMENT MAY BE PRIVILEGED, CONFIDENTIAL, PROPRIETARY OR OTHERWISE PROTECTED FROM DISCLOSURE. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this message and any attachment is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and permanently delete it from your computer and destroy any printout thereof.

6 8

RE: [syslog-ng] solaris UDP loss
by SOLIS, ALEX 14 Mar '06

14 Mar '06

What exactly is Garbage collecting? I don't get those in my logs. Alex ________________________________ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Cary, Kim Sent: Tuesday, March 14, 2006 12:30 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] solaris UDP loss Finally got around to reading the docs and applying the recommendations. FIRST - thanks much! NOW - my results: # ndd /dev/udp udp_recv_hiwat 65536 # ndd /dev/udp udp_max_buf 1073741824 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:16 udpInCksumErrs = 0 udpInOverflows =677996405 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:19 udpInCksumErrs = 0 udpInOverflows =677996571 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:22 udpInCksumErrs = 0 udpInOverflows =677996726 I'm pretty sure this is just the Firewall sending more than 55M lines of connection/deny logs over UDP. IF I was to switch the FW over to TCP logging from UDP, would this be a better solution? BTW, here are a few words from syslog-ng: Mar 14 09:32:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:42:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:52:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:02:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:04:53 syslog-ng[20685]: Garbage collecting while busy... Mar 14 10:04:58 syslog-ng[20685]: Objects alive: 310, garbage collected: 2999695 Mar 14 10:12:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:16:22 syslog-ng[20685]: Garbage collecting while idle... Mar 14 10:16:24 syslog-ng[20685]: Objects alive: 318, garbage collected: 1129970 Mar 14 10:22:32 syslog-ng[20685]: STATS: dropped 0 and its config: source s_udp_inetpix { udp(ip(192.168.0.1)); }; On Mar 7, 2006, at 3:01 PM, syslog-ng-request(a)lists.balabit.hu wrote: | 1) Am I reading that loss right?? Probably, you might however want to snoop on the interface to see what kind of udp packets come on your interface. | 2) Any tips from Solaris/syslog-ng tuners would be appreciated! udp_max_buf does not set the queue length of the udp socket, which by the way can have a different value for each socket... You could have a look at: http://sunsolve.sun.com/search/document.do?assetkey=1-30-3218-1 basically: increasing udp_max_buf without increasing udp_recv_hiwat has no meaning. Furthermore, you can increase you socket buffer that way up to 64k (Solaris 8 & 9), if you want to increase it further up you must use the setsockopt call (up to udp_max_buf which has a maximum value of 1GB). Here is the official SUN documentation regarding this: http://docs.sun.com/app/docs/doc/817-0404/6mg74vsb5?a=view#gbtag Now regarding your packet loss issue. I would increase udp_recv_hiwat -> 65536 udp_max_buf -> 1073741824 (you will never get here anyway) Then I would try to play with syslog-ng config: log_fifo_size, log_iw_size and log_fetch_limit. But here I'd appreciate a syslog-ng expert to step in and tell us what to do more preceisely. This e-mail contains Omaha Public Power District's confidential and proprietary information and is for use only by the intended recipient. Unless explicitly stated otherwise, this e-mail is not a contract offer, amendment, nor acceptance. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

2 1