- syslog-ng - lists.balabit.hu

Flow-control behaviour and Cached lines timestamping, plus a suspend error in 2.1beta1
by Behal, Pavel 05 Jun '08

05 Jun '08

Dear Bazsi, I have found two "features" and one bug I would like to discuss a little: 1) We use the syslog-ng 2.1alpha1 as a root system syslog. The problem is, we have had two destinations setup. First destination was the traditional write of system messages to the file like "/var/log/messages" etc. The second destination path was to the TCP stream to the central syslog server. And the problem is, we have used the flow-control flag on the central TCP path. But, when the central TCP syslog-ng becomes unavailable, after all the buffers filled up, the local system syslog-ng stopped processing any system messages. It did not write anything through the local file destinations until the TCP communication was re-established. Is this a correct behaviour? We have found, that to allow smooth processing of local system sources, we can not use flow-control flag on any destination paths it used.. 2) When messages are in syslog-ng buffer, they do not contain any timestamp. We have realized, that the timestamp is added at the time the syslog-ng flushes all the buffers to the disk. This is not very good in case you need to debug the messages ages and all of them have same and non-related timestamp. Is it possible to modify the logic of syslog-ng to timestamp the messages as soon as possible - even before the buffer, at the collection time? We have found it important for log integrity reasons. 3) Directly in the 2.1beta1 there has been introduced some error with the "suspend writing a destination file when an I/O error ..." patch. During our tests we have found it continously thinks the destination is unavailable and writes strange messages into to the internal log: May 29 15:00:02 europa syslog-ng[16798]: I/O error occurred while writing; fd='7', error='Connection timed out (110)' May 29 15:00:02 europa syslog-ng[16798]: Connection broken; time_reopen='60' May 29 15:00:02 europa syslog-ng[16798]: Suspending write operation because of an I/O error; fd='7', time_reopen='60' the lsof -p 16798: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME syslog-ng 16798 root cwd DIR 253,0 4096 2 / syslog-ng 16798 root rtd DIR 253,0 4096 2 / syslog-ng 16798 root txt REG 253,0 114584 671841 /sbin/syslog-ng syslog-ng 16798 root mem REG 253,0 106397 820372 /lib/ld-2.3.4.so syslog-ng 16798 root mem REG 253,0 1454546 820373 /lib/tls/libc-2.3.4.so syslog-ng 16798 root mem REG 253,0 28504 2919694 /usr/lib/libwrap.so.0.7.6 syslog-ng 16798 root mem REG 253,0 93985 820376 /lib/tls/libpthread-2.3.4.so syslog-ng 16798 root mem REG 253,0 47671 820386 /lib/tls/librt-2.3.4.so syslog-ng 16798 root mem REG 253,0 505200 2930064 /usr/lib/libglib-2.0.so.0.400.7 syslog-ng 16798 root mem REG 253,0 95148 820384 /lib/libnsl-2.3.4.so syslog-ng 16798 root mem REG 253,0 36639 2920956 /usr/local/lib/libevtlog.so.0.0.0 syslog-ng 16798 root 0r CHR 1,3 1619 /dev/null syslog-ng 16798 root 1w CHR 1,3 1619 /dev/null syslog-ng 16798 root 2w CHR 1,3 1619 /dev/null syslog-ng 16798 root 3r REG 0,3 0 4026531850 /proc/kmsg syslog-ng 16798 root 4u unix 0xc1be7ba0 351802 /dev/log syslog-ng 16798 root 5u IPv4 351804 TCP europa.training.hp.com:37205->europa.training.hp.com:5140 (ESTABLISHED) syslog-ng 16798 root 6w REG 253,0 4802 1655155 /var/log/messages syslog-ng 16798 root 8w REG 253,0 525 1655159 /var/log/boot.log This happens during standard system syslog processing on RH ELinux 4 and Solaris 9. You have the syslog-ng confs attached. Could you pleae check it? Best Regards, Pavel

3 3

Re: [syslog-ng] file() sourcing application logs
by Alan Edmonds 04 Jun '08

04 Jun '08

We are doing something similar; we use tomcat and apache. For apache, something like this (we are using 2.0): CustomLog "|/usr/bin/logger -p local1.info -t apache-access " access-format Then on syslog-ng I use the "tag" to point it to the right log file. For tomcat, that is tougher; stack dumps are a curse to syslog. There is a hook in the Catalina startup scripts to create a named pipe for gc.out and tomcat.out. The it spawns a logger -p local2.info < gc.out-named-pipe thingy to send to syslog-ng. I can send more details if you need them. Alan T-Mobile International UK Limited Company Registered Number: 3951860 Registered Office Address: Hatfield Business Park, Hatfield, Hertfordshire, AL10 9BW Registered in England and Wales NOTICE AND DISCLAIMER This email (including attachments) is confidential. If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose

2 1

file() sourcing application logs
by Chris Pratt 04 Jun '08

04 Jun '08

Hi syslog-ng, Been all over google and still not found what I need so I'm hoping to get some help here... I have syslog-ng capturing and duplicating all of my *normal* /var/logs. I still need to capture some application specific logs that are not using syslog at all currently but dumping to text in application log directories. I wanted to use the file() source but I am not getting the results I wanted. I have tried both of these options to get these files logged: # See http://www.syslog.org/syslog-ng/v2/#id2535001 # tail -f [file log] | logger -p local#.info # # See http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch09s01.html # source s_tail { file("/var/log/apache/access.log" follow_freq(1) flags(no-parse)); }; but I am running into two issues: 1.) Either above way of sourcing the files results in each line in my application log showing up as a separate syslog event even though it is a single application event that wraps lines in its own application log. For example: {Heap before GC invocations=1087 (full 0): par new generation total 307200K, used 219907K [0x5af10000, 0x73f10000, 0x73f10000) eden space 204800K, 100% used [0x5af10000, 0x67710000, 0x67710000) from space 102400K, 14% used [0x6db10000, 0x6e9d0d38, 0x73f10000) to space 102400K, 0% used [0x67710000, 0x67710000, 0x6db10000) concurrent mark-sweep generation total 991232K, used 116200K [0x73f10000, 0xb0710000, 0xb0710000) concurrent-mark-sweep perm gen total 71680K, used 46017K [0xb0710000, 0xb4d10000, 0xb4d10000) 27877.615: [GC 27877.615: [ParNew Desired survivor size 94371840 bytes, new threshold 15 (max 15) - age 1: 32802608 bytes, 32802608 total - age 2: 604752 bytes, 33407360 total - age 3: 40968 bytes, 33448328 total - age 4: 50728 bytes, 33499056 total - age 5: 54952 bytes, 33554008 total - age 6: 36096 bytes, 33590104 total - age 7: 47128 bytes, 33637232 total - age 8: 44048 bytes, 33681280 total - age 9: 40976 bytes, 33722256 total - age 10: 53912 bytes, 33776168 total - age 11: 63680 bytes, 33839848 total - age 12: 90944 bytes, 33930792 total - age 13: 33560 bytes, 33964352 total - age 14: 60552 bytes, 34024904 total - age 15: 59328 bytes, 34084232 total : 219907K->33808K(307200K), 0.0730690 secs] 336108K->150064K(1298432K), 0.0732270 secs] Heap after GC invocations=1088 (full 0): par new generation total 307200K, used 33808K [0x5af10000, 0x73f10000, 0x73f10000) eden space 204800K, 0% used [0x5af10000, 0x5af10000, 0x67710000) from space 102400K, 33% used [0x67710000, 0x69814218, 0x6db10000) to space 102400K, 0% used [0x6db10000, 0x6db10000, 0x73f10000) concurrent mark-sweep generation total 991232K, used 116256K [0x73f10000, 0xb0710000, 0xb0710000) concurrent-mark-sweep perm gen total 71680K, used 46017K [0xb0710000, 0xb4d10000, 0xb4d10000) } becomes something like... 2008-06-03T14:27:40-04:00 s_catalinag@tomcat {Heap before GC invocations=1087 (full 0): 2008-06-03T14:27:40-04:00 s_catalinag@tomcat par new generation total 307200K, used 219907K [0x5af10000, 0x73f10000, 0x73f10000) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat eden space 204800K, 100% used [0x5af10000, 0x67710000, 0x67710000) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat from space 102400K, 14% used [0x6db10000, 0x6e9d0d38, 0x73f10000) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat to space 102400K, 0% used [0x67710000, 0x67710000, 0x6db10000) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat concurrent mark-sweep generation total 991232K, used 116200K [0x73f10000, 0xb0710000, 0xb0710000) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat concurrent-mark-sweep perm gen total 71680K, used 46017K [0xb0710000, 0xb4d10000, 0xb4d10000) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat 27877.615: [GC 27877.615: [ParNew 2008-06-03T14:27:40-04:00 s_catalinag@tomcat Desired survivor size 94371840 bytes, new threshold 15 (max 15) 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 1: 32802608 bytes, 32802608 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 2: 604752 bytes, 33407360 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 3: 40968 bytes, 33448328 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 4: 50728 bytes, 33499056 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 5: 54952 bytes, 33554008 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 6: 36096 bytes, 33590104 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 7: 47128 bytes, 33637232 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 8: 44048 bytes, 33681280 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 9: 40976 bytes, 33722256 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 10: 53912 bytes, 33776168 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 11: 63680 bytes, 33839848 total 2008-06-03T14:27:40-04:00 s_catalinag@tomcat - age 12: 90944 bytes, 33930792 total 2008-06-03T14:27:41-04:00 s_catalinag@tomcat - age 13: 33560 bytes, 33964352 total 2008-06-03T14:27:41-04:00 s_catalinag@tomcat - age 14: 60552 bytes, 34024904 total 2008-06-03T14:27:41-04:00 s_catalinag@tomcat - age 15: 59328 bytes, 34084232 total 2008-06-03T14:27:41-04:00 s_catalinag@tomcat : 219907K->33808K(307200K), 0.0730690 secs] 336108K->150064K(1298432K), 0.0732270 secs] 2008-06-03T14:27:41-04:00 s_catalinag@tomcat Heap after GC invocations=1088 (full 0): 2008-06-03T14:27:41-04:00 s_catalinag@tomcat par new generation total 307200K, used 33808K [0x5af10000, 0x73f10000, 0x73f10000) 2008-06-03T14:27:41-04:00 s_catalinag@tomcat eden space 204800K, 0% used [0x5af10000, 0x5af10000, 0x67710000) 2008-06-03T14:27:41-04:00 s_catalinag@tomcat from space 102400K, 33% used [0x67710000, 0x69814218, 0x6db10000) 2008-06-03T14:27:41-04:00 s_catalinag@tomcat to space 102400K, 0% used [0x6db10000, 0x6db10000, 0x73f10000) 2008-06-03T14:27:41-04:00 s_catalinag@tomcat concurrent mark-sweep generation total 991232K, used 116256K [0x73f10000, 0xb0710000, 0xb0710000) 2008-06-03T14:27:41-04:00 s_catalinag@tomcat concurrent-mark-sweep perm gen total 71680K, used 46017K [0xb0710000, 0xb4d10000, 0xb4d10000) 2008-06-03T14:27:41-04:00 s_catalinag@tomcat } 2.) I also have a couple of log files that use the date in their name but from what I can tell macros only work with destinations, not sources? I'd like to do something like: #source s_tomcat { file("/usr/local/server/apache-tomcat-5.5.20/logs/tomcat.${YEAR}-${MONTH}-${DAY}.log"); }; to accommodate this. Anyone have any similar app log setup and a solution to these issues? TIA, Chris

1 0

Source optional() option causes syntax error
by Clay Loveless 04 Jun '08

04 Jun '08

Reposting in its own thread: I'm using a pipe source, and would like to make it optional, however the optional(yes) parameter seems to be considered a syntax error. Here's the entry: source test { pipe("/tmp/test_fifo" optional(yes) flags(no-parse) log_msg_size(1024)); }; Am I doing this incorrectly? This line is considered a syntax error by syslog-ng 2.0.9. -Clay -- Killersoft.com

1 0

Re: [syslog-ng] (no subject)
by Ivey, Chris 04 Jun '08

04 Jun '08

I ran into a similar issue with Netcool from syslog-ng, thinking the microsecond portion of the message was the hostname. What we found out is that there is a bug in the WCs. If you upgrade your WCs, you will see the hostnames coming in with your syslog messages (which you don't currently get, hence why your system is not working as expected). Hope that helps!! Chris Ivey Affiliated Computer Services Enterprise Management Integration Services Infrastructure Management Senior Analyst chris.ivey(a)acs-inc.com "I have not failed, I have simply found 10,000 ways which do not work!" -- Thomas Edison "When you find yourself in a hole, the best thing to do is stop digging!" -- Nick Stokes "I reject your reality, and substitute my own!" -- Adam Savage -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of chris packham Sent: Tuesday, June 03, 2008 4:59 PM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] (no subject) What version of syslog-ng does your distro use? (run syslog-ng --version to find out) >From looking at the source code history support for this Cisco extension was added in v2.0.5. I haven't got a Cisco device handy so I can't confirm that it is working but there is code to deal with the fraction of a second scenario. -----Original Message----- From: Andy Kanyer <AKanyer(a)directs.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] (no subject) Date: Tue, 3 Jun 2008 09:16:30 -0500 Hello everyone, I am currently workin on setting up a debian box as a central syslog server. One goal of this server is to filter syslog messages into different folders based on what server they were sent by. This works as expected with all devices EXCEPT for my cisco wireless controllers. When they log: [CODE] Jun 02 20:52:29.063 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:19:d2:78:ee:8f Jun 02 20:52:09.663 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:13:e8:d9:9d:eb Jun 02 20:50:49.064 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:19:d2:78:ee:8f [/CODE] My syslog-ng, rather than filtering these into a folder given by the wireless controllers ip/hostname, creates folders named .063 and .064. It seems like syslog-ng is reading the microsecond portion of the timestamp as the hostname! After sniffing some other syslog messages, I noticed that ONLY these cisco devices have timestamps that include microseconds. Does anyone have any idea how to work around this and filter the cisco messages by hostname? Someway to truncate the timestamp or force it to look further to find the actual hostname? Just FYI I will post the relevant portion of my syslog-ng.conf: [CODE] source remote_src { udp(); tcp(); }; destination remote_syslog { file( "/usr/local/syslog/$HOST/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; ### added 4-16-08 all three wireless controllers destinations ### manually creating directory names destination wir-c-syd-3-00 { file( "/usr/local/syslog/wir-c-syd-3-00/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; filter wir-c-syd-3-00 { netmask(172.25.198.10/32); }; log { source(remote_src); filter(wir-c-syd-3-00); destination(wir-c-syd-3-00); }; [/CODE] ____________________________________________________________________________ __ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ____________________________________________________________________________ __ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

1 0

local6 doesn't seem to work...
by ruud.feringa＠mondrian.nl 04 Jun '08

04 Jun '08

Hi, When I have the following in syslog-ng.conf: filter f_cyrus { level(warn, err, crit) and facility(local6); }; destination cyrus { file("/var/log/cyrus.log"); }; log { source(src); filter(f_cyrus); destination(cyrus); }; nothing is being logged in /var/log/cyrus.log however when I use: filter f_cyrus { level(warn, err, crit) and match("imap"); }; it works fine, why is local6 not working?? (cyrus-imap is default compiled with local6 enabled) Running on Suse 10.0 ( Cannot upgrade yet :-( ) with syslog-ng version 1.6.8 Greetings, Ruud Feringa.

3 3

syslog-ng to trigger rancid
by Mladen Tsvetkov 04 Jun '08

04 Jun '08

Hi, Guys, I am new to syslog-ng. I have successfully installed it and integrated with mysql and php. My question is how can I make syslog-ng to start rancid when a syslog configuration message is received from a cisco device: 112578: 1564643: May 15 09:05:19.141 EEST: %SYS-5-CONFIG_I: Configured from console by ******** on vty0 (10.10.10.10). So far I know that I have to use the program() directive. But how can I pass the hostname and other parameters to rancid? Thanks in advance, Longman

3 2

Re: [syslog-ng] (no subject)
by chris packham 03 Jun '08

03 Jun '08

What version of syslog-ng does your distro use? (run syslog-ng --version to find out) >From looking at the source code history support for this Cisco extension was added in v2.0.5. I haven't got a Cisco device handy so I can't confirm that it is working but there is code to deal with the fraction of a second scenario. -----Original Message----- From: Andy Kanyer <AKanyer(a)directs.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] (no subject) Date: Tue, 3 Jun 2008 09:16:30 -0500 Hello everyone, I am currently workin on setting up a debian box as a central syslog server. One goal of this server is to filter syslog messages into different folders based on what server they were sent by. This works as expected with all devices EXCEPT for my cisco wireless controllers. When they log: [CODE] Jun 02 20:52:29.063 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:19:d2:78:ee:8f Jun 02 20:52:09.663 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:13:e8:d9:9d:eb Jun 02 20:50:49.064 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:19:d2:78:ee:8f [/CODE] My syslog-ng, rather than filtering these into a folder given by the wireless controllers ip/hostname, creates folders named .063 and .064. It seems like syslog-ng is reading the microsecond portion of the timestamp as the hostname! After sniffing some other syslog messages, I noticed that ONLY these cisco devices have timestamps that include microseconds. Does anyone have any idea how to work around this and filter the cisco messages by hostname? Someway to truncate the timestamp or force it to look further to find the actual hostname? Just FYI I will post the relevant portion of my syslog-ng.conf: [CODE] source remote_src { udp(); tcp(); }; destination remote_syslog { file( "/usr/local/syslog/$HOST/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; ### added 4-16-08 all three wireless controllers destinations ### manually creating directory names destination wir-c-syd-3-00 { file( "/usr/local/syslog/wir-c-syd-3-00/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; filter wir-c-syd-3-00 { netmask(172.25.198.10/32); }; log { source(remote_src); filter(wir-c-syd-3-00); destination(wir-c-syd-3-00); }; [/CODE] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

1 0

(no subject)
by Andy Kanyer 03 Jun '08

03 Jun '08

Hello everyone, I am currently workin on setting up a debian box as a central syslog server. One goal of this server is to filter syslog messages into different folders based on what server they were sent by. This works as expected with all devices EXCEPT for my cisco wireless controllers. When they log: [CODE] Jun 02 20:52:29.063 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:19:d2:78:ee:8f Jun 02 20:52:09.663 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:13:e8:d9:9d:eb Jun 02 20:50:49.064 1x_auth_pae.c:2488 DOT1X-3-MAX_EAP_RETRIES: Max EAP identity request retries (21) exceeded for client 00:19:d2:78:ee:8f [/CODE] My syslog-ng, rather than filtering these into a folder given by the wireless controllers ip/hostname, creates folders named .063 and .064. It seems like syslog-ng is reading the microsecond portion of the timestamp as the hostname! After sniffing some other syslog messages, I noticed that ONLY these cisco devices have timestamps that include microseconds. Does anyone have any idea how to work around this and filter the cisco messages by hostname? Someway to truncate the timestamp or force it to look further to find the actual hostname? Just FYI I will post the relevant portion of my syslog-ng.conf: [CODE] source remote_src { udp(); tcp(); }; destination remote_syslog { file( "/usr/local/syslog/$HOST/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; ### added 4-16-08 all three wireless controllers destinations ### manually creating directory names destination wir-c-syd-3-00 { file( "/usr/local/syslog/wir-c-syd-3-00/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; filter wir-c-syd-3-00 { netmask(172.25.198.10/32); }; log { source(remote_src); filter(wir-c-syd-3-00); destination(wir-c-syd-3-00); }; [/CODE]

1 0

Re: [syslog-ng] Filtering duplicate messages
by chris packham 03 Jun '08

03 Jun '08

Hi Bazsi, Thanks for the feedback. I'll re-submit a new patch shortly. Is the organisational agreement Anthony mentioned good enough (and current?) for you. On Thu, 2008-05-29 at 22:13 +0200, Balazs Scheidler wrote: > Hi Chris, > > Thanks for your contribution. Please find my comments in-line. > > Please note that in order to get this integrated into syslog-ng, you'll > need to sign a "Contributory License Agreement" which is available on > our website here: > > http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… > > Thanks. > > So after the legalese, let's see your notes and patch below. > > On Thu, 2008-05-29 at 12:57 +1200, chris packham wrote: > > Here's a rough cut of my patch. It still needs a bit of cleanup and I > > have yet to implement the time-based behaviour. I just wanted to get > > some > > feedback. > > > > One particular thing I wanted to clarify is where in the log message > > pipeline would best to attach this behaviour to? I've currently > > attached it to the LogWriter. The disadvantage of that is that the new > > message I generate does not get passed through the filters or templates > > (and we've just done a bunch of wasted processing on a message that > > gets dropped) > > There are basically three options: > 1) sources > 2) as a processing element between sources/destinations (for example > filters, LogProcess in syslog-ng 2.2) > 3) in the LogWriter > > All three has pros and cons: > 1) suppress on input: > pros: > * drop messages as early as possible > * every destination would receive the same suppressions from the same > host > cons: > * would need a host-name indexed hashtable to store host specific > suppression info, this needs memory and CPU > > 2) suppress as a processing element: > pros: > * probably the most flexible, the user could choose which log pipes > have suppressions and which don't > cons: > * it requires the same amount of state as suppressing on input > * it is more complex than any other alternative and the flexibility > it provides is probably not needed > > 3) suppress in the LogWriter > pros: > * the easiest to implement > * no need for per-host state, it uses a per-destination state but > that's already available (every destination has a separate writer) > cons: > * different destinations will have a different set of suppressions > depending on the set of sources that feed the given destination > > I'd say that doing it in LogWriter is a sane choice. We could do better > but that'd cost us memory and CPU for an added flexibility which is > probably not needed. > > > > > > The behaviour implemented is as follows. > > > > As a log message is output to a destination it is remembered > > as the last message seen. > > > > When a new message is to be output it is checked against the > > last message seen. If its contents (message text, including > > hostname and pid if present) are the same the message is dropped > > and a couter incremented. If the message is different to the > > last message seen a new log message indicating the value of the > > last message counter will be inserted into the queue and the new > > message will be processed. > > > > TODO: Implement a time based flush of last seen messages > > > > --- > > src/cfg-grammar.y | 3 +- > > src/cfg-lex.l | 3 +- > > src/logmsg.c | 26 +++++++++++++++++++++++ > > src/logmsg.h | 2 + > > src/logwriter.c | 59 > > ++++++++++++++++++++++++++++++++++++++++++++++++++++- > > src/logwriter.h | 9 ++++++++ > > 6 files changed, 99 insertions(+), 3 deletions(-) > > > > diff --git a/src/cfg-grammar.y b/src/cfg-grammar.y > > index 39ff172..386c231 100644 > > --- a/src/cfg-grammar.y > > +++ b/src/cfg-grammar.y > > @@ -62,7 +62,7 @@ gint last_addr_family = AF_INET; > > %token KW_USER KW_DOOR KW_SUN_STREAMS KW_PROGRAM > > > > /* option items */ > > -%token KW_FSYNC KW_MARK_FREQ K> > +%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_SUPPRESS > > KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE > > > > %token KW_CHAIN_HOSTNAMES KW_NORMALIZE_HOSTNAMES KW_KEEP_HOSTNAME > > KW_CHECK_HOSTNAME KW_BAD_HOSTNAME > > %token KW_KEEP_TIMESTAMP > > @@ -671,6 +671,7 @@ dest_writer_option > > : KW_FLAGS '(' dest_writer_options_flags ')' { > > last_writer_options->options = $3; } > > | KW_LOG_FIFO_SIZE '(' NUMBER ')' { > > last_writer_options->fifo_size = $3; } > > | KW_FLUSH_LINES '(' NUMBER ')' { > > last_writer_options->flush_lines = $3; } > > + | KW_SUPPRESS '(' yesno ')' { > > last_writer_options->suppress= $3; } > > | KW_FLUSH_TIMEOUT '(' NUMBER ')' { > > last_writer_options->flush_timeout = $3; } > > | KW_TEMPLATE '(' string ')' { > > > > last_writer_options->template = cfg_lookup_template(configuration, $3); > > ok. > > > diff --git a/src/cfg-lex.l b/src/cfg-lex.l > > index 5c5f54f..4eb0c63 100644 > > --- a/src/cfg-lex.l > > +++ b/src/cfg-lex.l > > @@ -6,7 +6,7 @@ > > > > > > /*************************************************************************** > > * > > - * Copyright (c) 1999 Balzs Scheidler > > + * Copyright (c) 1999 Bal�zs Scheidler > > * Copyright (c) 1999-2007 BalaBit IT Ltd. > > * > > * This program is free software; you can redistribute it and/or modify > > @@ -86,6 +86,7 @@ static struct keyword keywords[] = { > > { "flush_timeout", KW_FLUSH_TIMEOUT }, > > { "sync_freq", KW_FLUSH_LINES }, /* obsolete */ > > { "sync", KW_FLUSH_LINES }, /* obsolete */ > > + { "suppress", KW_SUPPRESS }, > > { "fsync", KW_FSYNC }, > > { "long_hostnames", KW_CHAIN_HOSTNAMES }, > > { "chain_hostnames", KW_CHAIN_HOSTNAMES }, > > ok. > > > diff --git a/src/logmsg.c b/src/logmsg.c > > index 05354b5..4da3af5 100644 > > --- a/src/logmsg.c > > +++ b/src/logmsg.c > > @@ -766,3 +766,29 @@ log_msg_drop(LogMessage *m, guint path_flags) > > log_msg_ack(m); > > log_msg_unref(m); > > } > > + > > +/** > > + * log_msg_dup: > > + * @m: LogMessage instance > > + * > > + * This function duplicates an existing log message. The new message is > > not parsed. > > + */ > > +LogMessage * > > +log_msg_dup(LogMessage *m) > > +{ > > + LogMessage *nm = g_new0(LogMessage, 1); > > + > > + log_msg_init(nm, m->saddr); > > + nm->flags > > = m->flags; > > + nm->pri = m->pri; > > + g_string_assign_len(&nm->date, m->date.str, m->date.len); > > + g_string_assign_len(&nm->host, m->host.str, m->host.len); > > + g_string_assign_len(&nm->host_from, m->host_from.str, > > m->host_from.len); > > + g_string_assign_len(&nm->program, m->program.str, m->program.len); > > + g_string_assign_len(&nm->msg, m->msg.str, m->msg.len); > > + memcpy(&nm->stamp, &m->stamp, sizeof(LogStamp)); > > + memcpy(&nm->recvd, &m->recvd, sizeof(LogStamp)); > > + > > + return nm; > > +} > > I don't think this is needed, but see below. > > > + > > diff --git a/src/logmsg.h b/src/logmsg.h > > index 420a711..9dbb886 100644 > > --- a/src/logmsg.h > > +++ b/src/logmsg.h > > @@ -107,6 +107,8 @@ void log_msg_ack_block_end(LogMessage *m); > > void log_msg_ack(LogMessage *msg); > > void log_msg_drop(LogMessage *msg, guint path_flags); > > void log_msg_clear_matches(LogMessage *self); > > +LogMessage *log_msg_dup(LogMessage *m); > > + > > > > gchar *log_msg_find_cr_or_lf(gchar *s, gsize n); > > > > diff --git a/src/logwriter.c b/src/logwriter.c > > index bb82b43..4843100 100644 > > --- a/src/logwriter.c > > +++ b/src/logwriter.c > > @@ -178,11 +178,63 @@ log_writer_watch_new(LogWriter *writer, FDWrite > > *fd) > > return &self->super; > > } > > > > +static gboolean > > +log_writer_supress_duplicate(LogWriter *self, LogMessage *lm, gint > > path_flags) > > +{ > > + if (!self->options->suppress) > > + return FALSE; > > + > > + if(self->suppress.hash) > > + { > > + guint hash = g_string_hash(&lm->msg); > > + if(self->suppress.hash == hash) > > + { > > + msg_debug("Dropping duplicate message", > > + NULL); > > + self->suppress.count ++; > > + log_msg_drop(lm, path_flags); > > + return TRUE; > > + } > > I don't think g_string_hash buys us anything, in fact it might well be > slower than doing a direct string comparison (strcmp is performed in > chunks of sizeof(long) bytes, g_string_hash iterates over characters > individually). > > You mentioned that you are also checking the hostname, which is not the > case, lm->msg contains the application name and pid only. > > I think we should also check that. > > > + > > + if (self->suppress.count) > > + { > > + gchar *buf; > > + LogMessage *m; > > + GString *line = g_string_sized_new(128); > > + > > + buf = g_strdup_printf("<%d> syslog-ng[%d]: Last message > > '%.20s' repeated %d times\n", > > + self->suppress.msg->pri, > > + getpid(), > > + self->suppress.msg->msg.str, > > + self->suppress.count); > > we'd probably need a log_msg_new_internal() function, that would produce > the same message without having to parse it. This and the code in > messages.c would be a direct user of that function. > > > + m = log_msg_new(buf, strlen(buf), self->suppress.msg->saddr, > > 0, NULL); > > + g_queue_push_tail(self->queue, m); > > + g_queue_push_tail(self->queue, GUINT_TO_POINTER(0x80000000)); > > + g_free(buf); > > + } > > + log_msg_drop(self->suppress.msg, 0); > > + self->suppress.msg = log_msg_dup(lm); > > As a LogMessage is read-only (well most of, and I'm directing syslog-ng > 2.2 so that LogMessage is going to be read only), I think it is enough > to store a reference and no need to copy the complete message. This is > too expensive. > > > + self->suppress.hash = g_string_hash(&lm->msg); > > + self->suppress.count = 0; > > + } > > + else > > + { > > + self->suppress.msg = log_msg_dup(lm); > > + self->suppress.hash = g_string_hash(&lm->msg); > > + self->suppress.count = 0; > > + } > > + > > + return FALSE; > > +} > > + > > static void > > log_writer_queue(LogPipe *s, LogMessage *lm, gint path_flags) > > { > > LogWriter *self = (LogWriter *) s; > > - > > + > > + if(log_writer_supress_duplicate(self, lm, path_flags)) > > + return; > > + > > if ((self->queue->length / 2) == self->options->fifo_size) > > { > > /* drop incoming message, we must ack here, otherwise the sender > > might > > @@ -462,6 +514,9 @@ log_writer_new(guint32 flags, LogPipe *control, > > LogWriterOptions *options) > > self->queue = g_queue_new(); > > self->flags = flags; > > self->control = control; > > + self->suppress.msg = NULL; > > + self->suppress.hash = 0; > > + self->suppress.count = 0; > > return &self->super; > > } > > > > @@ -476,6 +531,8 @@ log_writer_options_defaults(LogWriterOptions > > *options) > > options->ts_format = -1; > > options->zone_offset = -1; > > options->frac_digits = -1; > > + options->suppress = FALSE; > > + > > } > > > > void > > diff --git a/src/logwriter.h b/src/logwriter.h > > index 284899b..d3a136f 100644 > > --- a/src/logwriter.h > > +++ b/src/logwriter.h > > @@ -41,6 +41,13 @@ > > /* several writers use the same counter */ > > #define LWOF_SHARE_STATS 0x0002 > > > > +typedef struct _LogSuppress > > +{ > > + LogMessage *msg; > > + guint hash; > > + guint32 count; > > +} LogSuppress; > > As I've removed hash from the picture, I don't think we need a separate > structure for this. Ju> > } LogWriterOptions; > > > > typedef struct _LogWriter > > @@ > > -7 9,6 +87,7 @@ typedef struct _LogWriter > > gint partial_pos; > > LogPipe *control; > > LogWriterOptions *options; > > + LogSuppress suppress; > > } LogWriter; > > > > void log_writer_set_options(LogWriter *self, LogPipe *control, > > LogWriterOptions *options); > > -- > > 1.5.4.5 > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.campin.net/syslog-ng/faq.html > >

2 2