August 2006 - syslog-ng - lists.balabit.hu

using the program destination driver
by Matt Zagrabelny 28 Aug '06

28 Aug '06

greetings list, i will (breifly) describe what i would like to do, what i have tried in order to accomplish that, and what i think is going wrong. i would very much appreciate any help. at the university that i work at we have a cisco vpn box that is configured to send syslog messages to a debian box running syslog-ng version 2.0rc1-2 (version in 'testing'). the types of messages that are sent from the vpn box are like 'user logging in using ip <ip>' and 'user logging out'. as these messages are coming in, i would like to spawn a process, extract the useful bits of info from the syslog message and update a database with the connection window for the user, local and remote ip address. what i have tried: excerpts from /etc/syslog-ng/syslog-ng.conf source s_remote { udp(); }; destination dp_process_vpn { program("/usr/local/sbin/process_vpn_syslog"); }; filter f_vpn { netmask("131.212.4.2/255.255.255.255"); }; log { source(s_remote); filter(f_vpn); destination(dp_process_vpn); }; end of excerpt from /etc/syslog-ng/syslog-ng.conf # cat /usr/local/sbin/process_vpn_syslog #!/usr/bin/perl # eventually this program will parse the data in the syslog entry # and update a database, but for now, it is just appending the # info to a file in /tmp use strict; $| = 1; my $file = '/tmp/vpn_info'; open FILE, '>>'.$file; while (<>) { print FILE $_; } end of cat /usr/local/sbin/process_vpn_syslog i do the following commands to test and confirm that the program process_vpn_syslog is working ok and expected. $ tail -f /tmp/vpn_info $ tail -f /storage/syslog/hosts/131.212.4.2/2006/08/28/local4 i get instant data from the file destination driver syslogs going to /storage/syslog/hosts/131.212.4.2/2006/08/28/local4. i get *very* latent data in the file /tmp/vpn_info. possibilities of what i think is going wrong: 1) data is not being flushed from syslog-ng to the program. i have tried adding flush_lines(1) to the program directive above, to no avail. 2) the syslog-ng documentation says: that it starts the program once and restarts it if it exits. i think i am looking for a program that is executed once for every *line* entry from syslog-ng, however i am not sure where to go from here. thanks again for taking time to look at this. sincerely, -- Matt Zagrabelny - mzagrabe(a)d.umn.edu - (218) 726 8844 University of Minnesota Duluth Information Technology Systems & Services PGP key 1024D/84E22DA2 2005-11-07 Fingerprint: 78F9 18B3 EF58 56F5 FC85 C5CA 53E7 887F 84E2 2DA2 He is not a fool who gives up what he cannot keep to gain what he cannot lose. -Jim Elliot

1 0

Equivalent to Syslog's -a for extra socket listening?
by Hari Sekhon 25 Aug '06

25 Aug '06

I'm reading up on chrooting bind and need a log socket inside the chroot jail. Is there an equivalent to Syslogd's -a switch that creates a second socket to listen on? I suspect not, and that I will have to create it myself and then just add a source for the extra unix-stream definition to my conf. Thanks for any feedback you can give on this. Hari Sekhon

2 2

RE: syslog-ng Digest, Vol 16, Issue 27
by Andrew Meyer 25 Aug '06

25 Aug '06

Message: 2 Date: Fri, 25 Aug 2006 13:51:44 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Cisco Pix 501 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1156506704.11997.10.camel(a)bzorp.balabit> Content-Type: text/plain On Tue, 2006-08-22 at 12:39 -0500, Andrew Meyer wrote: > After upgrading from 1.6.x on debian unstable to the latest 2.0.0, L > have noticed that syslog-ng is no longer picking up logs from my pix. > What do I need to change from the old way of doing things to make it > log? I'm not sure, it should work. Can you check if the logs actually arrive at the loghost? -- Bazsi I checked the logs on Debian, they are not being forwardede to my server. Andrew Meyer 314-277-3307

1 0

FQDN's in syslog-ng
by Steven Hajducko 25 Aug '06

25 Aug '06

Hi everyone, We've currently got a syslog setup that centralizes our logs from many of our different teirs into one location, using each system's prebuilt syslog. It works at the moment, but we've got some issues. Our main one is that we name our hosts the same in different environments, mainly because these environments are for moving code and configs from 1 step to the next, so to make it easier on people so they don't have to change configurations each time the code moves, hosts share similar names. The problem is that when all the logs come back to the central syslog server, they can't be seperated by host because of the similiarities. So 'web5' in our QA is the same as 'web5' in production. Because of that, we were looking at syslog-ng and while I found a couple references to using FQDN, I've been unable to tell whether or not this is possible. Our machines DO have different FQDNs. For example, web5.qa.location.domain.com vs. web5.prod.location.domain.com. If we run our main central server on syslog-ng and replace all the syslog daemon's on each system with it, can we force the FQDN to be given rather then just the base hostname and have the central log server sort the logs into different directories based on the FQDN? Thanks! -- sh

4 7

SEC and slct
by Brian Loe 25 Aug '06

25 Aug '06

So... I'm look at these applications and trying to figure out how best to implement them - does anyone have thoughts? I guess I haven't read enough of the man page yet, but I'm still not even sure how the SEC config file works (what you put into it)! But, both apps look promising - and the SEC page mentions another tool I'm going to check out as well...

2 1

Truncated Messages when using templates
by Ivan Stegic 25 Aug '06

25 Aug '06

Hello everyone... We have chosen syslog-ng for our corporate application's central logging daemon and are very happy with it so far. Just one problem: truncated messages. We are running syslog-ng 1.6.11-1 on a Fedora 5 Intel Xeon box. We have an application specific template that we are logging information for into a MySQL 5.0.22 database. If we ever try to log a message that is more than 10674 characters in length, the message is truncated. We expect to only log this large sized message when the DEBUG level of our application is switched to EXTREMELY_HIGH. Now, although this probably won't happen often, it will be at that point that we would most need that information! If the message is logged to a file, the truncation does NOT occur. Does anyone have any insight into this? I would appreciate a push in the right direction for more information, or maybe even a patch of some sort? Please help! Ivan Stegic.

3 3

RE: Performance tuning questions
by Martin, David M 25 Aug '06

25 Aug '06

Greg, I had 'messages being dropped' issue recently on rhel4 and sol8 hosts. since you are using rhl.. 'netstat -ucs' was reporting 500-2000 packet receive errors per second. after making a few adjustments it has dropped to 0. my sysctl.conf is similar to yours, except for: net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 * values mentioned worked for me, your mileage may vary. in my syslog-ng.conf file, I adjusted the source: (may be unnecessary since I use the default as set by sysctl (above), but for documentation sake) source abs_udp514 { udp(ip(0.0.0.0) port(514) so_rcvbuf(16777216)); }; set the log_fifo_size destination abd_sec { program( "/opt/sec/2.3.3/sec.pl -input=\"-\" -conf=/opt/sec/etc/sec.conf" template("$MONTH$DAY $UNIXTIME $HOST $MSG\n") log_fifo_size(8192) ); }; did something similar for my file destinations. and then add flow-control log { source(abs_udp514); filter(abf_postfix); destination(abd_sec); flags(flow-control); }; # netstat -ucs Udp: 1067529425 packets received 9835593 packets to unknown port received. 27851339 packet receive errors 772843529 packets sent Udp: 1067529464 packets received 9835593 packets to unknown port received. 27851339 packet receive errors 772843549 packets sent ------------------------------ Message: 3 Date: Tue, 22 Aug 2006 08:04:38 -0500 From: "King, John (Greg) (LMIT-HOU)" <Greg.King(a)lmit.com> Subject: [syslog-ng] Performance tuning questions To: <syslog-ng(a)lists.balabit.hu> Message-ID: <7E0DB88FB5A5C04AAB0A06E72D58B2B4026053AF(a)lmit-mb01-hou.SERVICE.lmit.com> Content-Type: text/plain; charset="us-ascii" I am currently using the stock syslog daemon from RedHat but it appears to not be able to keep up so I am looking at syslog-ng to improve things. The data below is to provide a baseline of what I am currently seeing and what I have attempted to do. Then if anyone would let me know if syslog-ng would be able to improve the performance and what measures I can take to achieve the improved performance that would be great. Logs have to be rotated each hour due to the amount of traffic. On average I am successfully logging 25,888 events per minute. That goes higher during the early morning login times. I have set the following sysctl params: net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 net.core.rmem_default = 65536 net.core.wmem_default = 65536 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_wmem = 4096 65536 33554432 net.ipv4.tcp_mem = 33554432 33554432 33554432 Netstat -su shows what might be data loss: Udp: 131725715 packets received 16642 packets to unknown port received. 4859684 packet receive errors 31571 packets sent The information transmitted (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is intended only for the person(s) or entity/entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

Cisco Pix 501
by Andrew Meyer 25 Aug '06

25 Aug '06

After upgrading from 1.6.x on debian unstable to the latest 2.0.0, L have noticed that syslog-ng is no longer picking up logs from my pix. What do I need to change from the old way of doing things to make it log? Andrew Meyer 314-277-3307

2 1

Performance tuning questions
by King, John (Greg) (LMIT-HOU) 25 Aug '06

25 Aug '06

I am currently using the stock syslog daemon from RedHat but it appears to not be able to keep up so I am looking at syslog-ng to improve things. The data below is to provide a baseline of what I am currently seeing and what I have attempted to do. Then if anyone would let me know if syslog-ng would be able to improve the performance and what measures I can take to achieve the improved performance that would be great. Logs have to be rotated each hour due to the amount of traffic. On average I am successfully logging 25,888 events per minute. That goes higher during the early morning login times. I have set the following sysctl params: net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 net.core.rmem_default = 65536 net.core.wmem_default = 65536 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_wmem = 4096 65536 33554432 net.ipv4.tcp_mem = 33554432 33554432 33554432 I have also set the ring params on the intel1000 nic (using intels latest driver e1000-7.1.9, not the default kernel one) to 512MB. Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 512 RX Mini: 0 RX Jumbo: 0 TX: 512 Based on the nic statistics, the bottleneck is not at the nic: NIC statistics: rx_packets: 148314357 tx_packets: 24906469 rx_bytes: 3023662070 tx_bytes: 3216438764 rx_errors: 0 tx_errors: 0 tx_dropped: 0 multicast: 8531 collisions: 0 rx_length_errors: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 10699059 tx_tcp_seg_failed: 0 rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 76038106102 rx_csum_offload_good: 148139283 rx_csum_offload_errors: 0 rx_header_split: 0 alloc_rx_buff_failed: 0 Sar -B output shows a lot of paging going on which is probably causing some loss 06:10:01 AM pgpgin/s pgpgout/s fault/s majflt/s 06:20:01 AM 0.00 160.46 3.02 0.00 06:30:01 AM 0.00 162.02 3.04 0.00 06:40:01 AM 0.00 164.49 4.14 0.00 06:50:01 AM 0.02 161.51 3.03 0.00 07:00:01 AM 0.00 174.21 3.04 0.00 07:10:01 AM 0.11 198.15 6.02 0.00 07:20:01 AM 0.01 193.53 3.03 0.00 07:30:01 AM 1278.66 1593.51 29.30 0.04 Average: 67.76 189.06 19.12 0.02 Strange though (or maybe I am misreading this) sar -W shows all zeroe's for pswpin/s and pswpout/s which says there is not swapping going on. Sar -N shows no errors either, but the udpsck value is locked at 5. I am thinking this is hardcoded somewhere or maybe compiled in syslog srpms as I cannot find a syslog setting. This is one reason for wanting to use syslog-ng as from what I read I can change the number of allocated sockets via the configuration file. Sar -P shows the CPU is avg 95% idle so I see no issue here. I have re-niced syslog to -10 to increase it's priority. Netstat -su shows what might be data loss: Udp: 131725715 packets received 16642 packets to unknown port received. 4859684 packet receive errors 31571 packets sent I have no way to tell if the udp errors are related to syslog data and with so much syslog data arriving setting up tcpdump to see what the errors relate to is going to be rather difficult. What I think is going on is the stock syslog daemon is simply unable to buffer enough to keep up with the syslog stream. Therefore I am wanting to look at using syslog-ng to see if the error rate drops. Also I was thinking of using TCP for this but at the data rate I am seeing I am thinking this would cause a potential denial of service on both the syslog transmitters and the syslog receiving server. Any thoughts, ideas? Thanks Greg

3 2

log rotation
by Brian Loe 25 Aug '06

25 Aug '06

I've been trying to get logrotate to work on AIX and it simply doesn't "do" anything (run it with debug and it tells me its reading the config file then exits, files remain the same - regardless of size or age). So, what else should I try? Eventually it looks like I'm going to be feeding pipes and files for use with SEC - but I need to archive each log for 30 days, then archive all of those files for a year, etc..

3 2