February 2010 - syslog-ng - lists.balabit.hu

What testing has been done for the new features in open source version of syslog-ng version 3.x
by Janani Janakiraman 06 Feb '10

06 Feb '10

I am trying to judge the level of confidence for the latest stable open-source version of syslog-ng version 3.0.5. Where can I find the test results of the tests that have been run again version 3.0.5 of syslog-ng? I want to find out if tests have been run to test the new features in Version 3.x specifically support for the new IETF-syslog protocol standard ( RFC5424 ). Do any of the major Linux distros support version 3.x of syslog-ng? Thanks, Janani

3 4

Re-post: What testing has been done for the new features in open source version of syslog-ng version 3.x
by Jeremy M. Guthrie 06 Feb '10

06 Feb '10

Sorry, I didn't see an answer. Just wanted to re-post to reconfirm: I have an issue where what is being written to file is different than what I have being written to disk. I don't know if the macro is broken, my understanding is wrong, or I am just using the wrong macro altogether. My understanding is roughly as follows: PROGRAM The name of the program sending the message. PID The PID of the program sending the message. MSG or MESSAGE Message contents including the program name and pid. MSGONLY Message contents without the program name. 1. The first thing I ran into: Example messages on file: Jan 25 08:34:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:34:15 XXXXXXXXXX service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$MSG');\n" ) 2. Then what I tried to get it closer.... Example messages on file: Jan 25 08:40:18 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:40:18 XXXXXXXXXX sshd: service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$PROGRAM: $MSG');\n" ) 3. Then what I tried to get even closer: Example messages on file: Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$PROGRAM[$PID]: $MSG');\n" ) However, this broke messages coming from Cisco devices... Jan 25 08:55:10 FIREWALL %FWSM-3-710003[]: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 The message above should be: Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 Notice the above '[]' after the FWSM code? I am running 3.0.5 syslog-ng. Ideally I'd get both: Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 AND Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Is the MSG/MESSAGE macro broken? What am I missing? -- -------------------------------------------------- Jeremy M. Guthrie jeremy.guthrie(a)cdw.com Managed Cisco Security Services Hosting and Managed Services Technical Architect Phone: 608-298-1061 CDW Fax: 608-288-3007 5520 Research Park Drive NOC: 608-298-1102 Madison, WI 53711 NOC Email: hmshelp(a)cdw.com

2 1

config file question
by Rory Toma 05 Feb '10

05 Feb '10

OK, I have a question. I've found that it is good to run several syslog-ng processes when dealing with tens of thousands of log files. So, I'd like to receive everything in one syslog-ng process, and forward them, via localhost, to about a hundred or so different syslog-ng processes on the same machine. Is there a good way to write a (log; filter; dest;) line that would round-robin between a bunch of destinations in a single line?

4 6

syslog-ng dropping events - update
by Terry Slattery 03 Feb '10

03 Feb '10

I did a lot of research last week on network tuning for Centos 5.3, the OS on which syslog-ng is showing missed events. I also have a RH Enterprise 3 system that doesn't show event loss. The RHE3 system is forwarding events to the Centos-based system that is dropping events. I compare the logs from the RHE3 system with the logs on the Centos to determine if loss is occurring. Here's what I did: ==== Mods to syslog-ng.conf ==== Enable dns cache options in syslog-ng.conf: use_dns (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire (86400); Set so_rcvbuf() option in syslog-ng.conf: udp(so_rcvbuf(1024000)); The resulting config: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { time_reopen (10); log_fifo_size (100000); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire (86400); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); }; # Set the UDP receive buffer to 1M. source s_net { udp(so_rcvbuf(1024000)); tcp(); #syslog(); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; ###### # Log local and network events log { source(s_local); source(s_net); destination(d_messages); }; ==== Mods to sysctl.conf ==== Modify kernel buffer parameters in /etc/sysctl.conf: # Increase the maximum total buffer-space allocatable net.ipv4.tcp_mem = 8388608 12582912 16777216 net.ipv4.udp_mem = 8388608 12582912 16777216 # Increase the maximum read-buffer space allocatable net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.udp_rmem_min = 16384 # Increase the maximum write-buffer-space allocatable net.ipv4.tcp_wmem = 8192 65536 16777216 net.ipv4.udp_wmem_min = 16384 # Increase the maximum and default receive socket buffer size net.core.rmem_max=16777216 net.core.rmem_default=87380 #Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them. Default was 1000. net.core.netdev_max_backlog = 2500 # Increase the max number of connections. Default was 128. # I think this is only for tcp and one web site recommended increasing # it for heavily used web servers. net.core.somaxconn=1024 # Default to 20480 net.core.optmem_max = 1024000 # Drop packets that look like they are spoofed. Default = 1. # 0 = don't check; # 1 = drop packets that as sourced at a directly connected interface # but were input from another interface; # 2 = drop any packets that look spoofed. net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.eth0.rp_filter = 0 Then restart sysctl: sysctl -p ==== How to check packet loss on Linux ==== [Look for the section titled UDP in netstat output. I like to preceed the netstat command with a date command and save both to a file as a record for later comparisons.] # date Wed Feb 3 16:33:55 EST 2010 # netstat -us ... Udp: 24264777 packets received 2831 packets to unknown port received. 14262 packet receive errors 24538489 packets sent ... # date Wed Jan 27 11:21:54 EST 2010 # netstat -us ... Udp: 24227388 packets received 470 packets to unknown port received. 10644 packet receive errors 24503312 packets sent ... Note the increase in UDP receive errors. The packet rate is pretty low on these systems, peaking at a few packets per second and averages well below a packet per second. ====== Other thoughts ===== The RHE3 system has been forwarding log messages using the address spoofing feature. I turned it off yesterday afternoon and the logs show no packet loss since then. Unfortunately, I did not do a netstat at that time, so I can't tell if the kernel knows of any loss. I also did not have eth0/rp_filter = 0 until today, figuring that the default/rp_filter = 0 parameter would cover it, but decided that it wouldn't hurt to have both set. Interestingly, the RHE3 system has both values set to 1 and it does not drop any events (that I can detect - it certainly is getting more events than the Centos system). I'm about to load RHE5 on a different server, forward events to it and see if it drops events. -tcs -- Terry Slattery CCIE# 1026

1 0

Third Party Application Logs under syslog-ng.conf -solaris 10
by manik tripathi 03 Feb '10

03 Feb '10

Hi, We have oracle siebel Apps , that generates bunch of Logs under a "dir" and we wanted to feed the logs under that dir to Centralized Syslog-ng server. Is there a sample syslog-ng for solaris for doing wildcard dir or wildcard logs files.... Thanks _________________________________________________________________ Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969

2 1

syslog-ng v2.09
by Clayton Dukes 02 Feb '10

02 Feb '10

Hey guys, I have a customer that is running v2.09 (with security patches). We need them to forward logs to our syslog-ng server but need to keep the original device as the source name. I'm assuming this is what they need: keep_hostname(yes); chain_hostnames(no); Can anyone confirm that v2.09 supports this? Is there anything else needed to accomplish this in v2.09? -- ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 3

syslog-ng poll
by Robert Fekete 01 Feb '10

01 Feb '10

Dear syslog-ng Users, As you might have already seen, we have recently restructured the webpage of syslog-ng Open Source Edition to resemble more to the webpages of our other products. We have also added some new information (e.g., references to third-party packages, etc.). We would also like to ask regularly your opinion about syslog-ng and its various features using anonymous, quick-to-complete polls. As a starter, we would like to know which version of syslog-ng do you most (meaning, on most of your hosts). If you have a few free seconds, please vote in the sidebox of the http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/ page. Also, any comments or would like to suggest questions to future polls is welcome. Regards, Robert

1 0