February 2010 - syslog-ng - lists.balabit.hu

Syslog-ng 3.x and kernel messages
by Phusion 22 Feb '10

22 Feb '10

I've used syslog-ng 1.x and 2.x, but am new to using 3.x. I'm using syslog-ng 3.0.3 on a FreeBSD 8.0 server. I'm having trouble getting kernel messages. I've made a very small config file, and this isn't working either. @version:3.0 source local { internal(); file("/dev/klog"); unix-dgram("/var/run/log"); }; destination all { file("/var/log/all.log"); }; log { source(local); destination(all); }; This picks up other things, but not the kernel messages. I assume it's a small configuration issue. Let me know. Thanks. Phusion

3 3

syslog-ng performance tuning
by Rémi BUISSON 22 Feb '10

22 Feb '10

Hi everybody, I'have an issue with syslog-ng configuration. I would like to centralize my logs on one server. I've a lot of logs to send. I don't know how many but I can estimate it to 500GB per day from decades of servers. But, it writes only 25 GB per day. For some reasons I work on a debian etchnhalf environnement. So, I'm working with syslog-ng 2.0.0. I wrote a perl program which spawn p "logger -p local5.info" processes and send n lines of m characters. I'have tested with: p: 1 000 n: 1 000 m: 1 000 Instead of having 1 000 000 lines in my logs I have nearly 10 000 lines ! But my test was not revelant because normal logs where not stopped. So, maybe normal. I've googled my issue and tried to play on log_fifo_size(), max_connections() and sync() on both server and client but nothing to do ... Here is my client configuration file: options { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(no); stats_freq(0); bad_hostname("^gconfd$"); }; source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); }; destination dn_fslog { tcp("xxx.xxx.xxx.xxx" port(514)); }; filter f_local5 { facility(local5); }; log { source(s_all); filter(f_local5); destination(dn_fslog); }; # for all my apache vhosts destination df_access_www.foo.com { file("/var/www/www.foo.com/logs/access.log"); }; filter f_local0_access_www.foo.com { level(info) and facility(local0) and program("www\.foo\.com"); }; log { source(s_all); filter(f_local0_access_www.foo.com); destination(df_access_www.foo.com); destination(dn_fslog); }; destination df_error_www.foo.com { file("/var/www/www.foo.com/logs/error.log"); }; filter f_local0_error_www.foo.com { level(error) and facility(local0) and program("www\.foo\.com"); }; log { source(s_all); filter(f_local0_error_www.foo.com); destination(df_error_www.foo.com); destination(dn_fslog); }; etc with www.foo2.com Here is my server configuration file: options { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(no); stats_freq(0); bad_hostname("^gconfd$"); }; source s_net { tcp(ip(xxx.xxx.xxx.xxx) port(514)); }; destination df_net_access_aaa.aaa.aaa.aaa-www.foo.com { file("/logs/www.foo.com/aaa.aaa.aaa.aaa-access.log"); }; filter f_local0_access_aaa.aaa.aaa.aaa-www.foo.com { level(info) and facility(local0) and program("www\.foo\.com") and host("aaa\.aaa\.aaa\.aaa"); }; log { source(s_net); filter(f_local0_access_aaa.aaa.aaa.aaa-www.foo.com); destination(df_net_access_aaa.aaa.aaa.aaa-www.foo.com); }; destination df_net_error_aaa.aaa.aaa.aaa-www.foo.com { file("/logs/www.foo.com/aaa.aaa.aaa.aaa-error.log"); }; filter f_local0_error_aaa.aaa.aaa.aaa-www.foo.com { level(error) and facility(local0) and program("www\.foo\.com") and host("aaa\.aaa\.aaa\.aaa"); }; log { source(s_net); filter(f_local0_error_aaa.aaa.aaa.aaa-www.foo.com); destination(df_net_error_aaa.aaa.aaa.aaa-www.foo.com); }; etc with aaa.aaa.aaa.bbb and www.foo2.com Just to give you a number I'm speaking about 20 web servers with 20 load balanced websites on each. Servers are on a Gigabit networks with bonded Gigabit interfaces. It is running on Linux 2.6.24-etchnhalf.1-686-bigmem. Maybe syslog-ng cannot handle 400 different filter rules but it would suprize me. If it's the case, in there any better way to separate, for instance, apache error and access log for in each server, in a file name /logs/<project = progname>/<ip>-access.log and /logs/<project = progname>/<ip>-error.log ? I don't think my hardware is in cause because cpu average 98% idle and no io wait. Can you tell me what I misunderstood or simply what's wrong in my configuration files. Or, it may work better with a 3.x version compiled with some options ? Thanks in advance. Regards, Rémi

3 6

Issue is resolved
by dwy1008 19 Feb '10

19 Feb '10

The main problem is that 'eventlog'.Because there are problems compiling,this results in the syslog-ng non start. I would like to thank you for helping me.

1 0

syslog-ng gets started twice when system reboots
by Xuan Zhong 18 Feb '10

18 Feb '10

I am using syslog-ng-3.0.5 RHEL5 x86. After the installation and rebooting the system: Starting syslog-ng: Starting syslog-ng: [ OK ] [ OK ] Does anyone know why syslog-ng gets started twice after reboot? Thanks Xuan _________________________________________________________________ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. http://clk.atdmt.com/GBL/go/201469229/direct/01/

4 6

syslog-ng with logrotate
by Tom Simons 16 Feb '10

16 Feb '10

We're replacing sysklogd on our RHEL4 & 5 servers with syslog-ng-premium-edition 3.0.4-1. Has anyone configured logrotate to work with syslog-ng? Can we just build /etc/logrotate.d/syslog-ng from /etc/logrotate.d/syslog?

3 2

UDP source performance improvement sugestions
by Jean F. Mousinho 15 Feb '10

15 Feb '10

Hello, I'm doing some performance tests between syslog-ng and rs####g (yes I guess I'm not the first to think/do something like this). I've noticed that syslog-ng is much slower in the case of a single UDP source and destination to file due to the nature of the message reception code. I then decided to play a bit with the buffering parameters that syslog-ng allows me to set to try improve its performance in this specific test. I expect that when I set time_sleep to 0, it even avoids calling nanosleep from what I saw, so it should continuously read from the UDP socket. I then set a big log_fifo_size 10000 (without knowing exactly if its useful or not in this case, destination is file) and on the source udp I set log_fetch_limit also something like 5000, log_iw_size to 10000. These steps didn't improved the performance that much, maybe I could be doing something wrong.. >From what I understood, this will make syslog-ng read 5000 messages from the socket (if they are available) and only after that it queues them to the fifo? fifo waits for flush_lines number of lines or flush_timeout to write the data to the file? Is there anything else that you know I could do to improve? flush_lines > 0? .. or maybe some resource/doc I could/should read. Thanks for your time. Jean Mousinho

2 2

Re: [syslog-ng] Syslog-NG Premium Edition installation on SuSE Linux Enterprise Server 10
by Heinze, René 15 Feb '10

15 Feb '10

>On Mon, Feb 15, 2010 at 05:05:21PM +0100, Balazs Scheidler wrote: >> On Mon, 2010-02-15 at 16:38 +0100, Heinze, René wrote: >> > Hello, >> > >> > I´m having trouble installing the Syslog-NG Premium Edition on SLES >> > 10. I´m installing the RPM package 3.04 x86 for OpenSuSE 10.1. The RPM >> > installes just fine, but I´m not able to run the .run installer. The >> > installation of the rpm package is completely unattended. How can I >> > start the .run installer? >> > >> >> That's right, the rpm package is not interactive. You can start the .run >> installer by removing the .rpm package first and starting the .run file >> as a command: >> >> # ./syslog-ng-premium-edition-3.0.4-linux-i386.run >Please make sure, that the .run file has the executable attribute set >before you attempt to run it: > ># chmod +x syslog-ng-premium-edition-3.0.4-linux-i386.run ># ./syslog-ng-premium-edition-3.0.4-linux-i386.run > >or force the shell to execute it: > ># sh syslog-ng-premium-edition-3.0.4-linux-i386.run Thanks for you support! I have overlooked the generic linux version on the Balabit webpage. What´s the best method installing Syslog-NG Premium on a SLES 10 System, generic or the rpm version? On the rpm version, how do i attach the license file? If the rpm version is not interactive then there is a error in the admin manual. The manual says on page 86 in the first line: "Answer the configuration questions of syslog-ng. These are described in detail in Section 4.1, “Installing syslogng using the .run installer” (p. 78)." Best Regards, René H. Besuchen Sie die Bundesdruckerei auf der CeBIT 2010 vom 2.-6.3.2010, Halle 9, Stand D80 Visit Bundesdruckerei at CeBIT, exhibition centre, hall 9 / stand D80 weitere Informationen unter: http://www.bundesdruckerei.de/de/unternehmen/untern_cebit2010/index.html find more information here http://www.bundesdruckerei.de/en/company/comp_cebit2010/index.html

1 0

Syslog-NG Premium Edition installation on SuSE Linux Enterprise Server 10
by Heinze, René 15 Feb '10

15 Feb '10

Hello, I´m having trouble installing the Syslog-NG Premium Edition on SLES 10. I´m installing the RPM package 3.04 x86 for OpenSuSE 10.1. The RPM installes just fine, but I´m not able to run the .run installer. The installation of the rpm package is completely unattended. How can I start the .run installer? Thanks! Best Regards, René Heinze ________________________________ Besuchen Sie die Bundesdruckerei auf der CeBIT 2010 vom 2.-6.3.2010, Halle 9, Stand D80 Visit Bundesdruckerei at CeBIT, exhibition centre, hall 9 / stand D80 weitere Informationen unter: http://www.bundesdruckerei.de/de/unternehmen/untern_cebit2010/index.html find more information here http://www.bundesdruckerei.de/en/company/comp_cebit2010/index.html

3 2

Getting a SQL insert to match what's being written to file
by Jeremy M. Guthrie 15 Feb '10

15 Feb '10

I have an issue where what is being written to file is different than what I have being written to disk. I don't know if the macro is broken, my understanding is wrong, or I am just using the wrong macro altogether. My understanding is roughly as follows: PROGRAM The name of the program sending the message. PID The PID of the program sending the message. MSG or MESSAGE Message contents including the program name and pid. MSGONLY Message contents without the program name. 1. The first thing I ran into: Example messages on file: Jan 25 08:34:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:34:15 XXXXXXXXXX service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$MSG');\n" ) 2. Then what I tried to get it closer.... Example messages on file: Jan 25 08:40:18 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:40:18 XXXXXXXXXX sshd: service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$PROGRAM: $MSG');\n" ) 3. Then what I tried to get even closer: Example messages on file: Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$PROGRAM[$PID]: $MSG');\n" ) However, this broke messages coming from Cisco devices... Jan 25 08:55:10 FIREWALL %FWSM-3-710003[]: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 The message above should be: Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 Notice the above '[]' after the FWSM code? I am running 3.0.5 syslog-ng. Ideally I'd get both: Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 AND Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Is the MSG/MESSAGE macro broken? What am I missing? -- -------------------------------------------------- Jeremy M. Guthrie jeremy.guthrie(a)cdw.com Managed Cisco Security Services Hosting and Managed Services Technical Architect Phone: 608-298-1061 CDW Fax: 608-288-3007 5520 Research Park Drive NOC: 608-298-1102 Madison, WI 53711 NOC Email: hmshelp(a)cdw.com

2 2

Syslog restarting
by Henric Larsson 11 Feb '10

11 Feb '10

Hi, We're using syslog-ng PE with disk buffering enabled. The syslog clients gets disconnected all the time, and when they restart I get these log messages: 2010-02-11T14:19:37+00:00 bruatosd003 syslog-ng[4686]: I/O error occurred while writing; fd='8', error='Connection reset by peer (104)' 2010-02-11T14:19:37+00:00 bruatosd003 syslog-ng[4686]: Syslog connection broken; fd='8', server='AF_INET(172.27.12.22:1998)', time_reopen='60' 2010-02-11T14:19:37+00:00 bruatosd003 syslog-ng[4686]: Closing log transport fd; fd='8' 2010-02-11T14:20:41+00:00 bruatosd003 syslog-ng[4686]: Syslog connection established; fd='8', server='AF_INET(172.27.12.22:1998)', local='AF_INET(0.0.0.0:0)' 2010-02-11T14:20:54+00:00 bruatosd003 syslog-ng[28626]: The current log file has a mismatching size/inode information, restarting from the beginning; filename='/var/ig/logs/orderserver/os-gc.1.log' 2010-02-11T14:20:54+00:00 bruatosd003 syslog-ng[28626]: Disk-buffer state saved; filename='/var/ig/logs/orderserver/syslog-ng-db/syslog-ng-00004.qf', qout_length='0', qbacklog_length='0', qoverflow_length='0', qdisk_length='0' 2010-02-11T14:20:54+00:00 bruatosd003 syslog-ng[28626]: The current log file has a mismatching size/inode information, restarting from the beginning; filename='/var/ig/logs/orderserver/os-gc.2.log' After that syslog starts to send the contents of the file from the start of the file again, after a while it restart this process. In a seemingly endless loop. As you can see the pid changes between the disconnect and the message about mismatching size/inode. I'm trying to work out what causes it and how to fix it, any suggestions? Best, Henric ________________________________ The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index Ltd is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.

1 0