- syslog-ng - lists.balabit.hu

Critical hang while opening tty
by Jean-Baptiste Quenot 07 Jun '09

07 Jun '09

Hi there, I'm using syslog-ng 2.0.9-4.1 on Ubuntu 9.04. In the configuration file, emergency messages are configured to be logged to all ttys. Sometimes syslog-ng blocks the whole system while opening /dev/ttyS0: # pgrep -f syslog-ng # strace -f -p 3172 Process 3172 attached - interrupt to quit open("/dev/ttyS0", O_WRONLY|O_NOCTTY|O_APPEND I'm surprised because I saw the same bug report, and this was supposed to be fixed in Ubuntu's 2.0.9-1 version as stated in the changelog at See http://changelogs.ubuntu.com/changelogs/pool/universe/s/syslog-ng/syslog-ng… The changelog says: syslog-ng (2.0.9-1) unstable; urgency=low * New upstream version. (Closes: #397650, #447105) This corresponds to this bug: syslog-ng hangs writing to /dev/ttyN http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=397650 I notice the bug is fixed in syslog-ng: http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=commit;h=c5cc92d54d2e33b… Is there maybe another call to tty open() that was not covered by alarm()? The issue happens on several machines, but I don't know what triggers this. Reloading syslog-ng fixes the problem. Thanks in advance, -- Jean-Baptiste Quenot http://jbq.caraldi.com/

2 2

TLS - Mutual authentication issue
by Hahusseau, Thomas 07 Jun '09

07 Jun '09

Hi, I'm using syslog-ng OSE with encrypted message transport thanks to TLS for few week. Now I try to activate the mutual authentication option. I have several issues with the TLS mutual authentication logs error : Jun 4 16:01:31 desktop syslog-ng[26644]: SSL error while reading stream; tls_error='SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca' Jun 4 16:01:31 desktop syslog-ng[26644]: I/O error occurred while reading; fd='14', error='Connection reset by peer (104)' Jun 4 16:01:31 desktop syslog-ng[26644]: Syslog connection closed; fd='14', client='AF_INET(10.254.1.172:43751)', local='AF_INET( <http://0.0.0.0:9999> 0.0.0.0:9999) This is samples of config file: Server : source s_net_tls { tcp(port(9999) tls(key_file("/etc/pfc/credentials/Server/server.key") cert_file("/etc/pfc/credentials/Server/server.pem") ca_dir("/etc/pfc/credentials/CA/") # peer_verify(optional-untrusted) peer_verify(required-trusted) ) ); }; Client : destination d_remote_server_tls { tcp("10.254.1.141" port(9999) tls(ca_dir("/etc/pfc/credentials/CA") key_file("/etc/pfc/credentials/Client/client.key") cert_file("/etc/pfc/credentials/Client/client.pem") peer_verify(required-trusted) # peer_verify(optional-untrusted) )); }; here is how I generated my CA certificate, server and client certificate : openssl genrsa 1024 > CA/ca.key openssl req -new -x509 -days 365 -key CA/ca.key -out CA/ca.cert cat CA/ca.cert CA/ca.key > CA/ca.pem openssl genrsa 1024 > Client/client.key openssl req -new -key Client/client.key -out Client/client.csr openssl x509 -req -days 365 -in Client/client.csr -CA CA/ca.cert -CAkey CA/ca.key -set_serial 01 -out Client/client.cert cat Client/client.cert Client/client.key > Client/client.pem openssl genrsa 1024 > Server/server.key openssl req -new -key Server/server.key -out Server/server.csr openssl x509 -req -days 365 -in Server/server.csr -CA CA/ca.cert -CAkey CA/ca.key -set_serial 01 -out Server/server.cert cat Server/server.cert Server/server.key > Server/server.pem Of course I done the link with Openssl x509 -noout -hash -in ca.pem Ln -s ca.pem XXXX If anyone can help me, or give a step by step procedure that works. I also tried the procedure described in the "Syslog-nd admin guide" it doesn't work too. Regards Thomas ------------------- Thomas Hahusseau Apprenti ingénieur EADS - DS / ENST Bretagne

2 1

udp drops
by Jan Schaumann 06 Jun '09

06 Jun '09

Hello, I have a FreeBSD 6.2 (amd64) host where I'd like to replace the stock syslogd with syslog-ng (3.0.2). This host receives a lot of syslog messages per second from a large number of clients via UDP. The stock syslogd configuration is trivial: *.* /var/log/all This host currently drops about 2-4% of all UDP packets, syslog takes about 50-65% of one CPU. A drop-in replacement configuration using syslog-ng: options { create_dirs(yes); use_dns(no); }; template t_default { template("${DATE} <${FACILITY}.${PRIORITY}> ${HOST} ${MSG}\n"); }; source s_standard { file("/dev/klog"); internal(); udp(); unix-dgram("/var/run/log"); }; destination d_all { file("/var/log/all" template(t_default) ); }; # *.* /var/log/all log { source(s_standard); destination(d_all); }; syslog-ng uses about 90% of one CPU and drops between 15% and 20% of UDP packets (and, based on traffic patterns and logfile size, concurrent logfile rotation etc. even as high as 30%). I've tried a number of things to improve this performance, including: log_fetch_limit(100); log_iw_size(10000); flush_lines(100000); flush_timeout(10); in the global options and log_fifo_size(100000) in the destination definition with flags(flow-control) in the log definition. The best I was able to get with these numbers was 5-7% of UDP drops (ie still double of what the stock syslogd drops). I also tried adjusting "so_rcvbuf" for UDP with no noticable difference. Now consider that I did not do any sysctl tuning, as those should equally influence the stock syslog and I'm trying to sort out why one performs so significantly better than the other. Leaving aside any of the things I can do with syslog-ng further down the road (such as filtering and intentionally dropping certain messages), what can I do to get syslog-ng up to the same performance as the stock syslogd? Many thanks in advance for any pointers and help. -Jan

4 8

New to syslog-ng, basic setup help reuqested
by lance raymond 05 Jun '09

05 Jun '09

Afternoon, looking to do some real basic logging for our webservers. They are all running ubuntu-server 8.10, and I have a standalone box which will act as the 'log server'. They have webmin installed, so I setup using that as it seemed pretty straight forward. So the server is running and when I start one of the webservers which I configured, I recive the following error; * Starting system logging syslog-ng Error binding socket; addr='AF_UNIX(/var/log/apache2/ws8.mydomain.com-access_log)', error='Address already in use (98)' I thought it might be due to apache running, so stopped that, but still the same error. the conf file is HUGE, and 95% is a stock install, so I will just put what the client / server has that is specific to this item I wish to remote log. Client: destination WFLogger { udp(192.168.2.60); }; source domain_access { unix-stream("/var/log/apache2/ws8.domain.com-access_log"); }; log { source(domain_access); destination(WFLogger); flags(catchall); }; On the server, I don't see anything that stands out that say's "accept connections" so not sure if there is anything special that has to be done, but the error on the client is immediate so that part seems to be a local issue. Thanks

2 2

no compatibility between syslog and apache log priority
by gatfi sami 05 Jun '09

05 Jun '09

Hi i a successfully configured a centralised syslog server using syslog-ng 2.09 here is the structure: syslog-client(read apache error log) -------> *send* to a syslog-relay --------->* forword* to a syslog server ----> centralise in a mysql tabe 'logs' CREATE TABLE `syslogtopnet`.`log` ( `seq` int(10) unsigned NOT NULL auto_increment, `host` varchar(60) NOT NULL, `facility` varchar(40) NOT NULL, `*priority*` varchar(40) NOT NULL, `*level*` varchar(40) NOT NULL, `date_log` date NOT NULL, `time_log` time NOT NULL, `program` varchar(40) NOT NULL, `*message*` text NOT NULL, PRIMARY KEY (`seq`), KEY `index_log` (`host`) ) ENGINE=InnoDB AUTO_INCREMENT=684 DEFAULT CHARSET=latin1; my problème is : i find for exemple priority ='*Notice*' and level='*Notic*e' but in the message i find *[warn]* here is an exemple: INSERT INTO `syslogtopnet`.`log` VALUES (320,'192.168.9.195','user',* 'notice'*,*'notice'*,'2009-05-19','19:33:24','','[Tue May 19 19:33:24 2009] *[warn]* Init: Session Cache is not configured [hint: SSLSessionCache]'); i discovered that apache log level are different then syslog's so !!!!! 1 wired isnt it lease is there a solution to keep the same level in the original priority? 2 beside in more than 20589 line in my table priority and level have the same values. is it normal that the level and priority are the same how can this be? thank you

2 1

New db-parser format and other improvements
by ILLES, Marton 05 Jun '09

05 Jun '09

Hi, I have already mentioned earlier that I am planning to back-port the new db-parser patterndb format to syslog-ng OSE. After Bazsi's plan on the 4.0 roadmap I had no more excuse and did the backporting last week. So anyone interested in the new format can find the code in my git tree. Format 2 is more flexible and hopefully easier to use. Meanwhile I removed support for version 1 format. Converting between the formats is very straight forward, but I also plan to add an XSLT for conversions. The new format has support and flexibility for couple of interesting things: * rules for one program can be separated into multiple rulesets * rulesets can be defined without program pattern where these act as a fail-back rule sets if no program pattern is matching or when message does not have program part * one rule can have any number of pattern attached to it (sometimes the same message has multiple format, just think about multilingual messages) * tags can be attached to rules (IMPORTANT: as of now tags are not considered, but at least the format has place for them!) The new patterndb format looks like this: <patterndb version="2" pub_date="2009-06-04"> <ruleset name="testset" id="4182799f-9d1e-48a2-aa1e-aa4bdac6aff8"> <rules> <rule provider="balabit" id="4a3b027a-3c5c-4c55-a28b-ce8052e8a327" class="system"> <tags> <tag>login</tag> </tags> <patterns> <pattern>HERE COMES THE PATTERN</pattern> </patterns> </rule> </rules> </patterndb> I have added new parsers to the db-parser: * IPv6 to parser IPv6 addresses with all the special notation styles * IPvANY to match either IPv4 or IPv6 addresses * NUMBER parser now handles hexadecimal numbers with the 0x or 0X prefix * DOUBLE parser to match and parse non integer I am trying to find some time in the near future to add the missing tag support and provide some more details on the new format. cheers, Marton -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

no compatibility between syslog and mysql log priority more
by gatfi sami 04 Jun '09

04 Jun '09

Hi i a successfully configured a centralised syslog server using syslog-ng 2.09 here is the structure: syslog-client(read mysql error log) -------> *send* to a syslog-relay --------->* forword* to a syslog server ----> centralise in a mysql tabe 'logs' CREATE TABLE `syslogtopnet`.`log` ( `seq` int(10) unsigned NOT NULL auto_increment, `host` varchar(60) NOT NULL, `facility` varchar(40) NOT NULL, `*priority*` varchar(40) NOT NULL, `*level*` varchar(40) NOT NULL, `date_log` date NOT NULL, `time_log` time NOT NULL, `program` varchar(40) NOT NULL, `*message*` text NOT NULL, PRIMARY KEY (`seq`), KEY `index_log` (`host`) ) ENGINE=InnoDB AUTO_INCREMENT=684 DEFAULT CHARSET=latin1; my problème is : i find for exemple priority ='*Notice*' and level='*Notic*e' but in the message i find *[warn]* here is an exemple: INSERT INTO `syslogtopnet`.`log` VALUES (320,'192.168.9.195','user',* 'notice'*,*'notice'*,'2009-05-19','19:33:24','','[Tue May 19 19:33:24 2009] *[warn]* Init: Session Cache is not configured [hint: SSLSessionCache]'); INSERT INTO `syslogtopnet`.`log` VALUES (24445,'192.168.9.197','user','notice','notice','2009-06-04','17:21:02','090604','090604 17:21:02 mysqld started'), (24446,'192.168.9.197','user','notice','notice','2009-06-04','17:21:02','090604','090604 17:21:02 mysqld started'), (24447,'192.168.9.197','user',*'notice'*,*'notice'*,'2009-06-04','17:21:02','InnoDB','InnoDB: Unable to lock ./ibdata1, *error: 11*'), i discovered that apache log level are different then syslog's so !!!!! 1 wired isnt it lease is there a solution to keep the same level in the original priority? 2 beside in more than 20589 line in my table priority and level have the same values. is it normal that the level and priority are the same how can this be? thank you

1 0

Beta-testers needed for documentation draft
by Robert Fekete 03 Jun '09

03 Jun '09

Dear syslog-ng users, I am Robert Fekete, the writer of the syslog-ng Administrator Guide (and of the documentation of other BalaBit products as well). First of all, thank you very much for all of you who took the time to send me feedback and corrections about the admin guide. In the last few days, I have prepared an update for the admin guide that covers the news of the recent syslog-ng 3.0.2 OSE release, and also includes documentation for the pattern database feature. However, it seems that currently Bazsi and the other developers and testers will not have time to review the docs for a few weeks. So if you have some time and would line to test the pattern database, the draft is available at http://people.balabit.hu/frobert/syslog-ng-v3.0-guide-admin-en-6ed-draft.pdf If you find anything that is unclear, not accurate, or otherwise wrong (especially in the config examples), please let me know either here on the list, or via the documentation(a)balabit.com address. Among others, the following sections need review: Section 2.12, “Classifying log messages” (p. 15), Section 3.9, “Classifying messages” (p. 61), and Section 8.6.2, “Pattern databases” (p. 201). Thank you very much for your help. Regards, Robert Fekete

2 1

syslog-ng 3.0.2 issues: Extraneous colon (:) in forwarded log; two instances of syslog-ng
by Marvin Nipper 03 Jun '09

03 Jun '09

Hi. I'm been running syslog-ng 2.0.9 for eons, and wanted to get 3.0.2 in place, but have run into two issues. First, the prerequisite info: This is on Solaris 10 x86, and I'm running the eventlog-0.2.9 component. It has all compiled cleanly. I'm using the same script to initialize syslog-ng as I was using before. In the conf file, added in some value() parameters on my filters, in accordance with the 3.0 "warnings", and added in the @version directive, but other than that, it's essentially the same (reasonably simple) conf file I was using with my 2.0.9 version of the code. For reference, I've appended my conf file at the end of the email. "Generally" syslog-ng, seems to start-up fine, and does feed the output files that I've indicated in the conf file. If I didn't "look too closely", I would have thought it was all fine, but... My problems: As part of my test environment, I'm forwarding data from another (production 2.0.9) server, to my test server, so that I can see how things are working. Here are some (truncated) messages, showing both the original message (as it looks in the output file on the 2.0.9 server), and the second message being what gets written to the file by the 3.0.2 component: Original: May 29 09:15:05 10.132.240.25 id=firewall sn=0006B10B2E88 <truncated> Received and written by 3.0.2: May 29 09:15:05 10.132.240.25 id=firewall: sn=0006B10B2E88 <truncated> So the "bug" (I assume it to be a bug) is that "colon" that is now sitting just behind the word "firewall". When my test server is running a 2.0.9 daemon, these two lines match, but the 3.0.2 daemon has inserted that colon. I can only assume that this is tied to a bug in the filtering activity, because if you look at my conf file, you will see these lines: filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; Bottom line, I am doing a filter that happens to "match on" that exact "firewall" string, and it seems like more than just coincidence that this is the exact location where that unwanted colon now appears. It seems like that filtering process has somehow injected that character into the output. (Maybe I'm just missing some "new functionality or feature" in the 3.0 docs, that would intentionally cause this?) Anyway... I'm obviously looking for any input as to why this is different, and/or how to fix it (or even just work-around the problem until it is fixed). And secondly, as my subject line implies, when I run my "/etc/init.d/syslog-ng start" scripting (the same one I've always used), I find, right away, that I now have two instances of syslog-ng running. I saw a bug report from back in 2006, mentioning a similar behavior that was related to (I think) syslog-ng restarting itself, but that was supposedly fixed. I'm ass-u-me-ing that something is now causing the daemon to restart itself, but again, when I run my 2.0.9 code on place of the 3.0.2 code, I get just the one copy running. Again, I'm looking for any guidance as to something that might be driving that behavior (and as I said, maybe it's just something stupid on my part, because I'm missing something "new" in the 3.0 behavior). Thanks, in advance, for any help and/or guidance as to what these issues might be. I'm glad to provide further information if it will help. Marvin Nipper syslog-ng.conf: @version:3.0 options { dir_perm(0755); perm(0600); chain_hostnames(no); keep_hostname(yes); log_fifo_size(1000); dns_cache_size(5000); dns_cache_expire(86400); dns_cache_expire_failed(86400); }; source any_udp { udp(); }; source any_tcp { tcp(port(601)); }; destination SEC {pipe("/tmp/sec"); }; destination routers_log { file("/var/adm/log/routers.log" create_dirs(yes)); }; destination ravlin_log { file("/var/adm/log/ravlin.log" create_dirs(yes)); }; destination windows_log { file("/var/adm/log/windows.log" create_dirs(yes)); }; destination workstation_log { file("/var/adm/log/workstation.log" create_dirs(yes)); }; destination catch-all_log { file("/var/adm/log/catch-all.log" create_dirs(yes)); }; destination test { file("/var/adm/log/test.log" create_dirs(yes)); }; destination dev_null {}; filter f_4 { facility(syslog) and level(info..emerg); }; log { source(any_udp); filter(f_4); destination(windows_log); flags(final); }; filter f_1 { facility(local5) and level(debug..emerg); }; log { source(any_udp); filter(f_1); destination(routers_log); flags(final); }; filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; filter SonicWallMsgs { match("m=97" value("MSG")) or match("m=98" value("MSG")) or match("m=537" value("MSG")); }; log { source(any_udp); filter(SonicWallNoise); destination(dev_null); flags(final); }; filter f_3 { facility(local0) and level(debug..emerg); }; log { source(any_udp); filter(f_3); destination(ravlin_log); flags(final); }; log { source(any_tcp); destination(workstation_log); flags(final); }; log { source(any_udp); destination(catch-all_log); flags(final); }; This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

2 2

Help on trying to build syslog-ng 3.0.2 on RHEL AS 4
by Joe Wood 03 Jun '09

03 Jun '09

Hello. I'm having troubles getting syslog-ng to compile on RHEL AS 4, with all appropriate libs installed. The configure script errors out with: checking pkg-config is at least version 0.14... yes checking for GLIB - version >= 2.10.1... no *** Could not run GLIB test program, checking why... *** The test program failed to compile or link. See the file config.log for the *** exact error that occured. This usually means GLIB is incorrectly installed. configure: error: Cannot find GLIB version >= 2.10.1: is pkg-config in path? When I examine config.log, I see the following: configure:8496: gcc -o conftest -g -O2 -Wall conftest.c >&5 conftest.c:42:18: glib.h: No such file or directory conftest.c: In function `main': conftest.c:48: error: `glib_major_version' undeclared (first use in this function) conftest.c:48: error: (Each undeclared identifier is reported only once conftest.c:48: error: for each function it appears in.) conftest.c:48: error: `glib_minor_version' undeclared (first use in this function) conftest.c:48: error: `glib_micro_version' undeclared (first use in this function) configure:8502: $? = 1 configure: failed program was: However, glib.h and all appropriate glib headers are installed in /usr/local/include, which even when specified to configure (through --includedir) it doesn't seem to pick up on.... Anyone encountered this before? Joe

2 1