June 2010 - syslog-ng - lists.balabit.hu

spoof-source performance issues
by Martin Holste 01 Jul '10

01 Jul '10

I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?

5 12

Memory leak can occur with supressed duplicate log messages
by andy kitchingman 30 Jun '10

30 Jun '10

Hi Due to my company firewall, I do not have access to the git public repository, so I will post this here. I believe there is a memory leak when duplicate messages are suppressed and messages cannot be consumed: logwriter.c: log_writer_last_msg_flush(LogWriter *self) ------------------------------------------- log_queue_push_tail(self->queue, m, &path_options) can return FALSE (message cannot be consumed), in which case the local LogMessage m is leaked. The fix should be something like if (!log_queue_push_tail(self->queue, m, &path_options)) { log_msg_unref(m); } Regards Andy Kitchingman

2 1

[Bug 89] New: DOCUMENTATION: incorrect structure of embedded log statement.
by bugzilla＠bugzilla.balabit.com 30 Jun '10

30 Jun '10

https://bugzilla.balabit.com/show_bug.cgi?id=89 Summary: DOCUMENTATION: incorrect structure of embedded log statement. Product: syslog-ng Version: 3.1.x Platform: PC OS/Version: Windows Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: alex-pub.balabit(a)reflexion.net Type of the Report: --- Estimated Hours: 0.0 In syslog-ng-ose-v3.1-guide-admin-en.pdf. Topic "4.5.1. Using embedded log statements" Shows incorrect structure of embedded log statement: log { source(s1); source(s2); ... optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... destination(d1); destination(d2); ... <<<<<<<<<<<<<<<<<<<<<< ! Wrong position ! flags(flag1[, flag2...]); <<<<<<<<<<<<<<<<<<<<<< #embedded log statement log { optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... destination(d1); destination(d2); ... #another embedded log statement log { optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... destination(d1); destination(d2); ... }; }; }; If you follow this structure, the resulting config file will not compile, because `flags` element must appear after the embedded log statement, at least according to cfg-grammar.y: log_stmt : '{' log_items log_forks log_flags '}' { LogPipeItem *pi = log_pipe_item_append_tail($2, $3); $$ = log_connection_new(pi, $4); } ; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

WARNING: the match() filter without the use of the value() ...
by ml ml 29 Jun '10

29 Jun '10

Hello List, i am getting this error: WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Restarting syslog-ng: Stopping syslog-ng: OK. Starting syslog-ng: WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; with this config: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # ### 1.) OPTIONS options { long_hostnames(off); }; source src { unix-stream("/dev/log"); internal(); }; source kernsrc { file("/proc/kmsg"); }; ### 2.) DESTINATION destination ldap { file("/var/log/ldap.log"); }; destination authlog { file("/var/log/auth.log"); }; destination syslogmsg { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kern { file("/var/log/kern.log"); }; destination user { file("/var/log/user.log"); }; destination mail { file("/var/log/mail.log"); }; destination lighttpd { file("/var/log/lighttpd.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination all { file("/var/log/all.log"); }; destination fcron { file("/var/log/fcron.log"); }; destination sshd { file("/var/log/sshd.log"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination console { usertty("root"); }; ### 3.) FILTERS filter f_ldap { match("slapd"); }; filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_mail { facility(mail); }; filter f_user { facility(user); }; filter f_debug { not facility(auth, authpriv, news, mail) ; }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; filter f_lighttpd { match("lighttpd"); }; filter f_fcron { match("fcron"); }; filter f_sshd { match("sshd"); }; filter f_syslog { not facility(authpriv, mail) and not filter(lapd) and not filter(sshd) and not filter(fcron) ; }; log { source(src); filter(f_sshd); destination(sshd); }; log { source(src); filter(f_fcron); destination(fcron); }; log { source(src); filter(f_ldap); destination(ldap); }; log { source(src); filter(f_syslog); destination(syslogmsg); }; log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(kernsrc); filter(f_kern); destination(kern); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_user); destination(user); }; log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); }; log { source(src); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(src); filter(f_debug); destination(debug); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_emergency); destination(console); }; log { source(src); destination(all); }; Any idea what i am doing wrong?

3 5

Two instances of syslog-ng on the same host?
by John R. Dunning 29 Jun '10

29 Jun '10

Hi wizards. Apologies if this is an FAQ or something, but I've dug all around and failed to find the answer. I have a system on which, for reasons I'd rather not go into here, it makes sense to run two instances of syslog-ng, one for standard logging of local events, the other acting as a proxy for a flock of other systems. The proxy starts first, very early in the init sequence, the regular one starts later. This all worked great with syslog-ng 2, but I recently upgraded to version 3.1.1 and I can't get it to work correctly. The proxy instance is supposed to only be listening on a tcp socket, but it seems to also be opening the AF_UNIX socket to /dev/log. This causes the launch of the main instance to fail. I've been through the docs, but it's not obvious to me how to get syslog-ng to start without opening the socket to /dev/log. Hints? Thanks in advance...

4 7

Re: [syslog-ng] Two instances of syslog-ng on the same host?
by Rory Toma 27 Jun '10

27 Jun '10

As a point of reference, I have a setup with over 200 syslog-ng processes and it works fiine. "Patrick H." <syslogng(a)feystorm.net> wrote: >Yes, that should work just fine. I run a box with 4 syslog instances and >they all place nice together. My setup isnt exactly the same, but I have >no problems which you are experiencing. My setup: I have a master >syslog-ng process that does nothing but accept connections from remote >hosts and from the local system. That process then relays the messages >to other local syslog-ng processes which do the actual work (lots and >lots of parsing). >I only included the options and source sections as theyre the only ones >that matter. > >Master syslog-ng process that accepts from local & remote & forwards to >other local processes >@version: 3.0 ># syslog-ng configuration file. ># ># This should behave pretty much like the original syslog on RedHat. But ># it could be configured a lot smarter. ># ># See syslog-ng(8) and syslog-ng.conf(5) for more information. ># > >options { > time_reopen(1); > use_dns(no); > use_fqdn(yes); > keep_hostname(yes); > create_dirs(yes); > perm(0644); > dir_perm(0755); > log_iw_size(50000); > log_fifo_size(100000); > #time_sleep(1); >}; > >source s_sys { > file("/proc/kmsg" program-override("kernel")); > unix-stream ("/dev/log"); > internal(); >}; >source s_net { > tcp(ip(0.0.0.0) port(514) max-connections(1000)); > udp(ip(0.0.0.0) port(514)); >}; > > >syslog-ng that accepts the forwarded messages >@version: 3.0 ># syslog-ng configuration file. ># ># This should behave pretty much like the original syslog on RedHat. But ># it could be configured a lot smarter. ># ># See syslog-ng(8) and syslog-ng.conf(5) for more information. ># > >options { > time_reopen(1); > long_hostnames(off); > use_dns(no); > use_fqdn(no); > keep_hostname(yes); > create_dirs(yes); > perm(0644); > dir_perm(0755); > #log_fetch_limit(100000); > #log_iw_size(200000); > #log_fifo_size(400000); > flush_lines(50); > flush_timeout(5000); > #stats_freq(10); > #stats_level(2); > #time_sleep(1); >}; > > >source s_master { > #syslog(ip(127.0.0.1) port(515) transport('tcp') so_keepalive(yes) >log_iw_size(1000)); > tcp(ip(127.0.0.1) port(515) flags('syslog-protocol')); >}; >source s_syslog { > internal(); >}; > > > > > > > > > > >Sent: Saturday, June 26, 2010 6:08:42 PM >From: John R. Dunning <jrd(a)jrd.org> >To: syslog-ng(a)lists.balabit.hu >Subject: [syslog-ng] Two instances of syslog-ng on the same host? >> Hi wizards. Apologies if this is an FAQ or something, but I've dug >> all around and failed to find the answer. >> >> I have a system on which, for reasons I'd rather not go into here, it >> makes sense to run two instances of syslog-ng, one for standard >> logging of local events, the other acting as a proxy for a flock of >> other systems. >> >> The proxy starts first, very early in the init sequence, the regular >> one starts later. >> >> This all worked great with syslog-ng 2, but I recently upgraded to >> version 3.1.1 and I can't get it to work correctly. The proxy >> instance is supposed to only be listening on a tcp socket, but it >> seems to also be opening the AF_UNIX socket to /dev/log. This causes >> the launch of the main instance to fail. >> >> I've been through the docs, but it's not obvious to me how to get >> syslog-ng to start without opening the socket to /dev/log. Hints? >> Thanks in advance... >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.campin.net/syslog-ng/faq.html >> >> > >______________________________________________________________________________ >Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >FAQ: http://www.campin.net/syslog-ng/faq.html >

1 0

few questions about patterndb
by Hendrik Pahl 25 Jun '10

25 Jun '10

Hi folks, i'm currently testing the patterndb with syslog-ng. After a few test runs,there are some open questions. Will there be some future releases of the patterndb by balabit? Last Snapshot has a timestamp from sep. 2009. Or is there a community somewhere out there, that i haven't seen yet?:-) I tested the classification into different logfiles (system,violation,unknown) with 1.5M lines of logfiles, and there was just one rule matching 'em all: 734adfc8-8a4d-40ad-923d-9d3fbabda34b, which is something about a newly connected usb mass storage device. This behaviour can't be correct? The patternfile "system-apache2.xml" still containts just plain messages, no XML anywhere - i thougt this should have been fixed yet? After some testing with logging messages to file like "messages-system/messages-violation/messages-unknown" i'm just asking myself about the profit i get when using patterndb. With a standard syslog-ng i usually can log things into files divided by the priority of the messages - that's all i want it to do, just have a logfile where all errors und failures are logged, just to get the pure number of lines down. When controlling a logfile i usually don't really care about a cronjob to be successful. What is the profit of using patterndb right there? Thanks in advance, i.A. Hendrik Pahl System Engineering team! datentechnik GmbH & Co.KG Werner von Siemens Straße 12a 49124 Georgsmarienhuette Tel.: +49 (0)5401-8226-50 Fax : +49 (0)5401-8226-55 E-Mail: pahl(a)team-datentechnik.de Internet: www.team-datentechnik.de HRA 110397, Amtsgericht Osnabrück Geschäftsführung: Reemt Lükenga Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Vielen Dank. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

5 8

proper filtering messages with java Exceptions
by Franciszek Zmarły 24 Jun '10

24 Jun '10

Hello I have a problem with filtering messages having java Exceptions. I send logs from many jvm processes to one syslog-ng server and save them to different files. Example one message from log4j (dump from wireshark) <139>Jun 18 10:53:03 td2/td2 [PROGRAM NAME - MainThread] ERROR - FATAL ERROR <139>Jun 18 10:53:03 td2/td2 java.lang.Exception: Invalid value TryCount: 0 <139>Jun 18 10:53:03 td2/td2 at com.td.Sender.initialize(Sender.java:30) <139>Jun 18 10:53:03 td2/td2 td2 at com.td.Main.main(Main.java:123) I use changed thread name (PROGRAM NAME) as filter in syslog-ng configuration filter f_td_p1 { match("PROGRAM NAME"); }; but in this configuration ony first line of message are saved (from log file) 2010-06-18 10:53:03 td2 [PROGRAM NAME-MainThread] ERROR - FATAL ERROR Is it possible to correct save all logs ? Of course all messages in examle is one log message. Log4j save it like this: 2010-06-18 10:53:03,242 [PROGRAM NAME-MainThread-MainThread] ERROR - FATAL ERROR java.lang.Exception: Invalid value TryCount: 0 at com.td.Sender.initialize(Sender.java:30) at com.td.Main.main(Main.java:123) I want to obtain same effect in syslog-ng. Have you any suggestion ? Please help. Regards Frank

4 4

FreeBSD 8 LOG_SECURITY and LOG_AUTH confused
by Brian A. Seklecki 23 Jun '10

23 Jun '10

Guys something shit the bed with 3.0.3 on 8.0-PL: ----- config ---------------- source src { unix-dgram("/var/run/log"); filter f_security { facility(security); }; filter f_auth { facility(auth); }; destination security { file("/var/log/security" template(cfi_template)); }; destination authlog { file("/var/log/auth.log" template(cfi_template)); }; log { source(src); filter(f_security); destination(security); flags(final); }; log { source(src); filter(f_auth); destination(authlog); flags(final); }; ------------------------- # /usr/local/sbin/syslog-ng -p /var/run/syslog.pid -tevdF Syslog connection established; fd='5', server='AF_INET(192.168.128.XXX:514)', local='AF_INET(0.0.0.0:0)' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.0.3' ^Z [5]+ Stopped /usr/local/sbin/syslog-ng -p /var/run/syslog.pid -tevdF [root@cfi-db-corp ~]# bg [root@cfi-db-corp ~]# echo test | logger -p auth.info Incoming log entry; line='<38>Mar 18 15:57:40 bseklecki: test' Filter rule evaluation begins; filter_rule='f_ams' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ams' Filter rule evaluation begins; filter_rule='f_auth' Filter node evaluation result; filter_result='match', filter_type='facility' Filter rule evaluation result; filter_result='match', filter_rule='f_auth' Filter rule evaluation begins; filter_rule='f_ams' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ams' Filter rule evaluation begins; filter_rule='f_authpriv' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_authpriv' Filter rule evaluation begins; filter_rule='f_kern' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_kern' Filter rule evaluation begins; filter_rule='f_user' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_user' Filter rule evaluation begins; filter_rule='f_mail' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_mail' Filter rule evaluation begins; filter_rule='f_security' Filter node evaluation result; filter_result='match', filter_type='facility' Filter rule evaluation result; filter_result='match', filter_rule='f_security' Initializing destination file writer; template='/var/log/security', filename='/var/log/security' The two fixes I can find are: - Move: "log { source(src); filter(f_security);.." below "log { source(src); filter(f_auth);..." ...wait, what?! - Remove reference to LOG_SECURITY /usr/include/syslog.h defines for Facility->index mappings haven't changed since RELENG_6, so I'm not sure what to make of this. Very strange, ~BAS

2 2

lists.balabit.com / lists.balabit.hu SSL Cert
by Brian A. Seklecki 23 Jun '10

23 Jun '10

All: The list headers have balabit.hu, but the SSL cert is for *.com. Sounds like a nightmare. I suspect Mailman domain name change is a nightmare. Maybe a good chance to experiment with RFC3546 / SNI (SSL Vhosting) :) ~BAS

1 0