- syslog-ng - lists.balabit.hu

[Bug 86] New: flush_timeout has no effect
by bugzilla＠bugzilla.balabit.com 09 Jul '10

09 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=86 Summary: flush_timeout has no effect Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 After upgrading to version 3.0.7 OSE the flush_timeout option has no effect for file destinations. The messages are buffered until flush_lines is reached and then they are flushed. TCP destinations are no affected by this issue. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

Files still opened after time_reap
by Erwan Miran 09 Jul '10

09 Jul '10

Hi, I'm using syslog-ng 2.0.0 (yes I know...) on a debian loghost. Files generated via df_monitor (see configuration extract below) are deleted by the monitoring process about 4 min after having been written by syslog-ng. My problem is that these files remain opened (lsof | grep deleted) by syslog-ng after deleted whereas time_reap option is set to 30s. I googled the issue and just found https://lists.balabit.hu/pipermail/syslog-ng/2003-January/004416.html which is not my exact case (macros are actually used in filenames). I didn't find any bugfix/feature on time_reap in the changelog of subsequent versions ... Thanks in advance for your help Regards Erwan options { chain_hostnames(0); time_reopen(10); time_reap(30); log_fifo_size(8192); create_dirs(yes); group(adm); perm(0660); dir_group(adm); dir_perm(0755); use_dns(no); log_msg_size(4096); use_fqdn(yes); keep_hostname(yes); use_time_recvd(no); }; source s_machines { tcp(ip("890.123.456.789") port(514) max_connections(100)); }; template tpl_monitor { template("$S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC $FULLHOST ==$LEVEL== [$PROGRAM] $MESSAGE\n"); template_escape(no); }; destination df_monitor { file("/var/log/monitor/$R_YEAR-$R_MONTH-$R_DAY-$R_HOUR-$R_MIN" template(tpl_monitor) group(monitor) dir_perm(0775) dir_group(monitor) owner(loghost-maintainer)); }; destination dh_backup-loghost { tcp("123.456.789.890" port(514)); }; log{ source(s_machines); destination(df_monitor); destination(dh_backup-loghost); };

2 3

file() source driver on regular file
by Risto Vaarandi 07 Jul '10

07 Jul '10

hi all, is file() driver intended for reading kernel messages only on Linux, or does it work for regular files also? I've tried to get the following simple configuration working, but without success: source textfile { file("/home/risto/test.log" flags(no-parse) follow-freq(1)); }; destination testlog { file("/var/log/test.log"); }; log { source(textfile); destination(testlog); }; The source definition is identical to the example from the syslog-ng administrator guide (I've played with the options, but the result is the same). When syslog-ng starts up, it produces the following error messages: Follow-mode file source not found, deferring open; filename='/home/risto/test.log' The current log file has a mismatching size/inode information, restarting from the beginning; filename='/home/risto/test.log' If /home/risto/test.log does not exist, only the first message is produced. Is reading from regular files only supported by PE? (I am using the 3.0.7 open source version of syslog-ng.) kind regards, risto

2 1

The latest version of eventlog?
by Sergei Zhirikov 05 Jul '10

05 Jul '10

Hi, I'm wondering what is the latest version of the eventlog library? If I look here: http://www.balabit.com/downloads/files/eventlog/0.2/ then the latest seems to be 0.2.9. Although on my other computer I have 0.2.10 installed (built from source), but I can't remember where I got it from. Is there something missing on the download page? -- Thanks and regards, Sergei.

2 2

POLLERR occurred while idle
by Tom Ritter 05 Jul '10

05 Jul '10

I'm getting a large volume of messages that look like this: 23:09:53 hn syslog-ng[30]: POLLERR occurred while idle; fd='13' 23:09:54 hn syslog-ng[30]: POLLERR occurred while idle; fd='14' 23:09:55 hn syslog-ng[30]: POLLERR occurred while idle; fd='13' 23:09:56 hn syslog-ng[30]: POLLERR occurred while idle; fd='14' 23:09:57 hn syslog-ng[30]: POLLERR occurred while idle; fd='13' 23:09:58 hn syslog-ng[30]: POLLERR occurred while idle; fd='14' 23:09:59 hn syslog-ng[30]: POLLERR occurred while idle; fd='13' I have no idea what they are but they won't stop =( I looked at the file descriptors: # ls /proc/30/fd total 0 lr-x------ 1 root root 64 Jul 4 23:21 0 -> /dev/null l-wx------ 1 root root 64 Jul 4 23:21 1 -> /dev/null l-wx------ 1 root root 64 Jul 4 23:21 10 -> /var/log/messages/messages l-wx------ 1 root root 64 Jul 4 23:21 11 -> /var/log/hn/20100704.log l-wx------ 1 root root 64 Jul 4 23:21 13 -> pipe:[28238824] l-wx------ 1 root root 64 Jul 4 23:21 2 -> /dev/null lrwx------ 1 root root 64 Jul 4 23:21 3 -> socket:[28238645] l-wx------ 1 root root 64 Jul 4 23:21 4 -> pipe:[28238641] lrwx------ 1 root root 64 Jul 4 23:21 5 -> socket:[28238650] lr-x------ 1 root root 64 Jul 4 23:21 6 -> /proc/kmsg l-wx------ 1 root root 64 Jul 4 23:21 7 -> /var/log/syslog/syslog.log l-wx------ 1 root root 64 Jul 4 23:21 8 -> /var/log/fw/important.log l-wx------ 1 root root 64 Jul 4 23:21 9 -> /var/log/debug/debug.log And I ran strace looking for fd's 13 and 14. Here are some grep results: strace.log.30964:open("/dev/null", O_WRONLY) = 14 ---- strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 29014) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 27100) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 25102) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 23291) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 21398) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 19680) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = ? ERESTART_RESTARTBLOCK (To be restarted) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 18183) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 16697) = 1 ([{fd=13, revents=POLLERR}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 0) = 0 (Timeout) strace.log.30962:poll([{fd=13, events=POLLOUT}], 1, 0) = 1 ([{fd=13, revents=POLLOUT}]) strace.log.30962:poll([{fd=13, events=0}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}, {fd=5, events=POLLIN}], 4, 15221) = 1 ([{fd=13, revents=POLLERR}]) ----- strace.log.30962:fcntl(13, F_GETFL) = 0x1 (flags O_WRONLY) strace.log.30962:fcntl(13, F_SETFL, O_WRONLY|O_NONBLOCK) = 0 strace.log.30962:write(13, "POLLERR occurred while idle; fd="..., 168) = 168 strace.log.30962:close(13) = 0 I mussed some pid's around to try and format the email as best I could. Can anyone suggest more debugging steps, or what might be causing this? I haven't seen this error elsewhere, the search results are barren, except the for commit logs. I'm on gentoo using 3.1.1: # syslog-ng --version syslog-ng 3.1.1 Installer-Version: 3.1.1 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#8747d74491eac3fdf5052194f47a68e659633ca9 Compile-Date: Jun 17 2010 13:37:53 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on -tom

3 2

Syslog-ng writing to files once per minute
by Eric Cables 02 Jul '10

02 Jul '10

I'm hoping this is a really stupid question, but after implementing what I thought to be the fix to this, I can't seem to get it working. I have a number of hosts logging to facility local5, and the destination log file is only updated once per minute (HH:MM.30s to be precise). I've implemented flush_lines(0); and flush_interval(0);, to the global options, but it doesn't seem to have an effect. syslog-ng 3.0.3 My goal is to have logs written to the destination file with minimal latency after syslog-ng receives the message. -- Eric Cables

2 6

[Bug 88] New: rewrite FACILITY does not work.
by bugzilla＠bugzilla.balabit.com 01 Jul '10

01 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=88 Summary: rewrite FACILITY does not work. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Windows Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: alex-pub.balabit(a)reflexion.net Type of the Report: --- Estimated Hours: 0.0 Here is the fragment of my syslog-ng.conf file that demonstrates the problem source remote_sys { tcp( ip( "0.0.0.0" ) port( 514 ) max-connections( 1152 ) ); udp( ip( "0.0.0.0" ) port( 514 ) ); }; destination d_remote_consolidated_log { file( "/var/log/remote/$FACILITY/$FACILITY.log" create_dirs( yes ) perm( 0644 ) dir_perm( 0755 ) template( "$R_ISODATE $HOST -$LEVEL- $PROGRAM[$PID]: $MSGONLY\n" ) frac_digits( 6 ) template_escape( no ) ); }; filter f_apache { facility( local1 ); }; rewrite r_rewrite_apache_facility { set("apache" value("FACILITY")); }; log { source( remote_sys ); filter( f_apache ); rewrite( r_rewrite_apache_facility ); destination( d_remote_consolidated_log ); flags( final ); }; If I issue the following command from the host connected to this syslog-ng server: $ echo this is apache log test | logger -t apache-test -p local1.info Then the output is sent to "/var/log/remote/local1/local1.log" instead of expected "/var/log/remote/apache/apache.log". Also, if I include $FACILITY into the destination template it prints "local1" If I run syslog-ng with -d flag I see this in output: >>Rewrite expression evaluation result; value='FACILITY', new_value='apache' Below is a full log for this test >>Incoming log entry; line='<142>Jun 22 17:36:11 host-1 apache-test: this is a test 2' >>Filter rule evaluation begins; filter_rule='f_filter1' >>Filter node evaluation result; filter_result='not-match', filter_type='facility' >>Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter1' >>Filter rule evaluation begins; filter_rule='f_filter2' >>Filter node evaluation result; filter_result='not-match', filter_type='facility' >>Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter2' >>Filter rule evaluation begins; filter_rule='f_apache' >>Filter node evaluation result; filter_result='match', filter_type='facility' >>Filter rule evaluation result; filter_result='match', filter_rule='f_apache' >>Rewrite expression evaluation result; value='FACILITY', new_value='apache' So this makes me believe that rewrite does happen, but the value is not picked up later. BTW, the documentation for `set` command says that there should be a comma between "apache" and value, but other samples do not put it there. I've tried it both ways to no avail. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

spoof-source performance issues
by Martin Holste 01 Jul '10

01 Jul '10

I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?

5 12

[Bug 91] New: Cisco ASA log parsing issues
by bugzilla＠bugzilla.balabit.com 01 Jul '10

01 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=91 Summary: Cisco ASA log parsing issues Product: syslog-ng Version: 3.1.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jon(a)jonjames.net Type of the Report: bug Estimated Hours: 0.0 Orignal LOG as seen my TCPDUMP 16:42:08.734684 IP 172.16.32.157.syslog > 172.16.32.172.syslog: SYSLOG local4.notice, length: 95 E..{.......^.. ... ......g<.<165>Jul 01 2010 00:32:59: %ASA-5-111001: Begin configuration: 172.16.16.154 writing to memory After being relayed as seen my TCPDUMP 16:42:59.208826 IP 172.16.32.172.33753 > 172.16.32.225.syslog: SYSLOG local4.notice, length: 120 E...a.@.@.?... ... ........?<165>1 2010-07-01T00:33:50-04:00 172.16.32.157 %ASA-5-111001 - - - Begin configuration: 172.16.16.154 writing to memory As you can see, syslog-ng appears to be messing with the datestamp and therefore the RSA envision device I'm relaying to does not recognize the log as being from a cisco asa. Is there a way to update the log parser in syslog to resolve this issue ? syslog-ng.conf (parts) @version: 3.0 options { time_sleep(30); time_reap(30); mark_freq(30); dns_cache(yes); use_fqdn(no); keep_hostname(yes); chain_hostnames(no); use_dns(no); dns_cache_size(250000); dns_cache_expire(300); dns_cache_expire_failed(300); stats_freq(3600); log_msg_size(10000); }; source s_extranet { udp(ip("172.16.32.172") port(514)); }; log { source(s_extranet); destination(d_syslog_udp); }; destination d_syslog_udp { syslog("172.16.32.225" transport("udp") port(514)); }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 3

Memory leak can occur with supressed duplicate log messages
by andy kitchingman 30 Jun '10

30 Jun '10

Hi Due to my company firewall, I do not have access to the git public repository, so I will post this here. I believe there is a memory leak when duplicate messages are suppressed and messages cannot be consumed: logwriter.c: log_writer_last_msg_flush(LogWriter *self) ------------------------------------------- log_queue_push_tail(self->queue, m, &path_options) can return FALSE (message cannot be consumed), in which case the local LogMessage m is leaked. The fix should be something like if (!log_queue_push_tail(self->queue, m, &path_options)) { log_msg_unref(m); } Regards Andy Kitchingman

2 1