June 2013 - syslog-ng - lists.balabit.hu

[Bug 238] New: syslog-protocol: allow NILVALUE for TIMESTAMP as specified in rfc5424
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=238 Summary: syslog-protocol: allow NILVALUE for TIMESTAMP as specified in rfc5424 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: spam(a)lerya.net Type of the Report: --- Estimated Hours: 0.0 RFC5424 states: - section 6: """ TIMESTAMP = NILVALUE / FULL-DATE "T" FULL-TIME """ - section 6.2.3 "" A syslog application MUST use the NILVALUE as TIMESTAMP if the syslog application is incapable of obtaining system time. "" But the 3.4 version of syslog-ng does not allow such a NILVALUE: https://github.com/balabit/syslog-ng-3.4/blob/master/modules/syslogformat/s… For example, the following message does not work: <6>1 - - netlog - - - [ 775.895618] /bin/ping[2936] UDP 10.0.2.15:33285 -> 192.168.0.1:53 (uid=0) Whereas this one works: <6>1 2003-10-11T22:14:15.003Z - netlog - - - [ 775.895618] /bin/ping[2936] UDP 10.0.2.15:33285 -> 192.168.0.1:53 (uid=0) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 13

IPv6 TLS source question
by Jeff Lange 05 Jul '13

05 Jul '13

I'm attempting to setup a TCP IPv6 source declaration that uses TLS but syslog-ng is not happy with the configuration file. The following is what I use for IPv4 and it works fine: source s_tls { syslog( ip(0.0.0.0) port(6514) transport( "tls" ) tls ( key_file("/srv/syslog-ng/certs/priv-key.pem") cert_file("/srv/syslog-ng/certs/id-cert.pem") ca_dir("/srv/syslog-ng/certs") cipher_suite("SHA256:!aNULL:!eNULL:!ECDH:!DSS") ) ); }; However the following is not valid: source s_tls6 { syslog( ip("::") port(6515) transport( "tls" ) tls ( key_file("/srv/syslog-ng/certs/priv-key.pem") cert_file("/srv/syslog-ng/certs/id-cert.pem") ca_dir("/srv/syslog-ng/certs") cipher_suite("SHA256:!aNULL:!eNULL:!ECDH:!DSS") ) ); }; I've tried the ip with and without quotes with no luck. What am I doing wrong? Thanks -Jeff

3 4

Upcoming changes regarding my Debian/Ubuntu packages
by Gergely Nagy 04 Jul '13

04 Jul '13

Hi! For a good while now, I've been providing Debian and Ubuntu packages[1] for syslog-ng, and rest assured, I will continue to do so. However, the way they are prepared and how the repository is structured, will change. At the moment, there are two different components one can choose from: syslog-ng and syslog-ng-devel, which have packages for 3.3 and 3.4, respectively. However, -devel is not appropriate for 3.4 anymore, but I don't want to move it to syslog-ng either, partly because up to this point, there were no Ubuntu Lucid packages of 3.4. So, I will be making the following changes: * I'll create syslog-ng-3.3, syslog-ng-3.4 and syslog-ng-3.5 components * syslog-ng will point to the latest stable branch (3.4) * syslog-ng will point to the current development branch (3.5) * I'll be preparing Ubuntu Lucid packages for 3.4 too, so all branches will be available for every debian or ubuntu system that are still supported by their respective vendors. To be able to provide Ubuntu Lucid (10.04) packages sanely, I have to do the packaging a little bit differently: the source package will not be shared between debian and ubuntu anymore, and systemd support will be explicitly disabled on Ubuntu. But since Ubuntu uses upstart by default, ubuntu systems with systemd are fairly unlikely anyway. I plan to make these changes once 3.4.2 comes out. Until then, I'll introduce the syslog-ng-3.3 and -3.4 components, but syslog-ng will point to -3.3 still, and -devel to -3.4. Once 3.4.2 is out, though, I'll change the symlinks. [1]: http://asylum.madhouse-project.org/projects/debian/ -- |8]

1 2

[PATCH (3.5)] affile: Error out when there's a syntax error in the filename template
by Gergely Nagy 04 Jul '13

04 Jul '13

When there is a syntax error in the filename template (eg, "/var/log/foo-$(+ 0"), error out at config load time. This is done by adding an error output variable to affile_dd_new_instance() and both affile_dd_new() and afpipe_dd_new(), then using that to store the result of log_template_compile(). If that function fails, we bail out early, and use CHECK_ERROR() in the grammar to notify the user of the issue. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/affile/affile-dest.c | 23 +++++++++++++++-------- modules/affile/affile-dest.h | 6 +++--- modules/affile/affile-grammar.ym | 12 +++++++++--- 3 files changed, 27 insertions(+), 14 deletions(-) diff --git a/modules/affile/affile-dest.c b/modules/affile/affile-dest.c index c8fa9c1..c9447ce 100644 --- a/modules/affile/affile-dest.c +++ b/modules/affile/affile-dest.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2002-2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2002-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 1998-2012 Balázs Scheidler * * This program is free software; you can redistribute it and/or modify it @@ -703,17 +703,24 @@ affile_dd_free(LogPipe *s) } static AFFileDestDriver * -affile_dd_new_instance(gchar *filename) +affile_dd_new_instance(gchar *filename, GError **error) { AFFileDestDriver *self = g_new0(AFFileDestDriver, 1); + LogTemplate *filename_template; + + filename_template = log_template_new(configuration, NULL); + if (!log_template_compile(filename_template, filename, error)) + { + log_template_unref(filename_template); + return NULL; + } log_dest_driver_init_instance(&self->super); self->super.super.super.init = affile_dd_init; self->super.super.super.deinit = affile_dd_deinit; self->super.super.super.queue = affile_dd_queue; self->super.super.super.free_fn = affile_dd_free; - self->filename_template = log_template_new(configuration, NULL); - log_template_compile(self->filename_template, filename, NULL); + self->filename_template = filename_template; log_writer_options_defaults(&self->writer_options); file_perm_options_defaults(&self->file_perm_options); self->writer_options.mark_mode = MM_NONE; @@ -728,15 +735,15 @@ affile_dd_new_instance(gchar *filename) } LogDriver * -affile_dd_new(gchar *filename) +affile_dd_new(gchar *filename, GError **error) { - return &affile_dd_new_instance(filename)->super.super; + return &affile_dd_new_instance(filename, error)->super.super; } LogDriver * -afpipe_dd_new(gchar *filename) +afpipe_dd_new(gchar *filename, GError **error) { - AFFileDestDriver *self = affile_dd_new_instance(filename); + AFFileDestDriver *self = affile_dd_new_instance(filename, error); self->is_pipe = TRUE; return &self->super.super; } diff --git a/modules/affile/affile-dest.h b/modules/affile/affile-dest.h index 3f4f21b..dcee654 100644 --- a/modules/affile/affile-dest.h +++ b/modules/affile/affile-dest.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2002-2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2002-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 1998-2012 Balázs Scheidler * * This program is free software; you can redistribute it and/or modify it @@ -53,8 +53,8 @@ typedef struct _AFFileDestDriver gint time_reap; } AFFileDestDriver; -LogDriver *affile_dd_new(gchar *filename); -LogDriver *afpipe_dd_new(gchar *filename); +LogDriver *affile_dd_new(gchar *filename, GError **error); +LogDriver *afpipe_dd_new(gchar *filename, GError **error); void affile_dd_set_create_dirs(LogDriver *s, gboolean create_dirs); void affile_dd_set_fsync(LogDriver *s, gboolean enable); diff --git a/modules/affile/affile-grammar.ym b/modules/affile/affile-grammar.ym index abecee4..0d2f099 100644 --- a/modules/affile/affile-grammar.ym +++ b/modules/affile/affile-grammar.ym @@ -1,5 +1,5 @@ /* - * Copyright (c) 2002-2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2002-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 1998-2012 Balázs Scheidler * * This program is free software; you can redistribute it and/or modify it @@ -144,7 +144,10 @@ dest_affile dest_affile_params : string { - last_driver = *instance = affile_dd_new($1); + GError *error; + + last_driver = *instance = affile_dd_new($1, &error); + CHECK_ERROR(last_driver, @1, "Error compiling template; %s", error->message); free($1); last_writer_options = &((AFFileDestDriver *) last_driver)->writer_options; last_file_perm_options = &((AFFileDestDriver *) last_driver)->file_perm_options; @@ -171,7 +174,10 @@ dest_affile_option dest_afpipe_params : string { - last_driver = *instance = afpipe_dd_new($1); + GError *error; + + last_driver = *instance = afpipe_dd_new($1, &error); + CHECK_ERROR(last_driver, @1, "Error compiling template; %s", error->message); free($1); last_writer_options = &((AFFileDestDriver *) last_driver)->writer_options; last_file_perm_options = &((AFFileDestDriver *) last_driver)->file_perm_options; -- 1.7.10.4

2 2

Problems with recovering from 'disk full' I/O error
by Johnson, Chris (HP TippingPoint Roseville) 26 Jun '13

26 Jun '13

Hi all, I've come across a situation where syslog-ng (3.3.3 and 3.3.9) aborts after trying to write to a disk that has no space and then is cleaned up. I've been able to reproduce the 'error' condition with the following Set up: In the config file: ########################################################### # test log destination # filter f_test{program("LOGID_99-*" type("glob"));}; destination d_test { file("/var/testpartition/test.log" perm(0644) flags(no-multi-line)); }; log { source(s_local); filter(f_test); destination(d_test); flags(final); }; 1) Fill up the '/var/testpartition' disk: root@device:/var/testpartition# dd if=/dev/zero of=/var/testpartition/foo bs=1M dd: writing '/var/testpartition/foo': No space left on device 3531+0 records in 3529+1 records out 2) Send messages ('TEST: pre I/O error')to 'test.log' until syslog-ng complains: 2013-06-14 22:41:09.623 [device] [syslog-ng-ERROR:] "I/O error occurred while writing; fd='22', error='No space left on device (28)'" 2013-06-14 22:41:09.623 [device] [syslog-ng-NOTICE:] "Suspending write operation because of an I/O error; fd='22', time_reopen='60'" 3) Restart syslog-ng with a SIGHUP root@device:/var/testpartition# kill -s HUP $(cat /var/run/syslog-ng.pid) Program received signal SIGHUP, Hangup. 0x00007f38ee8dad63 in __epoll_wait_nocancel () from /xxx/xxx/xxx/xxx/xxx/xxx/lib/libc.so.6 (gdb) continue Continuing. 4) Send another message ('TEST: post I/O error; post SIGHUP')to syslog-ng (it complains): 2013-06-14 22:41:37.147 [device] [syslog-ng-NOTICE:] "Configuration reload request received, reloading configuration;" 2013-06-14 22:41:59.610 [device] [syslog-ng-ERROR:] "I/O error occurred while writing; fd='23', error='No space left on device (28)'" 2013-06-14 22:41:59.610 [device] [syslog-ng-NOTICE:] "Suspending write operation because of an I/O error; fd='23', time_reopen='60'" 5) Clean up '/var/testpartition' disk: root@device:/var/testpartition# rm foo 6) Send another message ('TEST: post I/O error; post SIGHUP 2'); syslog-ng aborts: Program received signal SIGABRT, Aborted. 0x00007f38ee83eda5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. in ../nptl/sysdeps/unix/sysv/linux/raise.c (gdb) bt #0 0x00007f38ee83eda5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007f38ee8402c3 in abort () at abort.c:88 #2 0x00007f38ee87af18 in __libc_message (do_abort=2, fmt=0x7f38ee927cc8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:170 #3 0x00007f38ee8804b8 in malloc_printerr (action=2, str=0x7f38ee927dd0 "double free or corruption (!prev)", ptr=<value optimized out>) at malloc.c:5891 #4 0x00007f38ee8825c6 in __libc_free (mem=0x7f38ee91e820) at malloc.c:3626 #5 0x00007f38ef7d989a in ?? () #6 0x0000000000401800 in ?? () at elf-init.c:99 #7 0x00007f38efcf0257 in log_proto_file_writer_flush (s=0x674de0) at logproto.c:306 #8 0x00007f38efcf04a8 in log_proto_file_writer_post (s=0x674de0, msg=0x6558a0 "2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n", msg_len=89, consumed=0x7fffa53e570c) at logproto.c:380 #9 0x00007f38efcfc274 in log_proto_post (s=0x674de0, msg=0x6558a0 "2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2\n", msg_len=89, consumed=0x7fffa53e570c) at logproto.h:95 #10 0x00007f38efcfc095 in log_writer_flush (self=0x655a30, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:983 #11 0x00007f38efcf9be3 in log_writer_work_perform (s=0x655a30) at logwriter.c:129 #12 0x00007f38efcf9efe in log_writer_io_flush_output (s=0x655a30) at logwriter.c:198 #13 0x00007f38efd238b5 in iv_run_tasks (st=0x6036a0) at iv_task.c:46 #14 0x00007f38efd22a86 in iv_main () at iv_main.c:266 #15 0x00007f38efcfedf9 in main_loop_run () at mainloop.c:736 #16 0x00000000004017c9 in main (argc=1, argv=0x7fffa53e5968) at main.c:263 (gdb) The last two test messages are being written (run together with the last partial message that generated the 'disk full' I/O error): root@device:/var/testpartition# tail test.log 2013-06-14T22:40:58.140+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:40:59.150+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:00.180+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:01.176+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:02.170+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:03.010+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:03.930+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:05.040+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:06.180+00:00 device LOGID_99-INFO: TEST: pre I/O error 2013-06-14T22:41:07.140+00:00 device LOGID_99-INFO: T2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2013-06-14T22:42:30.476+00:00 device LOGID_99-INFO: TEST: post I/O error; post SIGHUP 2 root@device:/var/testpartition# Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

3 3

Syslog-ng unable to configure the client for sending messages to central server
by Rakesh Rajasekharan 26 Jun '13

26 Jun '13

I am getting some errors when i try to set up my client machine and try to send the messages to a remote central server. This is the error that I get, syslog-ng service starting. Plugin module not found in 'module-path'; module-path='/opt/soe/local/syslog-ng-3.3.7/lib/syslog-ng', module='afsocket' Plugin module not found in 'module-path'; module-path='/opt/soe/local/syslog-ng-3.3.7/lib/syslog-ng', module='afsocket' Error parsing destination, destination plugin tcp not found in /opt/soe/local/syslog_ng/etc/syslog-ng.conf at line 24, column 9: tcp("10.5.68.164" port(514)); ^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng. Any help on this please...

3 2

patterndb and log analysis
by Matt Zagrabelny 26 Jun '13

26 Jun '13

Greetings syslog-ng users, I've never used patterndb, but have been aware of it for a while. I just cloned the git://git.balabit.hu/bazsi/syslog-ng-patterndb.git and it looks like the project has not seen much activity since 2010. Are people still using patterndb? Do the patterns not change much and that is the reason that the git database has not changed much? What are free software enthusiasts using for log analysis? Thanks for the answers! -mz

3 3

problems with syslog-ng host filter
by Orangepeel Beef 26 Jun '13

26 Jun '13

I've got some strange problems with trying to use the syslog-ng host filter. It seems that every logline matches the filter f_comware but none of them match f_netscaler. It makes very little sense to me. Reverse dns is working as the logs that end up in /var/log/remote have the correct hostname being logged from the $HOST template. I have a second issue that SEC does not die when syslog-ng is restarted. (I have used this setup in the past and have had no problems, but i suppose things may have changed in both syslog-ng and sec since the last time) This is driving me crazy.. please help ;) #config @version: 3.1 # # Syslog-ng configuration file, compatible with default Debian syslogd # First, set some global options. options { long_hostnames(on); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("adm"); perm(0640); dns_cache_size(2000); dns_cache_expire(21600); dir_perm(0755); dir_group("adm"); stats_freq(0); log_fifo_size(200000); create_dirs(yes); bad_hostname("^gconfd$"); chain_hostnames(no); keep_hostname(no); }; source s_remote { udp();tcp(); }; destination d_remote { file("/var/log/remote/$HOST/$YEAR-$MONTH-$DAY.log" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; destination d_netscaler { program("/usr/local/sbin/sec_netscaler" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; destination d_comware { program("/usr/local/sbin/sec_comware" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; filter f_netscaler { host("lb*ae1.mydomain.com"); }; filter f_comware { host("(as|cs|r)*ae1.mydomain.com"); }; log { source(s_remote); destination(d_remote); }; log { source(s_remote); filter(f_netscaler); destination(d_netscaler); }; log { source(s_remote); filter(f_comware); destination(d_comware); }; #debug logs [13:38:54] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:38:56] Incoming log entry; line='<190>Jun 21 20:37:54 2013 R0507S3Z3AE1 %%10MSTP/6/MSTP_FORWARDING(l): -DevIP=x.x.x.x; Instance 0\'s Ten-GigabitEthernet1/0/2 has been set to forwarding state.' [13:38:58] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:00] Filter node evaluation result; filter_result='not-match' [13:39:02] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:04] Filter rule evaluation begins; filter_rule='f_comware' [13:39:06] Filter node evaluation result; filter_result='match' [13:39:08] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:10] Incoming log entry; line='<190>Jun 21 20:37:54 2013 R0507S3Z3AE1 %%10MSTP/6/MSTP_FORWARDING(l): -DevIP=x.x.x.x; Instance 1\'s Ten-GigabitEthernet1/0/2 has been set to forwarding state.' [13:39:12] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:14] Filter node evaluation result; filter_result='not-match' [13:39:16] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:18] Filter rule evaluation begins; filter_rule='f_comware' [13:39:20] Filter node evaluation result; filter_result='match' [13:39:22] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:24] Incoming log entry; line='<134> 06/21/2013:20:37:54 GMT lb1o1ae1 0-PPE-2 : UI CMD_EXECUTED 232044114 0 : User nsroot - Remote_ip x.x.x.x - Command "login nsroot "********"" - Status "Success"' [13:39:26] Initializing destination file writer; template='/var/log/remote/$HOST/$YEAR-$MONTH-$DAY.log', filename='/var/log/remote/lb1o1ae1.mydomain.com/2013-06-21.log' [13:39:28] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:31] Filter node evaluation result; filter_result='not-match' [13:39:33] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:35] Filter rule evaluation begins; filter_rule='f_comware' [13:39:37] Filter node evaluation result; filter_result='match' [13:39:39] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:41] ^CTermination requested via signal, terminating; [13:39:43] syslog-ng shutting down; version='3.1.3' #logs in /var/log/remote/lb2z2ae1.mydomain.com Jun 21 20:23:34 lb2z2ae1.mydomain.com 20:23:34 GMT lb2z2ae1 0-PPE-3 : UI CMD_EXECUTED 28261 0 : User nsroot - Remote_ip x.x.x.x - Command "show service GL-AE1-2AZ1-DB0001_9191" - Status "Success"

2 4

gsoc 2013 - syslog-ng: redis destination
by Tihamér Petrovics 24 Jun '13

24 Jun '13

Hi, I'm Tihamér Petrovics and I will spend this summer in a project of gsoc 2013 to develop redis destination for syslog-ng. If you are curious, you can find my blog here, I will update it approximately weekly: http://tichygsoc.blogspot.com/ All the best, Tihi

1 0

I/O errors
by Russell Fulton 17 Jun '13

17 Jun '13

getting these errors at a rate of several per second but I can't figure out what is causing them: Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='758', error='Connection reset by peer (104)' Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='808', error='Connection reset by peer (104)' Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='906', we have one network destination which is a tcp connection and which appears to be fine. No errors on the other end and stats show now dropped packets. Russell

2 3