- syslog-ng - lists.balabit.hu

Space in tag causes parsing problems - syslog-ng 2.0.8
by Fegan, Joe 21 May '09

21 May '09

Syslog-ng 2.0.8 I found that logging a message with a space in the tag causes the first word of the tag to be used as the hostname. This does not happen with the standard syslogd. Best illustrated by an example: [root@kudos8 ~]# logger -t "hello" "here is a message" [root@kudos8 ~]# tail -4 /var/log/messages May 18 16:43:53 kudos8 info snmpd[5296]: Connection from UDP: [127.0.0.1]:41757 May 18 16:43:53 kudos8 info snmpd[5296]: Connection from UDP: [127.0.0.1]:44965 May 18 16:43:53 kudos8 info snmpd[5296]: Received SNMP packet(s) from UDP: [127.0.0.1]:44965 May 18 16:43:55 kudos8 notice hello: this is a message [root@kudos8 ~]# This is expected behaviour. Now see what happens when I put a space in the tag: [root@kudos8 ~]# logger -t "hello world" "second message" [root@kudos8 ~]# tail -5 /var/log/messages May 18 16:43:53 kudos8 info snmpd[5296]: Connection from UDP: [127.0.0.1]:41757 May 18 16:43:53 kudos8 info snmpd[5296]: Connection from UDP: [127.0.0.1]:44965 May 18 16:43:53 kudos8 info snmpd[5296]: Received SNMP packet(s) from UDP: [127.0.0.1]:44965 May 18 16:43:55 kudos8 notice hello: this is a message May 18 16:44:04 hello notice world: second message [root@kudos8 ~]# See that "hello" was incorrectly used as the hostname. This is 100% reproducible. I stopped syslog-ng and started the standard syslogd instead. Standard syslogd does not have this problem: [root@kudos8 ~]# logger -t "hello" "here is a message" [root@kudos8 ~]# logger -t "hello world" "second message" [root@kudos8 ~]# service syslog-ng stop Stopping syslog-ng: [ OK ] [root@kudos8 ~]# service syslog start Starting system logger: [ OK ] Starting kernel logger: [ OK ] [root@kudos8 ~]# logger -t "hello world" "third message" [root@kudos8 ~]# tail -10 /var/log/messages May 18 16:54:25 kudos8 info snmpd[5296]: Connection from UDP: [127.0.0.1]:33587 May 18 16:54:25 kudos8 info snmpd[5296]: Connection from UDP: [127.0.0.1]:46177 May 18 16:54:25 kudos8 info snmpd[5296]: Received SNMP packet(s) from UDP: [127.0.0.1]:46177 May 18 16:54:30 kudos8 notice hello: here is a message May 18 16:54:31 hello notice world: second message May 18 16:54:34 kudos8 info syslog-ng[8737]: Termination requested via signal, terminating; May 18 16:54:34 kudos8 notice syslog-ng[8737]: syslog-ng shutting down; version=\'2.0.8\' May 18 16:54:38 kudos8 syslogd 1.4.1: restart. May 18 16:54:38 kudos8 kernel: klogd 1.4.1, log source = /proc/kmsg started. May 18 16:54:49 kudos8 hello world: third message [root@kudos8 ~]#

2 3

parsing netapp syslog (2)
by David - DCPC 21 May '09

21 May '09

Hello, I'm new on syslog-ng. I use v3.0.2 on a CentOS 4.4. I send syslog logs from a netapp FAS 2020 (*.*) to the syslog-ng server via udp. I have problem to analyse thoses logs. the lines have a different format than the standard syslog from another unix by example and are different than the BSD or IETF standard described in the syslog-ng documentation : (...) Tue May 19 11:27:30 CEST [netapp-L: wafl.quota.qtree.exceeded:notice]: tid 10: tree quota exceeded on volume linusers. Additional warnings will be suppressed for approximately 60 minutes or until a 'quota resize' is performed. (...) 1°) When i use it in syslog-ng i get this line in a sample extraction without filter : (...) May 19 11:27:30 netapp-b wafl.quota.qtree.exceeded:notice]: tid 10: tree quota exceeded on volume linusers. Additional warnings will be suppressed for approximately 60 minutes or until a 'quota resize' is performed. (...) (difference between host named should come from the renaming of hosts during tests, and dns aliases, don't think this is the problem). Is it possible to correct the processus that parse the original line ? it seems that the parser block can be used only for the $MESSAGE part of the line. 2°) It want to use other macro to make some filter and destination, like $PROGRAM. Is it possible to 'debug' the processing and have a details of macros and their contents, line by line ? Thx (sorry for the first miss) -- Salutations, David CHALON

2 1

NULL byte and Syslog-NG
by Jon Fredrickson 21 May '09

21 May '09

Hello, I am importing events from the Oracle sys.aud$ table via JDBC, then forwarding these events to a syslog-ng server. Every time a user makes an ODBC connection from a windows machine, a NULL byte appears directly after the host name in the USERHOST field. When syslog-ng gets this raw event it changes the NULL byte from 00 to a line feed 0a. Does anyone know how to stop this from happening, or allow syslog-ng to trim NULL characters? My syslog-ng version is 2.0.1 Thanks in advance, Jon

2 1

parsing netapp syslog
by David - DCPC 19 May '09

19 May '09

Hello, I'm new on syslog-ng. I use v3.0.2 on a CentOS 4.4. I send syslog logs from a netapp FAS 2020 (*.*) to the syslog-ng server via udp. I have problem to analyse thoses logs. the lines have a different format than the standard syslog from another unix by example. and different tha -- Salutations, David CHALON

1 0

Logging with db-parser issue
by Nate Hausrath 14 May '09

14 May '09

Hey everyone, Sorry to ask what is probably an easy question. I'm attempting to use the new db-parser feature in syslog-ng to parse Windows events generated by Snare. What I've attempted to do is write a simple XML file that will match one simple rule. This way I can test that everything is functioning properly and I can move on from there. Unfortunately, it's not working, so I'm unsure as to whether it is my rule or my installation. So I have a few basic questions that will hopefully clear up my confusion! - I did not install syslog-ng from source. I downloaded the .deb for version 3.01 and used that. Does this even include db-parser? (I'm assuming yes because it does not cause an error from the config file alone.) - My installation is in /opt/syslog-ng, so I've put the example XML files (http://www.balabit.com/downloads/files/patterndb/) in /opt/syslog-ng/var. Is this the right location? Other than that, here is my syslog-ng.conf file: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); }; source s_net { udp(); tcp(); }; parser p_db { db-parser(); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_parsed { file("/var/log/remote/parsed.log"); }; log { source(s_local); destination(d_messages); }; log { source(s_net); destination(d_parsed); parser(p_db); }; And here is my windows.xml file: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-05-07'> <ruleset name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <patterns> <pattern>540</pattern> </patterns> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </ruleset> </patterndb> As you see, to test I just want to match on the number '540' and put that log message in /var/log/remote/parsed.log. I have verified that messages that should match this are arriving at the machine when I send them. If i turn off the db-parser, the messages appear in the parsed.log file. If there is any documentation that explains the basics of this, sorry that I missed it. I've only been able to find some of the other messages on this mailing list and the blog entries on balabit. Thanks for any help! Nate

4 10

Compile error on Ubuntu 8.04
by Nate Hausrath 12 May '09

12 May '09

Hello everyone, I'm attempting to compile and install the latest syslog-ng 3.0 from source (downloaded from the git repository). I believe I have installed all the dependencies (configure does not complain). However, I am receiving the following error when running make. /bin/bash ../ylwrap cfg-lex.l .c cfg-lex.c -- /bin/bash /home/nhausrath/packages/syslog-ng.git/missing --run flex make[2]: *** [cfg-lex.c] Error 1 make[2]: Leaving directory `/home/nhausrath/packages/syslog-ng.git/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/nhausrath/packages/syslog-ng.git' make: *** [all] Error 2 It seems the error is in cfg-lex.l, but I can't tell what exactly is causing the problem. Of course, it may be something completely different that I am not familiar with. Thanks for any help! -Nate

3 4

Re: [syslog-ng] PATCH: Fix some small memory leaks and a SEGV
by mark tomlinson 11 May '09

11 May '09

On Fri, 2009-05-08 at 15:16 +0200, Balazs Scheidler wrote: > Hi, > > Thanks for your contribution. Although some of these might be genuine > leaks, they mostly occur of you use the same option multiple times in > the configuration file. > ... deletions ... > > > > diff --git a/src/dgroup.c b/src/dgroup.c > > index 7919a54..be02899 100644 > > --- a/src/dgroup.c > > +++ b/src/dgroup.c > > @@ -76,6 +76,16 @@ log_dest_group_deinit(LogPipe *s) > > NULL); > > success = FALSE; > > } > > + if (p->group) > > + { > > + g_free(p->group); > > + p->group = NULL; > > + } > > + if (p->id) > > + { > > + g_free(p->id); > > + p->id = NULL; > > + } > > } > > return success; > > } > > these should be freed when the driver in question gets freed in the > log_drv_free() function > This was a definite leak with our configuration and use. I believed the problem to be due to log_dest_group_deinit() being called, and then log_dest_group_init() being called, without log_drv_free() ever being called inbetween. I will have a look at our configuration files. We do change configuration between logging to a file and logging to/from network on the fly (and while many log messages are being generated). > > diff --git a/src/logsource.c b/src/logsource.c > > index 311e61f..93a8747 100644 > > --- a/src/logsource.c > > +++ b/src/logsource.c > > @@ -181,6 +181,10 @@ log_source_set_options(LogSource *self, LogSourceOptions *options, gint stats_le > > g_atomic_counter_set(&self->options->window_size, current_window); > > self->stats_level = stats_level; > > self->stats_source = stats_source; > > + if (self->stats_id) > > + g_free(self->stats_id); > > + if (self->stats_instance) > > + g_free(self->stats_instance); > > self->stats_id = stats_id ? g_strdup(stats_id) : NULL; > > self->stats_instance = stats_instance ? g_strdup(stats_instance): NULL; > > } > > > > > diff --git a/src/logwriter.c b/src/logwriter.c > > index 4c895cc..1d1e00e 100644 > > --- a/src/logwriter.c > > +++ b/src/logwriter.c > > @@ -792,6 +792,10 @@ log_writer_set_options(LogWriter *self, LogPipe *control, LogWriterOptions *opti > > > > self->stats_level = stats_level; > > self->stats_source = stats_source; > > + if (self->stats_id) > > + g_free(self->stats_id); > > + if (self->stats_instance) > > + g_free(self->stats_instance); > > self->stats_id = stats_id ? g_strdup(stats_id) : NULL; > > self->stats_instance = stats_instance ? g_strdup(stats_instance) : NULL; > > these again should only be called once, but better if the function is > idempotent. Again, I was seeing deinit()/init() happening, without any calls to free the driver from af_socket_dd. > > > > > diff --git a/src/sgroup.c b/src/sgroup.c > > index 904237c..0fdf8fb 100644 > > --- a/src/sgroup.c > > +++ b/src/sgroup.c > > @@ -79,7 +79,18 @@ log_source_group_deinit(LogPipe *s) > > evt_tag_str("id", p->id), > > NULL); > > success = FALSE; > > - } > > + } > > + if (p->group) > > + { > > + g_free(p->group); > > + p->group = NULL; > > + } > > + if (p->id) > > + { > > + g_free(p->id); > > + p->id = NULL; > > + } > > + log_pipe_append(&p->super, NULL); > > > hmm... can you show me the backtrace of the crash that this is supposed > to fix? when the configuration is deinitialized the old LogSource > instances should never be referenced again, thus this dangling pointer > will never be referenced again. > > Also, group/id is freed whenever the driver instance is freed, thus no > need to open code it. I see the possibility of a leak if syslog-ng falls > back to the old configuration as we coulnd't initialize the new one (in > which case log_sgroup_init is called multiple times on the same config), > but otherwise it should not be a leak. > The memory leaks again occured due to seeing deinit/init pairs being called. The fix for the crash is the log_pipe_append(), which prevents the driver using the sgroup again after it has been freed. I don't know the full reason for this happening, only that it did (with HUPs, changing configs, and lots of log messages all at the same time...) The stack trace was as follows: Thread Backtraces: Thread 1 (process 862): #0 log_source_group_queue (s=0x100795d8, msg=0x100aa598, path_options=0x7fa70ba4) at sgroup.c:98 #1 0x1000a238 in log_pipe_forward_msg (self=<value optimized out>, msg=0x1, path_options=0x3025134c) at logpipe.h:121 #2 0x1000a238 in log_pipe_forward_msg (self=<value optimized out>, msg=0x1, path_options=0x3025134c) at logpipe.h:121 #3 0x10026148 in log_source_queue (s=0x100bacf0, msg=0x100aa598, path_options=<value optimized out>) at logpipe.h:121 #4 0x1001b43c in log_reader_fd_dispatch (source=<value optimized out>, callback=<value optimized out>, user_data=<value optimized out>) at logpipe.h:121 #5 0x30070f9c in IA__g_main_context_dispatch (context=0x100507f0) at gmain.c:1916 #6 0x300732c4 in g_main_context_iterate (context=0x0, block=0, dispatch=1, self=<value optimized out>) at gmain.c:2547 #7 0x300733c8 in IA__g_main_context_iteration (context=0x0, may_block=268765168) at gmain.c:2606 #8 0x10005324 in main_loop_run (cfg=0x0) at main.c:175 #9 0x1000584c in main (argc=1, argv=0x7fa70de4) at main.c:428 The point it crashed at (sgroup.c:98) was the line: (*self->processed_messages)++; in log_source_group_queue(). But self has already been freed, so the memory address it was incrementing was bogus, causing the SEGV. The fix above means that the log_pipe_forward_msg() call will drop the message in this case.

2 1

syslog-ng 3.0.2 has been released
by devel＠balabit.hu 08 May '09

08 May '09

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.0.2 SUMMARY : new stable release DATE : May 8, 2009 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.0.2) has been released. For latest fixes in the 3.0.x branch you are recommended to upgrade to this version. CHANGES: 3.0.2 Thu, 07 May 2009 08:30:06 +0200 Highlights: * This release is the first to feature binary packages, installation packages for various platforms. Bugfixes: * Only assume that a source logfile was rotated if the new log file has a non-zero size. This fix solves a minor race which might cause some messages to be lost from the end of the old logfile. * Fixed the processing of the -u and -g command line options to set the user/group of the syslog-ng process. Previously all usernames were rejected as non-existant. * Resolve usernames before chrooting, to avoid the need to have an /etc/passwd & /etc/group inside the chroot. * Fixed the read rate of regular files, with the change in this release it is now possible to fully saturate a destination pipe by using plain text files. * Fixed a synchronization problem in the SQL destination on AIX causing random crashes when the SQL destination is used. * Make the "syslog" reserved word usable as a facility name, previously syslog-ng indicated a syntax error if syslog as a facility was not enclosed in quotation marks. * Don't write the pidfile in case only a syntax check was requested. * Fixed compilation problems if SSL support is not enabled. * Set the default fd limit to 256 on cygwin as that platform does not support the current default of 4096 used on all other platforms. * Fixed template expansion for non-inline templates defined using the template statement. * Fixed the handling of zero-sized packets in the udp() source. * Fixed the storing of regexp/db-parser() matches on big-endian, 32 bit CPUs. * Fixed character conversion in case one chunk of data has an incomplete sequence at the end. * Don't let the character conversion eat up unlimited memory to avoid an easy DoS condition. * Fixed SSL certificate validation, added X.509 policy check, fixed required-untrusted verify-mode and added basicConstraints validation. * Fixed /dev/klog handling on FreeBSD, syslog-ng attempted to open this device multiple times which is not permitted. * Fixed $TZOFS expansion in templates in case the target does not specify a conversion timezone (which is the default and is true in most cases). * Fixed the use of multiple SQL destinations, previously due to an initialization problem only the first SQL destination would ever start writing messages to the target database. * If you elected to use a host-file only name resolution, the check to see if the hosts file was changed was performed for each DNS resolution, causing a major performance drop. This was fixed. * Due to a typo in the configuration file parser, using SSL destinations always resulted in a segmentation fault. This was fixed. * Fixed counting the number of concurrent connections in the SSL enabled tcp() source, previously failing handshakes were counted, thus it is quite easy to reach the max-connection limit. * Clarified logging about connection rejected by /etc/hosts.deny & /etc/hosts.allow * HP-UX 11v3 compilation fixes * Fixed SOURCEIP expansion for IPv6 addresses. * Fixed match(type(string)) not to match the message in case only a prefix match was found. * Fixed capability setup code around file opens. Other changes: * Implemented 'store-legacy-msghdr' source flag which keeps the original value of MSGHDR intact, instead of parsing/rebuilding it at the cost of some performance penalty. * Implemented 'empty-lines' source flag to avoid dropping empty log messages which is the default behaviour. * Added support for file includes in the configuration file. * Allow follow_freq() to be a floating point number. * Cygwin packaging fixes. * Make it possible to create named pipes automatically in the pipe() destination. * Allow the use of macros in the filename of pipe() destinations. * Added support for setting the default priority/facility value for each source that get applied to messages that lack one. * Recognize "mssql" as an alias for "freetds" in the SQL destination. * Added the "greedy" flags to the csv-parser(): it makes the last column eat the complete end of the line without caring about column separation characters * Added the "drop-invalid" flag to the csv-parser() which makes the parser drop anything that has less columns than it expects to have. * Make it possible to use fractions of a second even in non-ISO timestamps. * The default values for SQL columns is being deprecated, please supply SQL schema information explicitly in your SQL destinations. A warning message was added in case you don't have a full schema and rely on the default values. * The SQL destination got a null() option that specifies a 'special' value for a column which get replaced by an SQL NULL. * Added local_timezone() option to file/sql destinations that control the timezone used when expanding the filename/tablename templates. It defaults to the local timezone. Configure changes: * This section lists changes that might affect distribution packages, please make sure that these changes do not break the syslog-ng packages in any way. * Changed the default value of pidfiledir to ${localstatedir} to match GNU conventions, you can specify it explicitly with a --with-pidfile-dir configure option. * Changed the default linking mode to "dynamic" to make it easier to compile syslog-ng, as most distributions prefer dynamic libraries over static ones. This means that syslog-ng might not be able to start up if /usr is on NFS. On platforms where syslog-ng is used as a system logger, the previous setting of --enable-mixed-linking is preferred. Loggen changes: * the sequence number and timestamping of generated messages were erroneously commented out, these were reverted. * do not allow message sizes that are larger than the predefined message buffer size. * in addition to reporting the number of messages/sec, also report the effective bandwidth used during testing. Test suite changes: * A lot of enhancements to the functional test program to cover more of the syslog-ng code, like SSL sources & SQL destinations. DOWNLOAD: You can download the latest binaries from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The latest versions of syslog-ng in source format can be found at: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/

2 1

Question on '$PRIORITY' and '$LEVEL'
by Arnold Wang 08 May '09

08 May '09

I have a question regarding how syslog-ng treats "priority" and "level" value from the logs. According to RFC, "priority" is different than "level". However, it appears to me that syslog-ng use "level" value for "priority" as well. My questions are: 1. Is that true that syslog-ng use "level" value for "priority" as well? 2. Is there a reason behind this behavior? 3. Is there a plan to change/fix this? Thanks.

2 2

PATCH: Fix some small memory leaks and a SEGV
by mark tomlinson 08 May '09

08 May '09

Fix various memory leaks caused by strings not being freed. Also, in log_source_group_deinit(), ensure no drivers are still pointing to this possibly soon-to-be-freed structure. Without this a SIGSEGV was possible when a SIGHUP occured while syslog was busy. diff --git a/src/afinet.c b/src/afinet.c index 28b9d50..144a08a 100644 --- a/src/afinet.c +++ b/src/afinet.c @@ -191,6 +191,8 @@ afinet_sd_set_transport(LogDriver *s, const gchar *transport) { AFInetSourceDriver *self = (AFInetSourceDriver *) s; + if (self->super.transport) + g_free(self->super.transport); self->super.transport = g_strdup(transport); if (strcasecmp(transport, "udp") == 0) { diff --git a/src/afstreams.c b/src/afstreams.c index df4e758..35d4812 100644 --- a/src/afstreams.c +++ b/src/afstreams.c @@ -104,6 +104,8 @@ afstreams_sd_set_sundoor(LogDriver *s, gchar *filename) { AFStreamsSourceDriver *self = (AFStreamsSourceDriver *) s; + if (self->door_filename) + g_string_free(self->door_filename, TRUE); self->door_filename = g_string_new(filename); } diff --git a/src/dgroup.c b/src/dgroup.c index 7919a54..be02899 100644 --- a/src/dgroup.c +++ b/src/dgroup.c @@ -76,6 +76,16 @@ log_dest_group_deinit(LogPipe *s) NULL); success = FALSE; } + if (p->group) + { + g_free(p->group); + p->group = NULL; + } + if (p->id) + { + g_free(p->id); + p->id = NULL; + } } return success; } diff --git a/src/logsource.c b/src/logsource.c index 311e61f..93a8747 100644 --- a/src/logsource.c +++ b/src/logsource.c @@ -181,6 +181,10 @@ log_source_set_options(LogSource *self, LogSourceOptions *options, gint stats_le g_atomic_counter_set(&self->options->window_size, current_window); self->stats_level = stats_level; self->stats_source = stats_source; + if (self->stats_id) + g_free(self->stats_id); + if (self->stats_instance) + g_free(self->stats_instance); self->stats_id = stats_id ? g_strdup(stats_id) : NULL; self->stats_instance = stats_instance ? g_strdup(stats_instance): NULL; } diff --git a/src/logwriter.c b/src/logwriter.c index 4c895cc..1d1e00e 100644 --- a/src/logwriter.c +++ b/src/logwriter.c @@ -792,6 +792,10 @@ log_writer_set_options(LogWriter *self, LogPipe *control, LogWriterOptions *opti self->stats_level = stats_level; self->stats_source = stats_source; + if (self->stats_id) + g_free(self->stats_id); + if (self->stats_instance) + g_free(self->stats_instance); self->stats_id = stats_id ? g_strdup(stats_id) : NULL; self->stats_instance = stats_instance ? g_strdup(stats_instance) : NULL; diff --git a/src/sgroup.c b/src/sgroup.c index 904237c..0fdf8fb 100644 --- a/src/sgroup.c +++ b/src/sgroup.c @@ -79,7 +79,18 @@ log_source_group_deinit(LogPipe *s) evt_tag_str("id", p->id), NULL); success = FALSE; - } + } + if (p->group) + { + g_free(p->group); + p->group = NULL; + } + if (p->id) + { + g_free(p->id); + p->id = NULL; + } + log_pipe_append(&p->super, NULL); } return success; }

3 2