November 2011 - syslog-ng - lists.balabit.hu

[Bug 111] New: RPM build errors: Installed (but unpackaged) file(s) found:
by bugzilla＠bugzilla.balabit.com 14 Nov '11

14 Nov '11

https://bugzilla.balabit.com/show_bug.cgi?id=111 Summary: RPM build errors: Installed (but unpackaged) file(s) found: Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: packaging AssignedTo: folti(a)balabit.hu ReportedBy: owen.mann(a)interactivedata.com Type of the Report: bug Estimated Hours: 0.0 when attempting `rpmbuild -tb syslog-ng_3.2beta1.tar.gz`, the compilation completes, but: [...] Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/syslog-ng-root error: Installed (but unpackaged) file(s) found: /bin/pdbtool /bin/update-patterndb /etc/syslog-ng/modules.conf /etc/syslog-ng/scl.conf /lib/libsyslog-ng.la /lib/libsyslog-ng.so /lib/libsyslog-ng.so.0 /lib/libsyslog-ng.so.0.0.0 /lib/syslog-ng/libaffile.la /lib/syslog-ng/libaffile.so /lib/syslog-ng/libafprog.la /lib/syslog-ng/libafprog.so /lib/syslog-ng/libafsocket-notls.la /lib/syslog-ng/libafsocket-notls.so /lib/syslog-ng/libafsocket-tls.la /lib/syslog-ng/libafsocket-tls.so /lib/syslog-ng/libafsocket.so /lib/syslog-ng/libafuser.la /lib/syslog-ng/libafuser.so /lib/syslog-ng/libbasicfuncs.la /lib/syslog-ng/libbasicfuncs.so /lib/syslog-ng/libconfgen.la /lib/syslog-ng/libconfgen.so /lib/syslog-ng/libconvertfuncs.la /lib/syslog-ng/libconvertfuncs.so /lib/syslog-ng/libcsvparser.la /lib/syslog-ng/libcsvparser.so /lib/syslog-ng/libdbparser.la /lib/syslog-ng/libdbparser.so /lib/syslog-ng/libdummy.la /lib/syslog-ng/libdummy.so /lib/syslog-ng/libsyslogformat.la /lib/syslog-ng/libsyslogformat.so /sbin/syslog-ng-ctl /share/include/scl/pacct/plugin.conf /share/include/scl/syslogconf/README /share/include/scl/syslogconf/convert-syslogconf.awk /share/include/scl/syslogconf/plugin.conf /share/include/scl/system/generate-system-source.sh /share/include/scl/system/plugin.conf /share/xsd/patterndb-1.xsd /share/xsd/patterndb-2.xsd /share/xsd/patterndb-3.xsd [...] If these files are added to the %files section in syslog-ng.spec, the build completes successfully. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

syslog-ng 3.3.2 has been released
by devel＠balabit.hu 14 Nov '11

14 Nov '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.2 SUMMARY : new stable release DATE : Nov 13, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.2) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.2 Sat, 12 Nov 2011 20:48:53 +0100 Bugfixes: ========= * Fixed a segmentation fault on reload when using the same rewrite rule from multiple log paths. * Fixed a segmentation fault when processing a reload request in case an existing tcp() source is removed from the config and there are open connections. * Fixed a possible segmentation fault in the scalable queue implementation, which happens in case a destination is slower to process messages than syslog-ng would like to send them. * Fixed a possible file() destination issue that could cause syslog-ng to omit data or to write garbage to the log file in case the kernel reports that only a smaller portion of the actual write request could be accomplished. * Fixed an "internal error duplicate config element" error during reload due to an invalid bugfix applied for 3.3.1. Older beta versions of 3.3 were not affected. * Fixed a memory leak that causes macro based file destinations to leak their queue when destination files are closed due to time-reap(). * Fixed the handling of the condition() option for rewrite rules. * Fixed a race condition in value-pairs support, potentially causing heap corruption problems when $(format-json) is used in threaded mode. * Fixed a memory leak in value-pairs template function argument parsing, fixing a leak if $(format-json) is used. * Repeated definitions of source, destination, filter, rewrite, parser and block elements are not allowed by default anymore. These are reported as configuration errors unless @define allow-config-dups 1 is specified in the configuration file. * Fixed pdbtool error reporting in "pdbtool test" to make it easier to understand what went wrong. * Added an SQL connection health check in case an INSERT failed. This way syslog-ng handles SQL server timeouts better. * Fixed support for systemd socket activation. Previously such sockets were not set to non-blocking mode, causing syslog-ng to hang. * Fixed the filter() function in the filter expression to work also when used as a part of an AND or OR construct. * Allow the sql() destination to operate even without an indexes() option. That parameter was meant to be optional, but it wasn't. * Fixed compilation issues if no OpenSSL is present. * Fixed a minor memory leak in the usertty() driver that can increase memory usage on every reload. (The username() parameter wasn't properly freed on reload). * Fixed a minor memory leak in the sql() driver that can increase the memory usage on every reload (indexes() parameter wasn't properly freed on reload). Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Anthony Lineham (Allied Telesys) Arkadiusz Miśkiewicz (PLD Linux) Attila Szalay (BalaBit) Balazs Scheidler (BalaBit) Balint Kovacs (BalaBit) Daniel Ankers Dave Rawks (Pandora) Enrico Scholz (Technische Universitat Chemnitz) Eric Lindvall (Seven Scale) Erik Maciejewski Gergely Nagy (BalaBit) Hery Fanomezantsoa Jack Nagel Jakub Jankowski (SuperHost.pl) Leonid Isaev (Indiana University) Martin Holste Michael Sterrett (Gentoo) Patrick H. Peter Czanik (BalaBit) Viktor Juhasz (BalaBit) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Logs after "\n" are getting truncated
by Swati4 M/TVM/TCS 14 Nov '11

14 Nov '11

Hi All, When logging multiline messages (simple strings with in-line newlines "\n") our logs only show the first line (ie) before "\n" is encountered and rest of the line after the first newline character("\n") is truncated. Example: Code Snippet: HS_MAP_INFO_LOG("[ %s ] Resource [ %d ] Sensor Number: %2d, Sensor Type: %d, Event Category: %d, Assertion: %d, \n" "Event State: %d, Previous State: %d, OEM Code: %d, Sensor Specific: %d\n", eidStr, pEvt->Source, thisSensorEvent->SensorNum, thisSensorEvent->SensorType, thisSensorEvent->EventCategory, thisSensorEvent->Assertion, thisSensorEvent->EventState, thisSensorEvent->PreviousState, thisSensorEvent->Oem, thisSensorEvent->SensorSpecific); Expected Output on Log: May 4 11:32:34 base0-0-0-1-0-11-1 shelfHs[5255]: 5257: sHs[Info]: 1304523154.708143: hsMapLog.cpp(47): hsMapHpiShelfHsBUtils.cpp(240): [ 0:0:9:0:11:1 ] Resource [ 68 ] Sensor Number: 9, Sensor Type: 2, Event Category: 1, Assertion: 1, Event State: 16, PreviousState: 0, OEM Code: 0, Sensor Specific: 0 Actual Output populated on Log: May 4 10:18:37 base0-0-0-1-0-11-1 shelfHs[4539]: 4542: sHs[Info]: 1304518717.501982: hsMapLog.cpp(47): hsMapHpiShelfHsBUtils.cpp(240): [ 0:0:9:0:11:1 ] Resource [ 98 ] Sensor Number: 9, Sensor Type: 2, Event Category: 1, Assertion: 1, So,if anyone know some kind of fix/patch for resolving this issue, please let me know. Your help would be highly appreciated. Thanks & regards Swati. =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you

4 7

[PATCH] gprocess: Fix --no-caps handling when building with hardening options.
by Gergely Nagy 13 Nov '11

13 Nov '11

When both glib and syslog-ng is built with hardening options enabled, overwriting a string pointer with a FALSE boolean does not have the expected results: the string will not become NULL. In this case, --no-caps will end up with a segfault, because process_opts.caps is non-NULL, and points to junk. To fix this, introduce a new function that sets process_opts.caps to NULL explicitly, and use this function as a callback, instead of assuming that a FALSE boolean will have the expected results. Reported-By: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) diff --git a/lib/gprocess.c b/lib/gprocess.c index 2ccaffe..aaefbab 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1440,6 +1440,14 @@ g_process_process_mode_arg(const gchar *option_name G_GNUC_UNUSED, const gchar * return TRUE; } +static gboolean +g_process_process_no_caps(const gchar *option_name G_GNUC_UNUSED, const gchar *value G_GNUC_UNUSED, + gpointer data G_GNUC_UNUSED, GError *error) +{ + process_opts.caps = NULL; + return TRUE; +} + static GOptionEntry g_process_option_entries[] = { { "foreground", 'F', G_OPTION_FLAG_REVERSE, G_OPTION_ARG_NONE, &process_opts.mode, "Do not go into the background after initialization", NULL }, @@ -1450,7 +1458,7 @@ static GOptionEntry g_process_option_entries[] = { "gid", 0, G_OPTION_FLAG_HIDDEN, G_OPTION_ARG_STRING, &process_opts.group, NULL, NULL }, { "chroot", 'C', 0, G_OPTION_ARG_STRING, &process_opts.chroot_dir, "Chroot to this directory", "<dir>" }, { "caps", 0, 0, G_OPTION_ARG_STRING, &process_opts.caps, "Set default capability set", "<capspec>" }, - { "no-caps", 0, G_OPTION_FLAG_REVERSE, G_OPTION_ARG_NONE, &process_opts.caps, "Disable managing Linux capabilities", NULL }, + { "no-caps", 0, G_OPTION_FLAG_NO_ARG, G_OPTION_ARG_CALLBACK, g_process_process_no_caps, "Disable managing Linux capabilities", NULL }, { "pidfile", 'p', 0, G_OPTION_ARG_STRING, &process_opts.pidfile, "Set path to pid file", "<pidfile>" }, { "enable-core", 0, 0, G_OPTION_ARG_NONE, &process_opts.core, "Enable dumping core files", NULL }, { "fd-limit", 0, 0, G_OPTION_ARG_INT, &process_opts.fd_limit_min, "The minimum required number of fds", NULL }, -- 1.7.7.2

3 4

Possible memleak in 3.3 HEAD
by Jakub Jankowski 12 Nov '11

12 Nov '11

Hi, I think I'm having a memleak in 3.3's HEAD. My production system, which receives ~300 messages/s throughout the day (so ~26.4 million messages daily) leaks ~700MB of memory per day. I did some investigation, here are the details: # syslog-ng -V syslog-ng 3.3.0beta2 Installer-Version: 3.3.0beta2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#no_branch#d6eea6a1130f8343b6600976d9f2319f9ed86c88 Compile-Date: Sep 9 2011 16:32:59 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: syslogformat,affile,afmongodb,basicfuncs,csvparser,dbparser,afsql,convertfuncs,afprog,confgen,afsocket,afuser,afsocket-tls,dummy Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: off # My build is patched with: - https://github.com/algernon/libmongo-client/commit/93c5cff05feec5d7f421acfd… (so it compiles on rhel5) - https://lists.balabit.hu/pipermail/syslog-ng/2011-April/016375.html (I need this feature) I don't think any of these is the reason, though. My test configuration is basically this: $ cat /etc/syslog-ng/syslog-ng.conf @version: 3.3 options { time_reap (1); log_fifo_size (100000); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); log_msg_size(16384); owner ("root"); group ("root"); perm (0644); dir_owner ("root"); dir_group ("root"); dir_perm (0755); }; source s_www { syslog(ip(0.0.0.0) port(515) flags(no-multi-line) log_iw_size(6000) max-connections(30) tags("src.syslog", "src.port.515", "www", "www.accesslog")); }; filter f_svc_www_accesslog { tags("www.accesslog"); }; parser p_apache_split_vhost_access { csv-parser( columns("APACHE.ACCESS.VHOST", "APACHE.ACCESS.LOG_MESSAGE") flags(greedy) delimiters(" ") ); }; parser dbp_apache_split_vhost_access { db-parser( file("/etc/syslog-ng/pdb1.xml") ); }; destination d_access_split { file("/logs/$S_YEAR/$S_MONTH/$S_DAY/${APACHE.ACCESS.VHOST:-default}-access.log-${S_YEAR}${S_MONTH}${S_DAY}" template("${APACHE.ACCESS.LOG_MESSAGE}\n")); }; log { source(s_www); filter(f_svc_www_accesslog); parser(dbp_apache_split_vhost_access); destination(d_access_split); flags(final); }; $ Those two parsers (csv-parser and db-parser) are doing exactly the same, they split my Apache HTTPd access logs into two parts: first column (which is the vhost name) and the rest of the line. I have them both because I needed to know if that's a leak in any particular parser type. (Apparently not) I'm not very experienced with valgrind, but I did some tests, and what I've found is that the number of "definitely lost" blocks depends on the number of files created by d_access_split destination. So I took 10000 lines of some real httpd logs and did two tests: 1) replace real vhost names with completely random strings, to maximize number of files created by d_access_split 2) replace real vhost names with constant string, so my destination always writes to the same file. And this is what I got, using "G_SLICE=always-malloc valgrind --leak-check=full -v --show-reachable=yes --track-origins=yes --log-file=valgrind.log syslog-ng -F --no-caps -p /var/run/syslog-ng.pid" 1) random files: ==22416== 3,675,952 bytes in 9,989 blocks are definitely lost in loss record 578 of 578 ==22416== at 0x4021EC2: calloc (vg_replace_malloc.c:418) ==22416== by 0x40FF57D: g_malloc0 (in /lib/libglib-2.0.so.0.1200.3) ==22416== by 0x4052899: log_queue_fifo_new (logqueue-fifo.c:438) ==22416== by 0x4046057: log_dest_driver_acquire_queue_method (driver.c:153) ==22416== by 0x47081D1: affile_dw_init (driver.h:178) ==22416== by 0x4707C41: affile_dd_open_writer (logpipe.h:239) ==22416== by 0x405C258: main_loop_call (mainloop.c:145) ==22416== by 0x4707A6B: affile_dd_queue (affile.c:1127) ==22416== by 0x40451C8: log_dest_group_queue (logpipe.h:288) ==22416== by 0x404B9DD: log_multiplexer_queue (logpipe.h:288) ==22416== by 0x404EC16: log_parser_queue (logpipe.h:288) ==22416== by 0x4047473: log_filter_pipe_queue (logpipe.h:288) ==22416== ==22416== LEAK SUMMARY: ==22416== definitely lost: 3,676,156 bytes in 9,992 blocks ==22416== indirectly lost: 6,541 bytes in 56 blocks ==22416== possibly lost: 16,967 bytes in 117 blocks ==22416== still reachable: 76,908 bytes in 3,286 blocks ==22416== suppressed: 0 bytes in 0 blocks 2) single file (constant filename): ==19460== 368 bytes in 1 blocks are definitely lost in loss record 525 of 577 ==19460== at 0x4021EC2: calloc (vg_replace_malloc.c:418) ==19460== by 0x40FF57D: g_malloc0 (in /lib/libglib-2.0.so.0.1200.3) ==19460== by 0x4052899: log_queue_fifo_new (logqueue-fifo.c:438) ==19460== by 0x4046057: log_dest_driver_acquire_queue_method (driver.c:153) ==19460== by 0x47081D1: affile_dw_init (driver.h:178) ==19460== by 0x4707C41: affile_dd_open_writer (logpipe.h:239) ==19460== by 0x405C258: main_loop_call (mainloop.c:145) ==19460== by 0x4707A6B: affile_dd_queue (affile.c:1127) ==19460== by 0x40451C8: log_dest_group_queue (logpipe.h:288) ==19460== by 0x404B9DD: log_multiplexer_queue (logpipe.h:288) ==19460== by 0x404EC16: log_parser_queue (logpipe.h:288) ==19460== by 0x4047473: log_filter_pipe_queue (logpipe.h:288) (...) ==19460== LEAK SUMMARY: ==19460== definitely lost: 572 bytes in 4 blocks ==19460== indirectly lost: 6,541 bytes in 56 blocks ==19460== possibly lost: 12,919 bytes in 106 blocks ==19460== still reachable: 76,908 bytes in 3,286 blocks ==19460== suppressed: 0 bytes in 0 blocks Could anyone proficient in syslog-ng source code take a look on this? Can I do anything else to help you diagnose this? (I can provide complete valgrind output logs if this helps.) Also, I'll try to reproduce this on other versions later this week, unless someone fixes this earlier. :) Cheers, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

7 43

RFC: value-pairs key rewrite framework, part N+1
by Gergely Nagy 12 Nov '11

12 Nov '11

Hi! After my former mails (see the thread starting at http://thread.gmane.org/gmane.comp.syslog-ng/11355/focus=11421) I'd like to do a recap, and ask for comments, as I came up with a few new ideas. The purpose of the value-pairs key rewrite framework is to make it possible to apply various transformations to the keys we selected with VP. So that we can add or remove prefixes, replace parts of the string, and so on. As of this writing, code to do this exists on one of my branches, but it hasn't been touched for a while, and I planned to update it in the near future. And that's when it dawned on me, that perhaps the syntax isn't all that great. To show why I think that, let's see an example first: value-pairs( scope("everything") rekey( add-prefix(".secevt" "events") add-prefix(".classifier" "syslog-ng") shift(".sdata.*" 1) replace("." "_") ) ); This will add an "events" prefix to each key that starts with ".secevt", so that ".secevt.verdict" becomes "events.secevt.verdict"; similary, ".classifier.class" becomes "syslog-ng.classifier.class"; keys that match '.sdata.*' get shifted to the right, removing the dot. And all remaining dots at the begininng of a key will get replaced by an underscore instead. This kinda makes sense, and I could even massage the syntax into format-json: $(format-json --scope everything --rekey --add-prefix .secevt=events --addprefix .classifier=syslog-ng --shift .sdata.*=1 --replace .=_ --end-rekey) However, this syntax has the downside of transformations being global: I can't choose subset of my keys, and apply a list of transformations on those and only those. Once I made a transformation, any transformations in the list afterwards will see the transformed key. So I can't easily say: "take all the keys starting with '.sdata.', shift them 6 chars, then replace any key names that start with 'win' with 'lose', and finally prefix them with 'whatever'". With the current syntax, that's next to impossible to do sanely. It's also not all that intuitive.. So I came up with a different syntax: wiring rekey into the key() option of value-pairs! That way, we already selected a subset to work on, and the transformations would apply to only those. (This could be combined with the global syntax aswell, though) So it'd look something like this: value-pairs( scope("everything") key(".secevt.*" rekey(add-prefix("events"))) key(".classifier.*" rekey(add-prefix("syslog-ng"))) key(".sdata.*" rekey(shift(1))) key(".*" rekey(replace ("." "_"))) ); This would achieve the exact same effect as the example above, but with a clearer syntax, perhaps. It would also mean that key rewriting can be described at the same place where the key is selected to begin with. The downside of this is that it'd be a bit harder to come up with a syntax for format-json that mimics the config file syntax. So, instead of trying to do that and end up with something horrible, I have another proposal: lets make value-pairs a top-level citizen, so that it joins the ranks of filter{} and rewrite{} and the like! That way, we could turn the following ugly thing: destination d_structured { mongodb( value-pairs( scope("everything") key(".secevt.*" rekey(add-prefix("events"))) key(".classifier.*" rekey(add-prefix("syslog-ng"))) key(".sdata.*" rekey(shift(1))) key(".*" rekey(replace ("." "_"))) ) ); file("/var/log/structured.json" template("$(format-json <repeat the above stuff, but with format-json syntax>)\n")); }; Into this beauty: valuepairs vp_example { scope(everything); key(".secevt.*" rekey(add-prefix("events"))); key(".classifier.*" rekey(add-prefix("syslog-ng"))); key(".sdata.*" rekey(shift(1))); key(".*" rekey(replace ("." "_"))); }; destination d_structured { mongodb(value-pairs(vp_example)); file("/var/log/structured.json" template("$(format-json --with-config vp_example)\n")); }; This would have the nice consequence of not having to keep two parsers in sync: format-json would only have a --with-config option, and nothing else. So, in the end, this whole boils down to two questions: * What do you think about allowing (or even moving) rekey() inside key() options? * What do you think about introducing a top-level valuepairs element, and dropping the format-json argument parsing stuff? -- |8]

2 5

[PATCH] resolve_sockaddr: Fix domain stripping on initial lookup.
by Gergely Nagy 12 Nov '11

12 Nov '11

When looking up a hostname, we were only setting the 'positive' bool during lookup, so it was only set when the host already existed in the cache. When it didn't, and we just put it there right after the failed lookup, the bool remained false. This resulted in the domain part not being stripped even when use_fqdn(no) was set. The fix implemented in this patch is dead simple: if we managed to resolve a host, we set the bool to TRUE. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/misc.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/lib/misc.c b/lib/misc.c index 80f5877..82c2d1b 100644 --- a/lib/misc.c +++ b/lib/misc.c @@ -211,7 +211,10 @@ resolve_sockaddr(gchar *result, gsize *result_len, GSockAddr *saddr, gboolean us hp = gethostbyaddr(addr, addr_len, saddr->sa.sa_family); hname = (hp && hp->h_name) ? hp->h_name : NULL; - + + if (hname) + positive = TRUE; + if (use_dns_cache && hname) { /* resolution success, store this as a positive match in the cache */ -- 1.7.7.1

2 1

syslog-ng 3.2.4 MySQL connection loss Server has gone away
by Erik Maciejewski 12 Nov '11

12 Nov '11

Hello, I have been researching an issue with syslog-ng 3.2.4 (both the Linux glibc2.3.6 and platform independent compiled from source versions) and MySQL on CentOS 5.6 in which syslog-ng tries to insert a log messages using a dead TCP or unix socket connection to a MySQL database. The issue results in the max attempt to insert a log message and then the message subsequently dropped. This seems to occur regularly in predominately low message volume environments, but has the potential to affect all environments using a MySQL (or other db) destination. The root cause of the issue seems to be directly related to the health of the connection between syslog-ng and MySQL and can be affected by the "wait_timeout" value used by MySQL to kill off inactive connections. I want to provide justification for implementing a fix to syslog-ng as I believe manipulating a, possibly tuned, environment variable for MySQL would seem like the wrong approach to take for remedial action. I feel that many MySQL instances are implemented in shared environments and are themselves shared by many applications. New connections to a MySQL instance are generally regarded as low cost and, it would seem, more often than not never used in a persisted fashion when supporting distributed applications. That being said, many times the "wait_timeout" value is set very low so that the MySQL instance can effectively control the number of simultaneous connections. By taking somewhat of a black-box approach to the issue (I'm just starting to explore syslog-ng), I noticed that there doesn't seem to be any health checking of the TCP or unix socket connection in the SQL destination implementation after the initial connection is made (most of the db actions being abstracted away by libdbi). Even if a syslog-ng database thread is suspended and reactivated due to error, there does not seem to be any attempt to check the health of the database connection. I have implemented a fix in my compiled version that will check a database connection's health and attempt to reconnect upon discovering a dead connection. I would like to know the policies/procedures/best practices for submitting such issues and/or fixes for discussion, review, and implementation. If this is a known issue or I am just wrong, all the better! Thank you guys for keeping this project going and I'm looking forward to implementing syslog-ng in a production environment in the very near future! Regards, Erik Maciejewksi

2 7

Variables in PatternDB - what am I doing wrong?
by Daniel Ankers 12 Nov '11

12 Nov '11

Hi, I've got the following as part of my ruleset:  <rule id='def42cfb-4364-4cfd-bbf1-cdf4f0bc3c88' class='lowpri' context-id="badauth-${badauth.errorsrc}" context-timeout='600' context-scope='host'> <patterns> <pattern>%TCP-6-BADAUTH: No MD5 digest from @IPvANY:badauth.errorsrc:@(@NUMBER::@) to @IPvANY:badauth.errordst:@(@NUMBER::@) (RST)</pattern> </patterns> <examples> <example> <test_message program='patternize'>%TCP-6-BADAUTH: No MD5 digest from 195.66.XXX.XX(179) to 195.66.YYYY.YY(56561) (RST)</test_message> </example> <example> <test_message program='patternize'>%TCP-6-BADAUTH: No MD5 digest from 2001:7F8:X::abcd:1(179) to 2001:7F8:X::1234:1(21296) (RST)</test_message> </example> </examples> <actions> <action trigger='match'> <message> <values> <value name='MESSAGE'>$(if ("${badauth.errorsrc}@1" eq "${badauth.errorsrc}@2") "Duplicate" "First") BGP Authentication Error to neighbour ${badauth.errorsrc}@1 - last one was ${badauth.errorsrc}@2</value> </values> </message> </action> </actions> </rule>  When I feed example messages in from a file, I expect to see something like the following: Nov 4 10:40:54 syslog First BGP Authentication Error to neighbour 2001:7F8:X::abcd:1 - last one was Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 2001:7F8:X::abcd:1 - last one was 2001:7F8:X::abcd:1 Nov 4 10:40:54 syslog First BGP Authentication Error to neighbour 195.66.Y.Z - last one was Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was 195.66.Y.Z Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was 195.66.Y.Z Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was 195.66.Y.Z Instead I see: Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 2001:7F8:X::abcd:1 - last one was Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 2001:7F8:X::abcd:1 - last one was 2001:7F8:4::abcd:1 Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was 195.66.Y.Z Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was 195.66.Y.Z Nov 4 10:40:54 syslog Duplicate BGP Authentication Error to neighbour 195.66.Y.Z - last one was 195.66.Y.Z What am I getting wrong with the "if" function? The background to this is that the first of these messages from any host is useful information - however they are sent every 10 seconds and I'm trying to suppress all but the first message for each neighbour. I've tried several different ways of doing this including rate-limiting on the <action> but none of them have worked the way I expected. Thanks, Dan Daniel Ankers

2 2

Problems writing to files with different templates/ownership in 3.x
by Brian De Wolf 12 Nov '11

12 Nov '11

Hello, I'm using syslog-ng as a remote syslog server for several of our internal groups. As such, when a syslog arrives, I use the source address to determine the ownership of the log file. Here is how I'm currently doing this: source net { udp(ip(0.0.0.0), port(514) flags(no-multi-line)); }; filter examplefilter { netmask("10.0.0.0/255.255.0.0"); }; filter example2filter { netmask("192.168.0.0/255.255.0.0"); }; destination examplelog { file("/var/log/remote/$HOST.log" group(examplegroup)); }; destination example2log { file("/var/log/remote/$HOST.log" group(example2group)); }; log { source(net); filter(examplefilter); destination(examplelog); flags(final); }; log { source(net); filter(example2filter); destination(example2log); flags(final); }; This worked fine in 1.6.x and appears to work fine in 3.2.x, but when logrotate pokes syslog-ng I get lines like this: syslog-ng: Internal error, duplicate configuration elements refer to the same persistent config; name='affile_dd_writers(/var/log/remote/$HOST.log)' Is there a better way to do this which only produces one reference to "/var/log/remote/$HOST.log"? Also, I'm using a similar set of filters and destinations to apply different templates to text that is being written to a single file. This also produces the error, but I am more concerned that the combined file might experience issues if the config isn't supposed to have multiple destinations be the same location. Is there a way to use filters to apply templates while still only having one destination defined? Could my current setup result in lost log lines? Thanks, Brian

2 1