- syslog-ng - lists.balabit.hu

Compiling syslog-ng on Ubuntu.
by andrea.meconi＠libero.it 16 Nov '11

16 Nov '11

Hello.I'm a begineer.I need to recompile syslog-ng with option spoof-source on Ubuntu 8.04 starting from a minimal cd installation.Make install terminates without errors and installation.Many thanks for your help.Regards.Andrea

2 1

Referencing earlier message
by Hery Fanomezantsoa 16 Nov '11

16 Nov '11

When using correlating message with syslog-ng 3.3.1 i get only the value in the last matching rule but not the earlier. Here is my patterndb.xml <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2011-11-07'> > <ruleset name='ecelerity' id='12345678'> > <pattern>ecelerity</pattern> > <rules> > <rule provider='me' id='123475980' class='system' > context-scope='program' context-id='${MSG.UID}' > context-timeout='10'> > <patterns> > > <pattern>@ESTRING:LOG.UTC:|@@ESTRING:LOG.UID:|@ORCPTS|@ANYSTRING:LOG.VAL:@ > </pattern> > </patterns> <examples> > <example> > <test_message > program="ecelerity">1319550976|c0a80a3c-b7f6c6d000002063-1f-4ea6c0004833|ORCPTS|s.andriamampianina@ > ***.**</test_message> > <test_values> > <test_value name="LOG.UTC">1319550976</test_value> > <test_value > name="LOG.UID">c0a80a3c-b7f6c6d000002063-1f-4ea6c0004833</test_value> > > <test_value name="LOG.VAL">s.andriamampianina@ > ***.**</test_value> > </test_values> > </example> > </examples> > </rule> > <rule provider='me' id='123475981' class='system' > context-id='${MSG.UID}'> > <patterns> > > <pattern>@ESTRING:LOG.UTC:|@@ESTRING:LOG.UID:|@SENDER|@ANYSTRING:LOG.VAL:@ > </pattern> > </patterns> > <actions> > <action> > <message> > <values> > <value name="MESSAGE">From ${LOG.VAL}@1 to ${LOG.VAL}@2. > </value> > <value name="TRIGGER">yes</value> > </values> > </message> > </action> > </actions> > </rule> > </rules> > </ruleset> > </patterndb> > And the message i get is "from *****(a)**.** to ." Where did I get wrong?

2 2

syslog-ng 3.3.2 seg faulting sending message via tcp
by Patrick H. 16 Nov '11

16 Nov '11

Syslog-ng 3.3.2 (obtained from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.2/source) is segfaulting when it tries to send a message to a TCP destination. I'm asking here instead of a bug report because I didnt see a 3.3.2 release announcement, so not sure if this is the final build or not. Anyway, gdb backtrace shows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x40f67940 (LWP 29440)] 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 288 s->queue(s, msg, path_options, s->queue_data); (gdb) where #0 0x00002b047a812261 in log_pipe_queue (s=0x12d0d94f, msg=0x12d04400, path_options=0x40f64150) at logpipe.h:288 #1 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at logmpx.c:125 #2 0x00002b047a80ce2c in log_pipe_queue (s=0x12cd7e60, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #3 0x00002b047a80cddb in log_pipe_forward_msg (self=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:275 #4 0x00002b047a80cd79 in log_filter_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0, user_data=0x0) at filter.c:698 #5 0x00002b047a812264 in log_pipe_queue (s=0x12d0d810, msg=0x12d04400, path_options=0x40f642d0) at logpipe.h:288 #6 0x00002b047a8121b2 in log_multiplexer_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at logmpx.c:125 #7 0x00002b047a82cd20 in log_pipe_queue (s=0x12cd7e00, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #8 0x00002b047a82cccf in log_pipe_forward_msg (self=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #9 0x00002b047a82cc8a in log_source_group_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550, user_data=0x0) at sgroup.c:102 #10 0x00002b047a81ee1f in log_pipe_queue (s=0x12d127e0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:288 #11 0x00002b047a81edce in log_pipe_forward_msg (self=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #12 0x00002b047a81ee32 in log_pipe_queue (s=0x12d125a0, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #13 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #14 0x00002b047a81ee32 in log_pipe_queue (s=0x12cfb920, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:292 #15 0x00002b047a81edce in log_pipe_forward_msg (self=0x12cd8800, msg=0x12d04400, path_options=0x40f64550) at logpipe.h:275 #16 0x00002b047a81eccc in log_source_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0, user_data=0x0) at logsource.c:290 #17 0x00002b047a81c969 in log_pipe_queue (s=0x12cd8800, msg=0x12d04400, path_options=0x40f645f0) at logpipe.h:288 #18 0x00002b047a81c919 in log_reader_handle_line (self=0x12cd8800, line=0x12d09800 "<22>Nov 14 02:48:00 postfix/smtpd[23425]: connect from slider.dev.usa.net[165.212.101.17]\n", length=89, saddr=0x0) at logreader.c:503 #19 0x00002b047a81cadd in log_reader_fetch_log (self=0x12cd8800) at logreader.c:561 #20 0x00002b047a81be25 in log_reader_work_perform (s=0x12cd8800) at logreader.c:115 #21 0x00002b047a824648 in main_loop_io_worker_job_start (self=0x12cd8a98) at mainloop.c:367 #22 0x00002b047a86ae96 in iv_work_thread_got_event (_thr=0x12cdb1f0) at iv_work.c:113 #23 0x00002b047a869e17 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67 #24 0x00002b047a86a3db in iv_event_raw_got_event (_this=0x40f67858) at iv_event_raw.c:82 #25 0x00002b047a866945 in iv_run_active_list (active=0x40f64c20) at iv_main.c:219 #26 0x00002b047a866b83 in iv_main () at iv_main.c:269 #27 0x00002b047a86b290 in iv_work_thread (_thr=0x12cdb1f0) at iv_work.c:196 #28 0x00002b047a86b81a in iv_thread_handler (_thr=0x12cda400) at iv_thread.c:100 #29 0x00000039820064a7 in start_thread () from /lib64/libpthread.so.0 #30 0x00000039818d3c2d in clone () from /lib64/libc.so.6 Config: @version: 3.3 options { time_reopen(1); use_dns(no); use_fqdn(yes); keep_hostname(yes); normalize_hostnames(yes); create_dirs(yes); dir_perm(0755); perm(0644); mark_freq(0); threaded(yes); }; @include "scl.conf" source s_local { system(); internal(); }; filter f_mail { facility(mail); }; destination d_central { tcp('syslog.usa.net' port(514) log_fifo_size(50000)); }; log { source(s_local); filter(f_mail); destination(d_central); };

3 5

[Bug 111] New: RPM build errors: Installed (but unpackaged) file(s) found:
by bugzilla＠bugzilla.balabit.com 14 Nov '11

14 Nov '11

https://bugzilla.balabit.com/show_bug.cgi?id=111 Summary: RPM build errors: Installed (but unpackaged) file(s) found: Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: packaging AssignedTo: folti(a)balabit.hu ReportedBy: owen.mann(a)interactivedata.com Type of the Report: bug Estimated Hours: 0.0 when attempting `rpmbuild -tb syslog-ng_3.2beta1.tar.gz`, the compilation completes, but: [...] Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/syslog-ng-root error: Installed (but unpackaged) file(s) found: /bin/pdbtool /bin/update-patterndb /etc/syslog-ng/modules.conf /etc/syslog-ng/scl.conf /lib/libsyslog-ng.la /lib/libsyslog-ng.so /lib/libsyslog-ng.so.0 /lib/libsyslog-ng.so.0.0.0 /lib/syslog-ng/libaffile.la /lib/syslog-ng/libaffile.so /lib/syslog-ng/libafprog.la /lib/syslog-ng/libafprog.so /lib/syslog-ng/libafsocket-notls.la /lib/syslog-ng/libafsocket-notls.so /lib/syslog-ng/libafsocket-tls.la /lib/syslog-ng/libafsocket-tls.so /lib/syslog-ng/libafsocket.so /lib/syslog-ng/libafuser.la /lib/syslog-ng/libafuser.so /lib/syslog-ng/libbasicfuncs.la /lib/syslog-ng/libbasicfuncs.so /lib/syslog-ng/libconfgen.la /lib/syslog-ng/libconfgen.so /lib/syslog-ng/libconvertfuncs.la /lib/syslog-ng/libconvertfuncs.so /lib/syslog-ng/libcsvparser.la /lib/syslog-ng/libcsvparser.so /lib/syslog-ng/libdbparser.la /lib/syslog-ng/libdbparser.so /lib/syslog-ng/libdummy.la /lib/syslog-ng/libdummy.so /lib/syslog-ng/libsyslogformat.la /lib/syslog-ng/libsyslogformat.so /sbin/syslog-ng-ctl /share/include/scl/pacct/plugin.conf /share/include/scl/syslogconf/README /share/include/scl/syslogconf/convert-syslogconf.awk /share/include/scl/syslogconf/plugin.conf /share/include/scl/system/generate-system-source.sh /share/include/scl/system/plugin.conf /share/xsd/patterndb-1.xsd /share/xsd/patterndb-2.xsd /share/xsd/patterndb-3.xsd [...] If these files are added to the %files section in syslog-ng.spec, the build completes successfully. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

syslog-ng 3.3.2 has been released
by devel＠balabit.hu 14 Nov '11

14 Nov '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.2 SUMMARY : new stable release DATE : Nov 13, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.2) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.2 Sat, 12 Nov 2011 20:48:53 +0100 Bugfixes: ========= * Fixed a segmentation fault on reload when using the same rewrite rule from multiple log paths. * Fixed a segmentation fault when processing a reload request in case an existing tcp() source is removed from the config and there are open connections. * Fixed a possible segmentation fault in the scalable queue implementation, which happens in case a destination is slower to process messages than syslog-ng would like to send them. * Fixed a possible file() destination issue that could cause syslog-ng to omit data or to write garbage to the log file in case the kernel reports that only a smaller portion of the actual write request could be accomplished. * Fixed an "internal error duplicate config element" error during reload due to an invalid bugfix applied for 3.3.1. Older beta versions of 3.3 were not affected. * Fixed a memory leak that causes macro based file destinations to leak their queue when destination files are closed due to time-reap(). * Fixed the handling of the condition() option for rewrite rules. * Fixed a race condition in value-pairs support, potentially causing heap corruption problems when $(format-json) is used in threaded mode. * Fixed a memory leak in value-pairs template function argument parsing, fixing a leak if $(format-json) is used. * Repeated definitions of source, destination, filter, rewrite, parser and block elements are not allowed by default anymore. These are reported as configuration errors unless @define allow-config-dups 1 is specified in the configuration file. * Fixed pdbtool error reporting in "pdbtool test" to make it easier to understand what went wrong. * Added an SQL connection health check in case an INSERT failed. This way syslog-ng handles SQL server timeouts better. * Fixed support for systemd socket activation. Previously such sockets were not set to non-blocking mode, causing syslog-ng to hang. * Fixed the filter() function in the filter expression to work also when used as a part of an AND or OR construct. * Allow the sql() destination to operate even without an indexes() option. That parameter was meant to be optional, but it wasn't. * Fixed compilation issues if no OpenSSL is present. * Fixed a minor memory leak in the usertty() driver that can increase memory usage on every reload. (The username() parameter wasn't properly freed on reload). * Fixed a minor memory leak in the sql() driver that can increase the memory usage on every reload (indexes() parameter wasn't properly freed on reload). Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Anthony Lineham (Allied Telesys) Arkadiusz Miśkiewicz (PLD Linux) Attila Szalay (BalaBit) Balazs Scheidler (BalaBit) Balint Kovacs (BalaBit) Daniel Ankers Dave Rawks (Pandora) Enrico Scholz (Technische Universitat Chemnitz) Eric Lindvall (Seven Scale) Erik Maciejewski Gergely Nagy (BalaBit) Hery Fanomezantsoa Jack Nagel Jakub Jankowski (SuperHost.pl) Leonid Isaev (Indiana University) Martin Holste Michael Sterrett (Gentoo) Patrick H. Peter Czanik (BalaBit) Viktor Juhasz (BalaBit) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Logs after "\n" are getting truncated
by Swati4 M/TVM/TCS 14 Nov '11

14 Nov '11

Hi All, When logging multiline messages (simple strings with in-line newlines "\n") our logs only show the first line (ie) before "\n" is encountered and rest of the line after the first newline character("\n") is truncated. Example: Code Snippet: HS_MAP_INFO_LOG("[ %s ] Resource [ %d ] Sensor Number: %2d, Sensor Type: %d, Event Category: %d, Assertion: %d, \n" "Event State: %d, Previous State: %d, OEM Code: %d, Sensor Specific: %d\n", eidStr, pEvt->Source, thisSensorEvent->SensorNum, thisSensorEvent->SensorType, thisSensorEvent->EventCategory, thisSensorEvent->Assertion, thisSensorEvent->EventState, thisSensorEvent->PreviousState, thisSensorEvent->Oem, thisSensorEvent->SensorSpecific); Expected Output on Log: May 4 11:32:34 base0-0-0-1-0-11-1 shelfHs[5255]: 5257: sHs[Info]: 1304523154.708143: hsMapLog.cpp(47): hsMapHpiShelfHsBUtils.cpp(240): [ 0:0:9:0:11:1 ] Resource [ 68 ] Sensor Number: 9, Sensor Type: 2, Event Category: 1, Assertion: 1, Event State: 16, PreviousState: 0, OEM Code: 0, Sensor Specific: 0 Actual Output populated on Log: May 4 10:18:37 base0-0-0-1-0-11-1 shelfHs[4539]: 4542: sHs[Info]: 1304518717.501982: hsMapLog.cpp(47): hsMapHpiShelfHsBUtils.cpp(240): [ 0:0:9:0:11:1 ] Resource [ 98 ] Sensor Number: 9, Sensor Type: 2, Event Category: 1, Assertion: 1, So,if anyone know some kind of fix/patch for resolving this issue, please let me know. Your help would be highly appreciated. Thanks & regards Swati. =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you

4 7

[PATCH] gprocess: Fix --no-caps handling when building with hardening options.
by Gergely Nagy 13 Nov '11

13 Nov '11

When both glib and syslog-ng is built with hardening options enabled, overwriting a string pointer with a FALSE boolean does not have the expected results: the string will not become NULL. In this case, --no-caps will end up with a segfault, because process_opts.caps is non-NULL, and points to junk. To fix this, introduce a new function that sets process_opts.caps to NULL explicitly, and use this function as a callback, instead of assuming that a FALSE boolean will have the expected results. Reported-By: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) diff --git a/lib/gprocess.c b/lib/gprocess.c index 2ccaffe..aaefbab 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1440,6 +1440,14 @@ g_process_process_mode_arg(const gchar *option_name G_GNUC_UNUSED, const gchar * return TRUE; } +static gboolean +g_process_process_no_caps(const gchar *option_name G_GNUC_UNUSED, const gchar *value G_GNUC_UNUSED, + gpointer data G_GNUC_UNUSED, GError *error) +{ + process_opts.caps = NULL; + return TRUE; +} + static GOptionEntry g_process_option_entries[] = { { "foreground", 'F', G_OPTION_FLAG_REVERSE, G_OPTION_ARG_NONE, &process_opts.mode, "Do not go into the background after initialization", NULL }, @@ -1450,7 +1458,7 @@ static GOptionEntry g_process_option_entries[] = { "gid", 0, G_OPTION_FLAG_HIDDEN, G_OPTION_ARG_STRING, &process_opts.group, NULL, NULL }, { "chroot", 'C', 0, G_OPTION_ARG_STRING, &process_opts.chroot_dir, "Chroot to this directory", "<dir>" }, { "caps", 0, 0, G_OPTION_ARG_STRING, &process_opts.caps, "Set default capability set", "<capspec>" }, - { "no-caps", 0, G_OPTION_FLAG_REVERSE, G_OPTION_ARG_NONE, &process_opts.caps, "Disable managing Linux capabilities", NULL }, + { "no-caps", 0, G_OPTION_FLAG_NO_ARG, G_OPTION_ARG_CALLBACK, g_process_process_no_caps, "Disable managing Linux capabilities", NULL }, { "pidfile", 'p', 0, G_OPTION_ARG_STRING, &process_opts.pidfile, "Set path to pid file", "<pidfile>" }, { "enable-core", 0, 0, G_OPTION_ARG_NONE, &process_opts.core, "Enable dumping core files", NULL }, { "fd-limit", 0, 0, G_OPTION_ARG_INT, &process_opts.fd_limit_min, "The minimum required number of fds", NULL }, -- 1.7.7.2

3 4

Possible memleak in 3.3 HEAD
by Jakub Jankowski 12 Nov '11

12 Nov '11

Hi, I think I'm having a memleak in 3.3's HEAD. My production system, which receives ~300 messages/s throughout the day (so ~26.4 million messages daily) leaks ~700MB of memory per day. I did some investigation, here are the details: # syslog-ng -V syslog-ng 3.3.0beta2 Installer-Version: 3.3.0beta2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#no_branch#d6eea6a1130f8343b6600976d9f2319f9ed86c88 Compile-Date: Sep 9 2011 16:32:59 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: syslogformat,affile,afmongodb,basicfuncs,csvparser,dbparser,afsql,convertfuncs,afprog,confgen,afsocket,afuser,afsocket-tls,dummy Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: off # My build is patched with: - https://github.com/algernon/libmongo-client/commit/93c5cff05feec5d7f421acfd… (so it compiles on rhel5) - https://lists.balabit.hu/pipermail/syslog-ng/2011-April/016375.html (I need this feature) I don't think any of these is the reason, though. My test configuration is basically this: $ cat /etc/syslog-ng/syslog-ng.conf @version: 3.3 options { time_reap (1); log_fifo_size (100000); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); log_msg_size(16384); owner ("root"); group ("root"); perm (0644); dir_owner ("root"); dir_group ("root"); dir_perm (0755); }; source s_www { syslog(ip(0.0.0.0) port(515) flags(no-multi-line) log_iw_size(6000) max-connections(30) tags("src.syslog", "src.port.515", "www", "www.accesslog")); }; filter f_svc_www_accesslog { tags("www.accesslog"); }; parser p_apache_split_vhost_access { csv-parser( columns("APACHE.ACCESS.VHOST", "APACHE.ACCESS.LOG_MESSAGE") flags(greedy) delimiters(" ") ); }; parser dbp_apache_split_vhost_access { db-parser( file("/etc/syslog-ng/pdb1.xml") ); }; destination d_access_split { file("/logs/$S_YEAR/$S_MONTH/$S_DAY/${APACHE.ACCESS.VHOST:-default}-access.log-${S_YEAR}${S_MONTH}${S_DAY}" template("${APACHE.ACCESS.LOG_MESSAGE}\n")); }; log { source(s_www); filter(f_svc_www_accesslog); parser(dbp_apache_split_vhost_access); destination(d_access_split); flags(final); }; $ Those two parsers (csv-parser and db-parser) are doing exactly the same, they split my Apache HTTPd access logs into two parts: first column (which is the vhost name) and the rest of the line. I have them both because I needed to know if that's a leak in any particular parser type. (Apparently not) I'm not very experienced with valgrind, but I did some tests, and what I've found is that the number of "definitely lost" blocks depends on the number of files created by d_access_split destination. So I took 10000 lines of some real httpd logs and did two tests: 1) replace real vhost names with completely random strings, to maximize number of files created by d_access_split 2) replace real vhost names with constant string, so my destination always writes to the same file. And this is what I got, using "G_SLICE=always-malloc valgrind --leak-check=full -v --show-reachable=yes --track-origins=yes --log-file=valgrind.log syslog-ng -F --no-caps -p /var/run/syslog-ng.pid" 1) random files: ==22416== 3,675,952 bytes in 9,989 blocks are definitely lost in loss record 578 of 578 ==22416== at 0x4021EC2: calloc (vg_replace_malloc.c:418) ==22416== by 0x40FF57D: g_malloc0 (in /lib/libglib-2.0.so.0.1200.3) ==22416== by 0x4052899: log_queue_fifo_new (logqueue-fifo.c:438) ==22416== by 0x4046057: log_dest_driver_acquire_queue_method (driver.c:153) ==22416== by 0x47081D1: affile_dw_init (driver.h:178) ==22416== by 0x4707C41: affile_dd_open_writer (logpipe.h:239) ==22416== by 0x405C258: main_loop_call (mainloop.c:145) ==22416== by 0x4707A6B: affile_dd_queue (affile.c:1127) ==22416== by 0x40451C8: log_dest_group_queue (logpipe.h:288) ==22416== by 0x404B9DD: log_multiplexer_queue (logpipe.h:288) ==22416== by 0x404EC16: log_parser_queue (logpipe.h:288) ==22416== by 0x4047473: log_filter_pipe_queue (logpipe.h:288) ==22416== ==22416== LEAK SUMMARY: ==22416== definitely lost: 3,676,156 bytes in 9,992 blocks ==22416== indirectly lost: 6,541 bytes in 56 blocks ==22416== possibly lost: 16,967 bytes in 117 blocks ==22416== still reachable: 76,908 bytes in 3,286 blocks ==22416== suppressed: 0 bytes in 0 blocks 2) single file (constant filename): ==19460== 368 bytes in 1 blocks are definitely lost in loss record 525 of 577 ==19460== at 0x4021EC2: calloc (vg_replace_malloc.c:418) ==19460== by 0x40FF57D: g_malloc0 (in /lib/libglib-2.0.so.0.1200.3) ==19460== by 0x4052899: log_queue_fifo_new (logqueue-fifo.c:438) ==19460== by 0x4046057: log_dest_driver_acquire_queue_method (driver.c:153) ==19460== by 0x47081D1: affile_dw_init (driver.h:178) ==19460== by 0x4707C41: affile_dd_open_writer (logpipe.h:239) ==19460== by 0x405C258: main_loop_call (mainloop.c:145) ==19460== by 0x4707A6B: affile_dd_queue (affile.c:1127) ==19460== by 0x40451C8: log_dest_group_queue (logpipe.h:288) ==19460== by 0x404B9DD: log_multiplexer_queue (logpipe.h:288) ==19460== by 0x404EC16: log_parser_queue (logpipe.h:288) ==19460== by 0x4047473: log_filter_pipe_queue (logpipe.h:288) (...) ==19460== LEAK SUMMARY: ==19460== definitely lost: 572 bytes in 4 blocks ==19460== indirectly lost: 6,541 bytes in 56 blocks ==19460== possibly lost: 12,919 bytes in 106 blocks ==19460== still reachable: 76,908 bytes in 3,286 blocks ==19460== suppressed: 0 bytes in 0 blocks Could anyone proficient in syslog-ng source code take a look on this? Can I do anything else to help you diagnose this? (I can provide complete valgrind output logs if this helps.) Also, I'll try to reproduce this on other versions later this week, unless someone fixes this earlier. :) Cheers, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

7 43

RFC: value-pairs key rewrite framework, part N+1
by Gergely Nagy 12 Nov '11

12 Nov '11

Hi! After my former mails (see the thread starting at http://thread.gmane.org/gmane.comp.syslog-ng/11355/focus=11421) I'd like to do a recap, and ask for comments, as I came up with a few new ideas. The purpose of the value-pairs key rewrite framework is to make it possible to apply various transformations to the keys we selected with VP. So that we can add or remove prefixes, replace parts of the string, and so on. As of this writing, code to do this exists on one of my branches, but it hasn't been touched for a while, and I planned to update it in the near future. And that's when it dawned on me, that perhaps the syntax isn't all that great. To show why I think that, let's see an example first: value-pairs( scope("everything") rekey( add-prefix(".secevt" "events") add-prefix(".classifier" "syslog-ng") shift(".sdata.*" 1) replace("." "_") ) ); This will add an "events" prefix to each key that starts with ".secevt", so that ".secevt.verdict" becomes "events.secevt.verdict"; similary, ".classifier.class" becomes "syslog-ng.classifier.class"; keys that match '.sdata.*' get shifted to the right, removing the dot. And all remaining dots at the begininng of a key will get replaced by an underscore instead. This kinda makes sense, and I could even massage the syntax into format-json: $(format-json --scope everything --rekey --add-prefix .secevt=events --addprefix .classifier=syslog-ng --shift .sdata.*=1 --replace .=_ --end-rekey) However, this syntax has the downside of transformations being global: I can't choose subset of my keys, and apply a list of transformations on those and only those. Once I made a transformation, any transformations in the list afterwards will see the transformed key. So I can't easily say: "take all the keys starting with '.sdata.', shift them 6 chars, then replace any key names that start with 'win' with 'lose', and finally prefix them with 'whatever'". With the current syntax, that's next to impossible to do sanely. It's also not all that intuitive.. So I came up with a different syntax: wiring rekey into the key() option of value-pairs! That way, we already selected a subset to work on, and the transformations would apply to only those. (This could be combined with the global syntax aswell, though) So it'd look something like this: value-pairs( scope("everything") key(".secevt.*" rekey(add-prefix("events"))) key(".classifier.*" rekey(add-prefix("syslog-ng"))) key(".sdata.*" rekey(shift(1))) key(".*" rekey(replace ("." "_"))) ); This would achieve the exact same effect as the example above, but with a clearer syntax, perhaps. It would also mean that key rewriting can be described at the same place where the key is selected to begin with. The downside of this is that it'd be a bit harder to come up with a syntax for format-json that mimics the config file syntax. So, instead of trying to do that and end up with something horrible, I have another proposal: lets make value-pairs a top-level citizen, so that it joins the ranks of filter{} and rewrite{} and the like! That way, we could turn the following ugly thing: destination d_structured { mongodb( value-pairs( scope("everything") key(".secevt.*" rekey(add-prefix("events"))) key(".classifier.*" rekey(add-prefix("syslog-ng"))) key(".sdata.*" rekey(shift(1))) key(".*" rekey(replace ("." "_"))) ) ); file("/var/log/structured.json" template("$(format-json <repeat the above stuff, but with format-json syntax>)\n")); }; Into this beauty: valuepairs vp_example { scope(everything); key(".secevt.*" rekey(add-prefix("events"))); key(".classifier.*" rekey(add-prefix("syslog-ng"))); key(".sdata.*" rekey(shift(1))); key(".*" rekey(replace ("." "_"))); }; destination d_structured { mongodb(value-pairs(vp_example)); file("/var/log/structured.json" template("$(format-json --with-config vp_example)\n")); }; This would have the nice consequence of not having to keep two parsers in sync: format-json would only have a --with-config option, and nothing else. So, in the end, this whole boils down to two questions: * What do you think about allowing (or even moving) rekey() inside key() options? * What do you think about introducing a top-level valuepairs element, and dropping the format-json argument parsing stuff? -- |8]

2 5

[PATCH] resolve_sockaddr: Fix domain stripping on initial lookup.
by Gergely Nagy 12 Nov '11

12 Nov '11

When looking up a hostname, we were only setting the 'positive' bool during lookup, so it was only set when the host already existed in the cache. When it didn't, and we just put it there right after the failed lookup, the bool remained false. This resulted in the domain part not being stripped even when use_fqdn(no) was set. The fix implemented in this patch is dead simple: if we managed to resolve a host, we set the bool to TRUE. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/misc.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/lib/misc.c b/lib/misc.c index 80f5877..82c2d1b 100644 --- a/lib/misc.c +++ b/lib/misc.c @@ -211,7 +211,10 @@ resolve_sockaddr(gchar *result, gsize *result_len, GSockAddr *saddr, gboolean us hp = gethostbyaddr(addr, addr_len, saddr->sa.sa_family); hname = (hp && hp->h_name) ? hp->h_name : NULL; - + + if (hname) + positive = TRUE; + if (use_dns_cache && hname) { /* resolution success, store this as a positive match in the cache */ -- 1.7.7.1

2 1