On Tue, Sep 17, 2013 at 11:54 AM, Alexandre Biancalana <biancalana@gmail.com

wrote:

On Tue, Sep 17, 2013 at 10:30 AM, Gergely Nagy <algernon@balabit.hu>wrote:

...

Alexandre Biancalana <biancalana@gmail.com> writes:

...

Hi list,

I'm running syslog-ng 3.4.3 (installed from FreeBSD ports) on FreeBSD 9.1 with self-written patterndb file to parse apache error_log, mod_security and pagespeed errors.

The syslog-ng source is httpd error_log file, after starting syslog-ng it core dumps:

ERROR:logmsg.c:1303:log_msg_unref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) Abort trap (core dumped)

Let me know if you need more info.

Could you post a backtrace too, please?

Here we go:

GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found) ... Core was generated by `syslog-ng'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libsyslog-ng-3.4.3.so...done. Loaded symbols for /usr/local/lib/libsyslog-ng-3.4.3.so Reading symbols from /usr/lib/librt.so.1...done. Loaded symbols for /usr/lib/librt.so.1 Reading symbols from /usr/local/lib/libgmodule-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgmodule-2.0.so.0 Reading symbols from /usr/local/lib/libgthread-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgthread-2.0.so.0 Reading symbols from /usr/local/lib/libglib-2.0.so.0...done. Loaded symbols for /usr/local/lib/libglib-2.0.so.0 Reading symbols from /usr/local/lib/libintl.so.9...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/lib/libevtlog.so.0...done. Loaded symbols for /usr/local/lib/libevtlog.so.0 Reading symbols from /usr/local/lib/libpcre.so.3...done. Loaded symbols for /usr/local/lib/libpcre.so.3 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/syslog-ng/libsystem-source.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libsystem-source.so Reading symbols from /usr/lib/libz.so...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /usr/local/lib/syslog-ng/libconfgen.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libconfgen.so Reading symbols from /usr/local/lib/syslog-ng/libafsocket-tls.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libafsocket-tls.so Reading symbols from /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so Reading symbols from /usr/local/lib/libcrypto.so.8...done. Loaded symbols for /usr/local/lib/libcrypto.so.8 Reading symbols from /usr/local/lib/libssl.so.8...done. Loaded symbols for /usr/local/lib/libssl.so.8 Reading symbols from /usr/local/lib/syslog-ng/libaffile.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libaffile.so Reading symbols from /usr/local/lib/syslog-ng/libdbparser.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libdbparser.so Reading symbols from /usr/local/lib/syslog-ng/libafamqp.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libafamqp.so Reading symbols from /usr/local/lib/syslog-ng/libafuser.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libafuser.so Reading symbols from /usr/local/lib/syslog-ng/libsyslogformat.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libsyslogformat.so Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0000000801f9938c in thr_kill () from /lib/libc.so.7 [New Thread 80240a800 (LWP 109042/syslog-ng)] [New Thread 802409c00 (LWP 109041/syslog-ng)] [New Thread 802409800 (LWP 109040/syslog-ng)] [New Thread 802407400 (LWP 107992/syslog-ng)] (gdb) bt #0 0x0000000801f9938c in thr_kill () from /lib/libc.so.7 #1 0x000000080203b99b in abort () from /lib/libc.so.7 #2 0x00000008011223f4 in g_assertion_message () from /usr/local/lib/libglib-2.0.so.0 #3 0x00000008011229c2 in g_assertion_message_expr () from /usr/local/lib/libglib-2.0.so.0 #4 0x000000080084cc87 in log_msg_unref (self=Variable "self" is not available. ) at logmsg.c:1303 #5 0x0000000803d0836d in afamqp_worker_thread (arg=Variable "arg" is not available. ) at afamqp.c:479 #6 0x000000080085b6de in worker_thread_func (st=0x802421fd0) at misc.c:580 #7 0x0000000801124a65 in g_thread_proxy () from /usr/local/lib/libglib-2.0.so.0 #8 0x0000000801d220a4 in pthread_getprio () from /lib/libthr.so.3 #9 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffffbfe000

Hi, Any idea of why this is happening ? Do you need anything else ? Regards, Alexandre

On Thu, Sep 19, 2013 at 11:22 AM, Gergely Nagy <algernon@balabit.hu> wrote:

Alexandre Biancalana <biancalana@gmail.com> writes:

...

Any idea of why this is happening ? Do you need anything else ?

I have a few ideas, will try to reproduce the problem. At the moment, I don't need anything else. Although, if you could send your config (in private works too), that may help me a bit.

I forgot to mention that I'm using amqp destination. Going forward with the troubleshooting I've done some tests changing configuration and isolated that the core dump occur only when destination is amqp, even when I don't use patterndb the problem still occur. The config to reproduce the problem is very simple: @version:3.4 options { chain_hostnames(off); flush_lines(0); threaded(yes); }; # # sources # source src_local { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; destination d_amqp { amqp( vhost("/") host("127.0.0.1") port(5672) username("guest") # required option, no default password("guest") # required option, no default exchange("syslog") exchange_type("header") routing_key("") body("") persistent(yes) value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; log { source(src_local); destination(d_amqp); };

On Tue, Sep 24, 2013 at 9:57 PM, Alexandre Biancalana <biancalana@gmail.com>wrote:

On Thu, Sep 19, 2013 at 11:28 AM, Alexandre Biancalana < biancalana@gmail.com> wrote:

...

On Thu, Sep 19, 2013 at 11:22 AM, Gergely Nagy <algernon@balabit.hu>wrote:

...

Alexandre Biancalana <biancalana@gmail.com> writes:

...

Any idea of why this is happening ? Do you need anything else ?

I have a few ideas, will try to reproduce the problem. At the moment, I don't need anything else. Although, if you could send your config (in private works too), that may help me a bit.

I forgot to mention that I'm using amqp destination.

Going forward with the troubleshooting I've done some tests changing configuration and isolated that the core dump occur only when destination is amqp, even when I don't use patterndb the problem still occur.

The config to reproduce the problem is very simple:

@version:3.4

options { chain_hostnames(off); flush_lines(0); threaded(yes); };

# # sources # source src_local { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); };

destination d_amqp { amqp( vhost("/") host("127.0.0.1") port(5672) username("guest") # required option, no default password("guest") # required option, no default exchange("syslog") exchange_type("header") routing_key("") body("") persistent(yes) value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };

log { source(src_local); destination(d_amqp); };

I've made some tests and can't reproduce this "bug" using linux (centos 6.3)

applying the initial config with my patterndb and amqp destination it core dumped on linux too. [root@br.c1.sf05 logs]# gdb /usr/local/sbin/syslog-ng GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done. (gdb) -f /usr/local/etc/syslog-ng.conf -d Undefined command: "-f". Try "help". (gdb) run -f /usr/local/etc/syslog-ng.conf -d Starting program: /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -d [Thread debugging using libthread_db enabled] Reading path for candidate modules; path='/usr/local/lib/syslog-ng' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='confgen.so', module='confgen' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='dbparser.so', module='dbparser' Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afamqp.so', module='afamqp' Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='basicfuncs.so', module='basicfuncs' Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afuser.so', module='afuser' Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-notls.so', module='afsocket-notls' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='network', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='network', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket.so', module='afsocket' Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='affile.so', module='affile' Registering candidate plugin; module='affile', context='source', name='file', preference='0' Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' Registering candidate plugin; module='affile', context='destination', name='file', preference='0' Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='csvparser.so', module='csvparser' Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afprog.so', module='afprog' Registering candidate plugin; module='afprog', context='source', name='program', preference='0' Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslog-ng-crypto.so', module='syslog-ng-crypto' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslogformat.so', module='syslogformat' Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afmongodb.so', module='afmongodb' Registering candidate plugin; module='afmongodb', context='destination', name='mongodb', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='system-source.so', module='system-source' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-tls.so', module='afsocket-tls' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='network', preference='100' Compiling #unnamed sequence [log] at [/usr/local/etc/syslog-ng.conf:4] Compiling httpd_error_log reference [source] at [/usr/local/etc/syslog-ng.conf:4] Compiling httpd_error_log sequence [source] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling pattern_db reference [parser] at [/usr/local/etc/syslog-ng.conf:4] Compiling pattern_db sequence [parser] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling d_amqp reference [destination] at [/usr/local/etc/syslog-ng.conf:4] Compiling d_amqp sequence [destination] at [/usr/local/etc/syslog-ng.conf:2] Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:2] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:2] Log pattern database reloaded; file='/home/ale/httpd.xml', version='3', pub_date='2013-09-12' [New Thread 0x7ffff7fe3700 (LWP 11172)] Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.4.3' Worker thread started; driver='d_amqp#0' Connecting to AMQP succeeded; driver='d_amqp#0' Incoming log entry; line='[2013-09-01 03:49:03.051560] [core:notice] [pid 17230:tid 139776559662848] AH00094: Command line: \'/opt/apps/httpd/sbin/httpd -D NO_DETACH\'' patterndb rule matches; rule_id='52a31c61-1bff-11e3-919d-ca66d2f45ab4' Advancing patterndb current time because of an incoming message; utc='1380159639' Message parsing complete; result='1' Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7fe3700 (LWP 11172)] 0x00007ffff7b903d5 in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so Missing separate debuginfos, use: debuginfo-install syslog-ng-3.4.3-1.x86_64 (gdb) bt #0 0x00007ffff7b903d5 in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so #1 0x00007ffff7b8f868 in nv_table_foreach_entry () from /usr/local/lib/ libsyslog-ng-3.4.3.so #2 0x00007ffff7b8f926 in nv_table_foreach () from /usr/local/lib/ libsyslog-ng-3.4.3.so #3 0x00007ffff7b9af3f in value_pairs_foreach_sorted () from /usr/local/lib/ libsyslog-ng-3.4.3.so #4 0x00007ffff4069e6a in ?? () from /usr/local/lib/syslog-ng/libafamqp.so #5 0x00007ffff406a9a2 in ?? () from /usr/local/lib/syslog-ng/libafamqp.so #6 0x00007ffff7b8e63b in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so #7 0x00007ffff70a3004 in ?? () from /lib64/libglib-2.0.so.0 #8 0x00007ffff6a23851 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ffff677190d in clone () from /lib64/libc.so.6 Any ideas ?

Gergely Nagy <algernon@balabit.hu> writes:

Alexandre Biancalana <biancalana@gmail.com> writes:

...

Incoming log entry; line='[2013-09-01 03:49:03.051560] [core:notice] [pid 17230:tid 139776559662848] AH00094: Command line: \'/opt/apps/httpd/sbin/httpd -D NO_DETACH\'' patterndb rule matches; rule_id='52a31c61-1bff-11e3-919d-ca66d2f45ab4' Advancing patterndb current time because of an incoming message; utc='1380159639' Message parsing complete; result='1'

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7fe3700 (LWP 11172)] 0x00007ffff7b903d5 in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so Missing separate debuginfos, use: debuginfo-install syslog-ng-3.4.3-1.x86_64 (gdb) bt #0 0x00007ffff7b903d5 in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so #1 0x00007ffff7b8f868 in nv_table_foreach_entry () from /usr/local/lib/ libsyslog-ng-3.4.3.so #2 0x00007ffff7b8f926 in nv_table_foreach () from /usr/local/lib/ libsyslog-ng-3.4.3.so #3 0x00007ffff7b9af3f in value_pairs_foreach_sorted () from /usr/local/lib/ libsyslog-ng-3.4.3.so #4 0x00007ffff4069e6a in ?? () from /usr/local/lib/syslog-ng/libafamqp.so #5 0x00007ffff406a9a2 in ?? () from /usr/local/lib/syslog-ng/libafamqp.so #6 0x00007ffff7b8e63b in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so #7 0x00007ffff70a3004 in ?? () from /lib64/libglib-2.0.so.0 #8 0x00007ffff6a23851 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ffff677190d in clone () from /lib64/libc.so.6

Any ideas ?

I have a reasonable guess why this happens, but didn't get around to try and reproduce it yet. I hope I will be able to spend some time on this issue this weekend.

Reproduced, I should have a fix soon. Thanks for the report and the config! -- |8]

On Thu, Sep 26, 2013 at 8:00 AM, Gergely Nagy <algernon@balabit.hu> wrote:

I have pushed a fix to 3.4 master[1], it was a stupid copy & paste mistake.

[1]: https://github.com/balabit/syslog-ng-3.4/commit/d24e7add4bdc9143025f6165d4c8...

Hi Gergely ! Thank you for your time and the patch, but it's still happening... # gdb ./syslog-ng/.libs/syslog-ng GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (gdb) run -d -f /usr/local/etc/syslog-ng.conf Starting program: /usr/ports/sysutils/syslog-ng/work/syslog-ng-3.4.3/syslog-ng/.libs/syslog-ng -d -f /usr/local/etc/syslog-ng.conf [New LWP 100524] [New Thread 802407400 (LWP 100524/syslog-ng)] nanosleep() is not accurate enough to introduce minor stalls on the reader side, multi-threaded performance may be affected; Reading path for candidate modules; path='/usr/local/lib/syslog-ng' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afamqp.so', module='afamqp' Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='affile.so', module='affile' Registering candidate plugin; module='affile', context='source', name='file', preference='0' Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' Registering candidate plugin; module='affile', context='destination', name='file', preference='0' Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afmongodb.so', module='afmongodb' Registering candidate plugin; module='afmongodb', context='destination', name='mongodb', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afprog.so', module='afprog' Registering candidate plugin; module='afprog', context='source', name='program', preference='0' Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-notls.so', module='afsocket-notls' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='network', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='network', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-tls.so', module='afsocket-tls' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='network', preference='100' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket.so', module='afsocket' Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afuser.so', module='afuser' Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='basicfuncs.so', module='basicfuncs' Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='confgen.so', module='confgen' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='csvparser.so', module='csvparser' Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='dbparser.so', module='dbparser' Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslog-ng-crypto.so', module='syslog-ng-crypto' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslogformat.so', module='syslogformat' Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='system-source.so', module='system-source' Compiling #unnamed sequence [log] at [/usr/local/etc/syslog-ng.conf:3] Compiling src_local reference [source] at [/usr/local/etc/syslog-ng.conf:3] Compiling src_local sequence [source] at [/usr/local/etc/syslog-ng.conf:8] Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:8] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:8] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:9] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling d_amqp reference [destination] at [/usr/local/etc/syslog-ng.conf:3] Compiling d_amqp sequence [destination] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] [New Thread 802409000 (LWP 109094/syslog-ng)] Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.4.3' Incoming log entry; line='<6>pid 85554 (syslog-ng), uid 0: exited on signal 6 (core dumped)' Worker thread started; driver='d_amqp#0' Connecting to AMQP succeeded; driver='d_amqp#0' ** ERROR:logmsg.c:1303:log_msg_unref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) Program received signal SIGABRT, Aborted. [Switching to Thread 802409000 (LWP 109094/syslog-ng)] 0x0000000801f9938c in thr_kill () from /lib/libc.so.7 (gdb) bt #0 0x0000000801f9938c in thr_kill () from /lib/libc.so.7 #1 0x000000080203b99b in abort () from /lib/libc.so.7 #2 0x00000008011223f4 in g_assertion_message () from /usr/local/lib/libglib-2.0.so.0 #3 0x00000008011229c2 in g_assertion_message_expr () from /usr/local/lib/libglib-2.0.so.0 #4 0x000000080084cc87 in log_msg_unref (self=Variable "self" is not available. ) at logmsg.c:1303 #5 0x000000080368736d in afamqp_worker_thread (arg=Variable "arg" is not available. ) at afamqp.c:479 #6 0x000000080085b6de in worker_thread_func (st=0x8024259a0) at misc.c:580 #7 0x0000000801124a65 in g_thread_proxy () from /usr/local/lib/libglib-2.0.so.0 #8 0x0000000801d220a4 in pthread_getprio () from /lib/libthr.so.3 #9 0x0000000000000000 in ?? () Error accessing memory address 0x7fffffbfe000: Bad address. (gdb)

On Thu, Sep 26, 2013 at 3:34 PM, Alexandre Biancalana <biancalana@gmail.com>wrote:

Testing on linux it goes a little further but also crashes:

# gdb syslog-ng/.libs/syslog-ng

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /tmp/syslog-ng-3.4/syslog-ng/.libs/syslog-ng...done.

(gdb) run -d -f /usr/local/etc/syslog-ng.conf Starting program: /tmp/syslog-ng-3.4/syslog-ng/.libs/syslog-ng -d -f /usr/local/etc/syslog-ng.conf

[Thread debugging using libthread_db enabled] Reading path for candidate modules; path='/usr/local/lib/syslog-ng' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='confgen.so', module='confgen' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='dbparser.so', module='dbparser' Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afamqp.so', module='afamqp' Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='basicfuncs.so', module='basicfuncs' Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afuser.so', module='afuser' Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-notls.so', module='afsocket-notls' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='network', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='network', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket.so', module='afsocket' Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='affile.so', module='affile' Registering candidate plugin; module='affile', context='source', name='file', preference='0' Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' Registering candidate plugin; module='affile', context='destination', name='file', preference='0' Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='csvparser.so', module='csvparser' Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afprog.so', module='afprog' Registering candidate plugin; module='afprog', context='source', name='program', preference='0' Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslog-ng-crypto.so', module='syslog-ng-crypto' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslogformat.so', module='syslogformat' Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afmongodb.so', module='afmongodb' Registering candidate plugin; module='afmongodb', context='destination', name='mongodb', preference='0' Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='system-source.so', module='system-source'

Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-tls.so', module='afsocket-tls' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='network', preference='100' Compiling #unnamed sequence [log] at [/usr/local/etc/syslog-ng.conf:4] Compiling httpd_error_log reference [source] at [/usr/local/etc/syslog-ng.conf:4] Compiling httpd_error_log sequence [source] at [/usr/local/etc/syslog-ng.conf:1]

Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling pattern_db reference [parser] at [/usr/local/etc/syslog-ng.conf:4] Compiling pattern_db sequence [parser] at [/usr/local/etc/syslog-ng.conf:1]

Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1] Compiling d_amqp reference [destination] at [/usr/local/etc/syslog-ng.conf:4] Compiling d_amqp sequence [destination] at [/usr/local/etc/syslog-ng.conf:2] Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:2] Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:2] Log pattern database reloaded; file='/home/ale/httpd.xml', version='3', pub_date='2013-09-12' [New Thread 0x7ffff7fe3700 (LWP 3098)]

Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.4.3' Worker thread started; driver='d_amqp#0' Connecting to AMQP succeeded; driver='d_amqp#0' Incoming log entry; line='[2013-09-07 21:37:18.103339] [-:error] [pid 29919:tid 139776059467520] [client 10.10.10.10] ModSecurity: Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file "/opt/apps/httpd/conf/owasp-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "40"] [id "981205"] [msg "Outbound Anomaly Score Exceeded (score 4): The application is not available"] [hostname "www.xxx.com"] [uri "/wp/wp-content/themes/musicpro/js/solo-jplayer.min.js"] [unique_id "UiucjcCoALkAAHTfttcAAACM"] [2013-09-07 21:37:18.131577] [-:error] [pid 29777:tid 139776257115904] [client 10.10.10.10] ModSecurity: Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file "/opt/apps/httpd/conf/owasp-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "40"] [id "981205"] [msg "Outbound Anomaly Score Exceeded (score 4): The application is not available"] [hostname "www.xxx.com"] [uri "/wp/wp-includes/js/comment-reply.js"] [unique_id "UiucjsCoALkAAHRR1KoAAAAA"]' patterndb rule matches; rule_id='6bcd01cd-1bff-11e3-919d-ca66d2f45ab4' Advancing patterndb current time because of an incoming message; utc='1380220111'

Message parsing complete; result='1' Incoming log entry; line='[2013-09-07 21:37:18.135037] [-:error] [pid 29777:tid 139776153876224] [client 10.10.10.10] ModSecurity: Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file "/opt/apps/httpd/conf/owasp-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "40"] [id "981205"] [msg "Outbound Anomaly Score Exceeded (score 4): The application is not available"] [hostname "www.xxx.com"] [uri "/wp/wp-content/themes/musicpro/js/tooltipsy.js"] [unique_id "UiucjsCoALkAAHRR1KgAAAAD"]' patterndb rule matches; rule_id='6bcd01cd-1bff-11e3-919d-ca66d2f45ab4' Advancing patterndb current time because of an incoming message; utc='1380220111'

Message parsing complete; result='1' Incoming log entry; line='[2013-09-07 21:37:18.136092] [-:error] [pid 29777:tid 139776132896512] [client 10.10.10.10] ModSecurity: Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file "/opt/apps/httpd/conf/owasp-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "40"] [id "981205"] [msg "Outbound Anomaly Score Exceeded (score 4): The application is not available"] [hostname "www.xxx.com"] [uri "/wp/wp-content/themes/musicpro/js/custom.js"] [unique_id "UiucjsCoALkAAHRR1KsAAAAF"]' patterndb rule matches; rule_id='6bcd01cd-1bff-11e3-919d-ca66d2f45ab4' Advancing patterndb current time because of an incoming message; utc='1380220111'

Message parsing complete; result='1' Incoming log entry; line='[2013-09-07 21:37:18.138618] [-:error] [pid 29777:tid 139776143386368] [client 10.10.10.10] ModSecurity: Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file "/opt/apps/httpd/conf/owasp-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "40"] [id "981205"] [msg "Outbound Anomaly Score Exceeded (score 4): The application is not available"] [hostname "www.xxx.com"] [uri "/wp/wp-content/themes/musicpro/js/izotope.js"] [unique_id "UiucjsCoALkAAHRR1KkAAAAE"]'

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7fe3700 (LWP 3098)] 0x00007ffff7b8de6e in msg_set_context () from /usr/local/lib/ libsyslog-ng-3.4.3.so Missing separate debuginfos, use: debuginfo-install glib2-2.22.5-7.el6.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libnet-1.1.5-1.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 openssl-1.0.0-27.el6.x86_64 pcre-7.8-6.el6.x86_64 syslog-ng-3.4.3-1.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007ffff7b8de6e in msg_set_context () from /usr/local/lib/ libsyslog-ng-3.4.3.so #1 0x00007ffff3e3e995 in ?? () from /usr/local/lib/syslog-ng/libafamqp.so #2 0x00007ffff7b8e63b in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so #3 0x00007ffff70a3004 in ?? () from /lib64/libglib-2.0.so.0 #4 0x00007ffff67f7851 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff654590d in clone () from /lib64/libc.so.6 (gdb)

Hi List, Is there anything else that I can do to help to track/solve this ?

On Thu, Oct 10, 2013 at 7:08 AM, Gergely Nagy <algernon@balabit.hu> wrote:

Alexandre Biancalana <biancalana@gmail.com> writes:

...

On Thu, Sep 26, 2013 at 8:00 AM, Gergely Nagy <algernon@balabit.hu> wrote:

...

I have pushed a fix to 3.4 master[1], it was a stupid copy & paste mistake.

[1]:

https://github.com/balabit/syslog-ng-3.4/commit/d24e7add4bdc9143025f6165d4c8...

...

Hi Gergely !

Thank you for your time and the patch, but it's still happening...

I can't reproduce it, neither on FreeBSD, nor GNU/Linux, using 3.4 master. Valgrind doesn't see anything suspicios, either. Could you perhaps run a valgrind on it too?

I gave a try to 3.5.0beta3 and can't reproduce it anymore. Thanks, Alexandre