On Tue, Sep 24, 2013 at 9:57 PM, Alexandre Biancalana <biancalana@gmail.com> wrote:



On Thu, Sep 19, 2013 at 11:28 AM, Alexandre Biancalana <biancalana@gmail.com> wrote:



On Thu, Sep 19, 2013 at 11:22 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Alexandre Biancalana <biancalana@gmail.com> writes:

>  Any idea of why this is happening ?  Do you need anything else ?

I have a few ideas, will try to reproduce the problem. At the moment, I
don't need anything else. Although, if you could send your config (in
private works too), that may help me a bit.


I forgot to mention that I'm using amqp destination.

Going forward with the troubleshooting I've done some tests changing configuration and isolated that the core dump occur only when destination is amqp, even when I don't use patterndb the problem still occur.

The config to reproduce the problem is very simple:

@version:3.4

options { chain_hostnames(off); flush_lines(0); threaded(yes); };

#
# sources
#
source src_local { unix-dgram("/var/run/log");
                unix-dgram("/var/run/logpriv" perm(0600));
                internal();
                file("/dev/klog"); };


destination d_amqp {
    amqp(
        vhost("/")
        host("127.0.0.1")
        port(5672)
        username("guest") # required option, no default
        password("guest") # required option, no default
        exchange("syslog")
        exchange_type("header")
        routing_key("")
        body("")
        persistent(yes)
       value-pairs(
            scope("selected-macros" "nv-pairs" "sdata")
        )
    );
};

log { source(src_local); destination(d_amqp); };




I've made some tests and can't reproduce this "bug" using linux (centos 6.3)


applying the initial config with my patterndb and amqp destination it core dumped on linux too.



[root@br.c1.sf05 logs]# gdb /usr/local/sbin/syslog-ng
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done.
(gdb) -f /usr/local/etc/syslog-ng.conf -d
Undefined command: "-f".  Try "help".
(gdb) run -f /usr/local/etc/syslog-ng.conf -d
Starting program: /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -d
[Thread debugging using libthread_db enabled]
Reading path for candidate modules; path='/usr/local/lib/syslog-ng'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='confgen.so', module='confgen'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='dbparser.so', module='dbparser'
Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afamqp.so', module='afamqp'
Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='basicfuncs.so', module='basicfuncs'
Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afuser.so', module='afuser'
Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-notls.so', module='afsocket-notls'
Registering candidate plugin; module='afsocket-notls', context='source', name='unix-stream', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-stream', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='unix-dgram', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-dgram', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='tcp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='tcp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='udp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='udp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='udp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='udp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='syslog', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='syslog', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='network', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='network', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket.so', module='afsocket'
Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='network', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='affile.so', module='affile'
Registering candidate plugin; module='affile', context='source', name='file', preference='0'
Registering candidate plugin; module='affile', context='source', name='pipe', preference='0'
Registering candidate plugin; module='affile', context='destination', name='file', preference='0'
Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='csvparser.so', module='csvparser'
Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afprog.so', module='afprog'
Registering candidate plugin; module='afprog', context='source', name='program', preference='0'
Registering candidate plugin; module='afprog', context='destination', name='program', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslog-ng-crypto.so', module='syslog-ng-crypto'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='syslogformat.so', module='syslogformat'
Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0'
Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afmongodb.so', module='afmongodb'
Registering candidate plugin; module='afmongodb', context='destination', name='mongodb', preference='0'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='system-source.so', module='system-source'
Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsocket-tls.so', module='afsocket-tls'
Registering candidate plugin; module='afsocket-tls', context='source', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='tcp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='udp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='udp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='udp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='udp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='syslog', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='syslog', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='network', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='network', preference='100'
Compiling #unnamed sequence [log] at [/usr/local/etc/syslog-ng.conf:4]
  Compiling httpd_error_log reference [source] at [/usr/local/etc/syslog-ng.conf:4]
    Compiling httpd_error_log sequence [source] at [/usr/local/etc/syslog-ng.conf:1]
      Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:1]
        Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1]
  Compiling pattern_db reference [parser] at [/usr/local/etc/syslog-ng.conf:4]
    Compiling pattern_db sequence [parser] at [/usr/local/etc/syslog-ng.conf:1]
      Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:1]
  Compiling d_amqp reference [destination] at [/usr/local/etc/syslog-ng.conf:4]
    Compiling d_amqp sequence [destination] at [/usr/local/etc/syslog-ng.conf:2]
      Compiling #unnamed junction [log] at [/usr/local/etc/syslog-ng.conf:2]
        Compiling #unnamed single [log] at [/usr/local/etc/syslog-ng.conf:2]
Log pattern database reloaded; file='/home/ale/httpd.xml', version='3', pub_date='2013-09-12'
[New Thread 0x7ffff7fe3700 (LWP 11172)]
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.4.3'
Worker thread started; driver='d_amqp#0'
Connecting to AMQP succeeded; driver='d_amqp#0'
Incoming log entry; line='[2013-09-01 03:49:03.051560] [core:notice] [pid 17230:tid 139776559662848] AH00094: Command line: \'/opt/apps/httpd/sbin/httpd -D NO_DETACH\''
patterndb rule matches; rule_id='52a31c61-1bff-11e3-919d-ca66d2f45ab4'
Advancing patterndb current time because of an incoming message; utc='1380159639'
Message parsing complete; result='1'

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe3700 (LWP 11172)]
0x00007ffff7b903d5 in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so
Missing separate debuginfos, use: debuginfo-install syslog-ng-3.4.3-1.x86_64
(gdb) bt
#0  0x00007ffff7b903d5 in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so
#1  0x00007ffff7b8f868 in nv_table_foreach_entry () from /usr/local/lib/libsyslog-ng-3.4.3.so
#2  0x00007ffff7b8f926 in nv_table_foreach () from /usr/local/lib/libsyslog-ng-3.4.3.so
#3  0x00007ffff7b9af3f in value_pairs_foreach_sorted () from /usr/local/lib/libsyslog-ng-3.4.3.so
#4  0x00007ffff4069e6a in ?? () from /usr/local/lib/syslog-ng/libafamqp.so
#5  0x00007ffff406a9a2 in ?? () from /usr/local/lib/syslog-ng/libafamqp.so
#6  0x00007ffff7b8e63b in ?? () from /usr/local/lib/libsyslog-ng-3.4.3.so
#7  0x00007ffff70a3004 in ?? () from /lib64/libglib-2.0.so.0
#8  0x00007ffff6a23851 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff677190d in clone () from /lib64/libc.so.6


Any ideas ?