Hi,

I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment:

- a new laptop

- a new version of vmware workstation

- a new version of glibc and openjdk inside the vm-s (I always start with a security update...)

The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7.

Have you ever seen anything similar?

Bye,

CzP

This one is from CentOS7 where I already tried rolling back glibc and openjdk:

[2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3]
[2017-06-23T13:25:22.816968] Seeking the journal to the start position;
[2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC'
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40
#
# JRE version: (8.0_121-b13) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1651.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)

This one is from openSUSE, where I had gdb installed...

linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng
GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done.
[New LWP 2015]
[New LWP 2016]
[New LWP 2017]
[New LWP 2018]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `syslog-ng -Fvde'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673
673    /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
[Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))]
Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64
(gdb) bt full
#0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673
        sp = 0x7ffe43f7a120 ""
        size = 37153
        p = 0x7ffe43f70ff0 ""
#1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686
        mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
        old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4,
            140730038723200, 140680140758410, 140730038723208, 18446744069414584320}}
#2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1)
    at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330
        addr = <optimized out>
        uc = 0x7ffe43f7a400
        thread = 0x19e9000
        stub = 0x0
        newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}}
        err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4,
          _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8
     <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3,
          _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48,
          _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false,
          static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>},
              _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false},
          static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0,
              _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}}
        t = 0x19e9000
        shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000}
        vmthread = 0x0
        pc = <optimized out>
#3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354
        orig_errno = 2
#4 <signal handler called>
No symbol table info available.
#5 0x00007ff29453dc26 in ?? ()
No symbol table info available.
#6 0x00007ffe43f7a990 in ?? ()
No symbol table info available.
#7 0x00007ff28a7d0d50 in ?? ()
No symbol table info available.
#8 0x00007ffe43f7a9d8 in ?? ()
No symbol table info available.
#9 0x00007ff28a892ff8 in ?? ()
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

Peter Czanik (CzP) <peter.czanik@balabit.com>
Balabit / syslog-ng upstream
https://www.balabit.com/blog/author/peterczanik/
https://twitter.com/PCzanik