Hi, I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment: - a new laptop - a new version of vmware workstation - a new version of glibc and openjdk inside the vm-s (I always start with a security update...) The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7. Have you ever seen anything similar? Bye, CzP This one is from CentOS7 where I already tried rolling back glibc and openjdk: [2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped) This one is from openSUSE, where I had gdb installed... linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb) Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik