Hi, I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment: - a new laptop - a new version of vmware workstation - a new version of glibc and openjdk inside the vm-s (I always start with a security update...) The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7. Have you ever seen anything similar? Bye, CzP This one is from CentOS7 where I already tried rolling back glibc and openjdk: [2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped) This one is from openSUSE, where I had gdb installed... linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb) Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
Hi, Are you using recent kernels? The enhanced stack protection fixing CVE-2017-1000364 causes interoperability issues for certain Java apps... Regards, Sandor On 06/23/2017 02:12 PM, Czanik, Péter wrote:
Hi,
I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment: - a new laptop - a new version of vmware workstation - a new version of glibc and openjdk inside the vm-s (I always start with a security update...)
The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7.
Have you ever seen anything similar?
Bye, CzP
This one is from CentOS7 where I already tried rolling back glibc and openjdk:
[2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped)
This one is from openSUSE, where I had gdb installed...
linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb)
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Yes. Going back to an older kernel resolved (worked around...) the problem. Thank you and have a nice weekend! CzP Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Fri, Jun 23, 2017 at 2:18 PM, Sandor Geller <sandor.geller@ericsson.com> wrote:
Hi,
Are you using recent kernels? The enhanced stack protection fixing CVE-2017-1000364 causes interoperability issues for certain Java apps...
Regards,
Sandor
On 06/23/2017 02:12 PM, Czanik, Péter wrote:
Hi,
I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment: - a new laptop - a new version of vmware workstation - a new version of glibc and openjdk inside the vm-s (I always start with a security update...)
The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7.
Have you ever seen anything similar?
Bye, CzP
This one is from CentOS7 where I already tried rolling back glibc and openjdk:
[2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped)
This one is from openSUSE, where I had gdb installed...
linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0. 46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl. html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/ vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/ vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/ openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/ linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/ openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb)
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Exact same situation running kernel 4.11.6-1.el7.elrepo.x86_64. Unfortunately my Linux/Java skills aren’t particularly impressive – is there a feasible workaround to maintain my present kernel without having to go back to 3.10.x at all? Thanks! From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Czanik, Péter Sent: 23 June 2017 13:43 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] java destinations Hi, Yes. Going back to an older kernel resolved (worked around...) the problem. Thank you and have a nice weekend! CzP Peter Czanik (CzP) <peter.czanik@balabit.com<mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd T: +44 20 7334 5483 Email: Damian.Bell@clarksons.com<mailto:Damian.Bell@clarksons.com> On Fri, Jun 23, 2017 at 2:18 PM, Sandor Geller <sandor.geller@ericsson.com<mailto:sandor.geller@ericsson.com>> wrote: Hi, Are you using recent kernels? The enhanced stack protection fixing CVE-2017-1000364 causes interoperability issues for certain Java apps... Regards, Sandor On 06/23/2017 02:12 PM, Czanik, Péter wrote: Hi, I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment: - a new laptop - a new version of vmware workstation - a new version of glibc and openjdk inside the vm-s (I always start with a security update...) The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7. Have you ever seen anything similar? Bye, CzP This one is from CentOS7 where I already tried rolling back glibc and openjdk: [2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped) This one is from openSUSE, where I had gdb installed... linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb) Peter Czanik (CzP) <peter.czanik@balabit.com<mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________
Hi, If you've got control how JVM gets started then you could try limiting the size of the stack (-Xss option) to a large enough value - so this is a bit of iterative process... Regards, Sandor On 06/23/2017 04:54 PM, Damian Bell wrote:
Exact same situation running kernel 4.11.6-1.el7.elrepo.x86_64. Unfortunately my Linux/Java skills aren’t particularly impressive – is there a feasible workaround to maintain my present kernel without having to go back to 3.10.x at all?
Thanks!
*From:*syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Czanik, Péter *Sent:* 23 June 2017 13:43 *To:* Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] java destinations
Hi,
Yes. Going back to an older kernel resolved (worked around...) the problem.
Thank you and have a nice weekend!
CzP
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
Damian Bell Infrastructure Engineer| Support| H Clarkson & Co Ltd T: +44 20 7334 5483 Email: Damian.Bell@clarksons.com <mailto:Damian.Bell@clarksons.com>
On Fri, Jun 23, 2017 at 2:18 PM, Sandor Geller <sandor.geller@ericsson.com <mailto:sandor.geller@ericsson.com>> wrote:
Hi,
Are you using recent kernels? The enhanced stack protection fixing CVE-2017-1000364 causes interoperability issues for certain Java apps...
Regards,
Sandor
On 06/23/2017 02:12 PM, Czanik, Péter wrote:
Hi,
I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment:
- a new laptop
- a new version of vmware workstation
- a new version of glibc and openjdk inside the vm-s (I always start with a security update...)
The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7.
Have you ever seen anything similar?
Bye,
CzP
This one is from CentOS7 where I already tried rolling back glibc and openjdk:
[2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped)
This one is from openSUSE, where I had gdb installed...
linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb)
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
______________________________________________________________________________
Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ:http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
------------------------------------------------------------------------ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence.
Emails may be monitored.
Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure <http://www.clarksons.com/disclosure/> ------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, Jun 23, 2017 at 05:07:29PM +0200, Sandor Geller wrote:
If you've got control how JVM gets started then you could try limiting the size of the stack (-Xss option) to a large enough value - so this is a bit of iterative process...
Thanks Sandor. The problem is that syslog-ng is using JNI and I have no idea if it's even possible to pass JVM params in this case.
Hi,
Thanks Sandor. The problem is that syslog-ng is using JNI and I have no idea if it's even possible to pass JVM params in this case.
Currently this is not possible, syslog-ng has a predefined list of JVM parameters. We might make this configurable in the future. -- László Várady
On Sun, Jun 25, 2017 at 02:25:00AM +0200, Várady, László wrote:
Hi,
Thanks Sandor. The problem is that syslog-ng is using JNI and I have no idea if it's even possible to pass JVM params in this case.
Currently this is not possible, syslog-ng has a predefined list of JVM parameters. We might make this configurable in the future.
Okay, thanks. That being said, if it successfully addresses the issue introduced by the CVE-2017-1000364 kernel fix, I believe this should make it to a bugfix release, or of course if there is a better way to fix it.
On Fri, Jun 23, 2017 at 05:07:29PM +0200, Sandor Geller wrote:
If you've got control how JVM gets started then you could try limiting the size of the stack (-Xss option) to a large enough value - so this is a bit of iterative process...
Apparently it can be set here: modules/java/native/java_machine.c: self->options[0].optionString = g_strdup_printf( "-Djava.class.path=%s", self->class_path->str); self->options[1].optionString = g_strdup_printf( "-Djava.library.path=%s", resolvedConfigurablePaths.initial_module_path); self->options[2].optionString = g_strdup("-Xrs"); I'll give it a try and report back
Hi again, FTR I created a gh issue: https://github.com/balabit/syslog-ng/issues/1562 let's add additional info there
participants (5)
-
Czanik, Péter
-
Damian Bell
-
Fabien Wernli
-
Sandor Geller
-
Várady, László