syslog-ng 3.13 key-value parser crashes and aborts
Hi, I was upgrading syslog-ng from 3.12 to 3.13 using the prebuilt unofficial RPMs. I was testing my config on 3.13 and ran into the following issue where the key-value parsing aborts.This appears to be a regression in 3.13, the abort never happened in 3.12. If I remove the kv_parser from my config, there is no issue. Here's the end of the debug and verbose output I get. Notice the garbage values in the key-value names parsed from the message. [2018-01-25T17:33:08.924895] Filter rule evaluation result; msg='0x7fc0a807b370', result='match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.924919] Setting value; msg='0x7fc0a80612f0', name='@\x15\x01pÀ', value='2017-11-21 19:11:24.817041' [2018-01-25T17:33:08.924930] Setting value; msg='0x7fc0a80612f0', name='', value='debug' [2018-01-25T17:33:08.924936] Setting value; msg='0x7fc0a80612f0', name='f_compliant_hosts', value='[robotnats kafka transport prepared]' [2018-01-25T17:33:08.924944] Setting value; msg='0x7fc0a80612f0', name='PK\x06¨À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924952] Setting value; msg='0x7fc0a80612f0', name='', value='nats://nats:4222' [2018-01-25T17:33:08.924957] Setting value; msg='0x7fc0a80612f0', name='', value='[robot-kafka:9092]' [2018-01-25T17:33:08.924963] Setting value; msg='0x7fc0a80612f0', name='kv_kafka', value='robotnats_kafka' [2018-01-25T17:33:08.924969] Setting value; msg='0x7fc0a80612f0', name=' »\x08¬À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924975] Message parsing complete; result='1', rule='p_kv', location='/etc/syslog-ng/syslog-ng.conf:75:5' [2018-01-25T17:33:08.924998] Incoming log entry; line='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925030] Setting value; msg='0x7fc0a807bd60', name='MESSAGE', value='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925040] Setting value; msg='0x7fc0a807bd60', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf' [2018-01-25T17:33:08.925046] Setting value; msg='0x7fc0a807bd60', name='HOST', value='syslog-ng-logging-1040168119-qsmrf/syslog-ng-logging-1040168119-qsmrf' [2018-01-25T17:33:08.925053] Setting value; msg='0x7fc0a807bd60', name='FILE_NAME', value='/mnt/logfs/neo4j-topo-svc/1/servicetracking_stdout.log' [2018-01-25T17:33:08.925057] Setting value; msg='0x7fc0a807bd60', name='SOURCE', value='s_file_stdout' [2018-01-25T17:33:08.925062] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5' [2018-01-25T17:33:08.824606] Incoming log entry; line='time="2017-11-21 19:10:56.399341" level=debug msg="[section not required]" section=etcd tag=ROBOT_serviceTracking_robot-topo-svc-2107321943-48rnw' [2018-01-25T17:33:09.022629] Filter rule evaluation begins; msg='0x7fc0a807bd60', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.824700] Setting value; msg='0x7fc0a409b810', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf' # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # ... [2018-01-25T17:33:09.022911] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5' [2018-01-25T17:33:09.022916] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022922] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022927] Filter rule evaluation result; msg='0x7fc08c0f5550', result='not-match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022932] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' [2018-01-25T17:33:09.022937] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022942] Filter node evaluation result; msg='0x7fc08c0f5550', result='match', type='filter(f_compliant_hosts)' [2018-01-25T17:33:09.022947] Filter rule evaluation result; msg='0x7fc08c0f5550', result='match', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' Aborted Is this a known issue with the key-value parser in 3.13? Regards, Don
Hello Don! We did have a regression in kv-parser, it has been fixed recently, merged to upstream and it will be in the upcoming syslog-ng release 3.14.1. You can find the patch here: https://github.com/balabit/syslog-ng/commit/aba5d41c1f092981501e75f009ffffee... Best Regards, Gabor On Thu, Jan 25, 2018 at 6:37 PM, Don C <lawsuit_loser@yahoo.com> wrote:
Hi,
I was upgrading syslog-ng from 3.12 to 3.13 using the prebuilt unofficial RPMs. I was testing my config on 3.13 and ran into the following issue where the key-value parsing aborts. This appears to be a regression in 3.13, the abort never happened in 3.12.
If I remove the kv_parser from my config, there is no issue.
Here's the end of the debug and verbose output I get. Notice the garbage values in the key-value names parsed from the message.
[2018-01-25T17:33:08.924895] Filter rule evaluation result; msg='0x7fc0a807b370', result='match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.924919] Setting value; msg='0x7fc0a80612f0', name='@\x15\x01pÀ', value='2017-11-21 19:11:24.817041'
[2018-01-25T17:33:08.924930] Setting value; msg='0x7fc0a80612f0', name='', value='debug'
[2018-01-25T17:33:08.924936] Setting value; msg='0x7fc0a80612f0', name='f_compliant_hosts', value='[robotnats kafka transport prepared]' [2018-01-25T17:33:08.924944] Setting value; msg='0x7fc0a80612f0', name='PK\x06¨À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760- 1sdn1' [2018-01-25T17:33:08.924952] Setting value; msg='0x7fc0a80612f0', name='', value='nats://nats:4222'
[2018-01-25T17:33:08.924957] Setting value; msg='0x7fc0a80612f0', name='', value='[robot-kafka:9092]'
[2018-01-25T17:33:08.924963] Setting value; msg='0x7fc0a80612f0', name='kv_kafka', value='robotnats_kafka'
[2018-01-25T17:33:08.924969] Setting value; msg='0x7fc0a80612f0', name=' »\x08¬À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760- 1sdn1' [2018-01-25T17:33:08.924975] Message parsing complete; result='1', rule='p_kv', location='/etc/syslog-ng/syslog-ng.conf:75:5'
[2018-01-25T17:33:08.924998] Incoming log entry; line='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925030] Setting value; msg='0x7fc0a807bd60', name='MESSAGE', value='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_ neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925040] Setting value; msg='0x7fc0a807bd60', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf'
[2018-01-25T17:33:08.925046] Setting value; msg='0x7fc0a807bd60', name='HOST', value='syslog-ng-logging-1040168119-qsmrf/syslog-ng- logging-1040168119-qsmrf' [2018-01-25T17:33:08.925053] Setting value; msg='0x7fc0a807bd60', name='FILE_NAME', value='/mnt/logfs/neo4j-topo- svc/1/servicetracking_stdout.log' [2018-01-25T17:33:08.925057] Setting value; msg='0x7fc0a807bd60', name='SOURCE', value='s_file_stdout'
[2018-01-25T17:33:08.925062] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5'
[2018-01-25T17:33:08.824606] Incoming log entry; line='time="2017-11-21 19:10:56.399341" level=debug msg="[section not required]" section=etcd tag=ROBOT_serviceTracking_robot-topo-svc-2107321943- 48rnw'
[2018-01-25T17:33:09.022629] Filter rule evaluation begins; msg='0x7fc0a807bd60', rule='f_compliant_hosts', location='/etc/syslog-ng/ syslog-ng.conf:66:27' [2018-01-25T17:33:08.824700] Setting value; msg='0x7fc0a409b810', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf'
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
... [2018-01-25T17:33:09.022911] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5'
[2018-01-25T17:33:09.022916] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_compliant_hosts', location='/etc/syslog-ng/ syslog-ng.conf:66:27' [2018-01-25T17:33:09.022922] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='=='
[2018-01-25T17:33:09.022927] Filter rule evaluation result; msg='0x7fc08c0f5550', result='not-match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27'
[2018-01-25T17:33:09.022932] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_noncompliant_hosts', location='/etc/syslog-ng/ syslog-ng.conf:70:30' [2018-01-25T17:33:09.022937] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022942] Filter node evaluation result; msg='0x7fc08c0f5550', result='match', type='filter(f_compliant_hosts)' [2018-01-25T17:33:09.022947] Filter rule evaluation result; msg='0x7fc08c0f5550', result='match', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' Aborted
Is this a known issue with the key-value parser in 3.13?
Regards, Don
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The following blogpost might help you to test the patch: https://syslog-ng.com/blog/build-syslog-ng-rpm-patched-git-sources-rhel-cent... On Mon, Feb 19, 2018 at 1:57 PM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hello Don!
We did have a regression in kv-parser, it has been fixed recently, merged to upstream and it will be in the upcoming syslog-ng release 3.14.1. You can find the patch here: https://github.com/balabit/syslog-ng/commit/ aba5d41c1f092981501e75f009ffffee76fc77ea
Best Regards, Gabor
On Thu, Jan 25, 2018 at 6:37 PM, Don C <lawsuit_loser@yahoo.com> wrote:
Hi,
I was upgrading syslog-ng from 3.12 to 3.13 using the prebuilt unofficial RPMs. I was testing my config on 3.13 and ran into the following issue where the key-value parsing aborts. This appears to be a regression in 3.13, the abort never happened in 3.12.
If I remove the kv_parser from my config, there is no issue.
Here's the end of the debug and verbose output I get. Notice the garbage values in the key-value names parsed from the message.
[2018-01-25T17:33:08.924895] Filter rule evaluation result; msg='0x7fc0a807b370', result='match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.924919] Setting value; msg='0x7fc0a80612f0', name='@\x15\x01pÀ', value='2017-11-21 19:11:24.817041'
[2018-01-25T17:33:08.924930] Setting value; msg='0x7fc0a80612f0', name='', value='debug'
[2018-01-25T17:33:08.924936] Setting value; msg='0x7fc0a80612f0', name='f_compliant_hosts', value='[robotnats kafka transport prepared]' [2018-01-25T17:33:08.924944] Setting value; msg='0x7fc0a80612f0', name='PK\x06¨À', value='ROBOT_serviceTracking_n eo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924952] Setting value; msg='0x7fc0a80612f0', name='', value='nats://nats:4222'
[2018-01-25T17:33:08.924957] Setting value; msg='0x7fc0a80612f0', name='', value='[robot-kafka:9092]'
[2018-01-25T17:33:08.924963] Setting value; msg='0x7fc0a80612f0', name='kv_kafka', value='robotnats_kafka'
[2018-01-25T17:33:08.924969] Setting value; msg='0x7fc0a80612f0', name=' »\x08¬À', value='ROBOT_serviceTracking_n eo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924975] Message parsing complete; result='1', rule='p_kv', location='/etc/syslog-ng/syslog-ng.conf:75:5'
[2018-01-25T17:33:08.924998] Incoming log entry; line='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925030] Setting value; msg='0x7fc0a807bd60', name='MESSAGE', value='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925040] Setting value; msg='0x7fc0a807bd60', name='HOST_FROM', value='syslog-ng-logging-10401 68119-qsmrf' [2018-01-25T17:33:08.925046] Setting value; msg='0x7fc0a807bd60', name='HOST', value='syslog-ng-logging-1040168119-qsmrf/syslog-ng-logging- 1040168119-qsmrf' [2018-01-25T17:33:08.925053] Setting value; msg='0x7fc0a807bd60', name='FILE_NAME', value='/mnt/logfs/neo4j-topo-s vc/1/servicetracking_stdout.log' [2018-01-25T17:33:08.925057] Setting value; msg='0x7fc0a807bd60', name='SOURCE', value='s_file_stdout'
[2018-01-25T17:33:08.925062] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5'
[2018-01-25T17:33:08.824606] Incoming log entry; line='time="2017-11-21 19:10:56.399341" level=debug msg="[section not required]" section=etcd tag=ROBOT_serviceTracking_robot-topo-svc-2107321943 <(210)%20732-1943> -48rnw'
[2018-01-25T17:33:09.022629] Filter rule evaluation begins; msg='0x7fc0a807bd60', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.824700] Setting value; msg='0x7fc0a409b810', name='HOST_FROM', value='syslog-ng-logging-10401 68119-qsmrf' #
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
... [2018-01-25T17:33:09.022911] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5'
[2018-01-25T17:33:09.022916] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022922] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='=='
[2018-01-25T17:33:09.022927] Filter rule evaluation result; msg='0x7fc08c0f5550', result='not-match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27'
[2018-01-25T17:33:09.022932] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' [2018-01-25T17:33:09.022937] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022942] Filter node evaluation result; msg='0x7fc08c0f5550', result='match', type='filter(f_compliant_hosts)' [2018-01-25T17:33:09.022947] Filter rule evaluation result; msg='0x7fc08c0f5550', result='match', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' Aborted
Is this a known issue with the key-value parser in 3.13?
Regards, Don
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Don C
-
Fekete, Róbert
-
Nagy, Gábor