format-json incorrectly parsing some events
I'm using syslog-ng rpm version 3.12.1-2 on CentOS 7 When we receive events remotely from another CentOS 7 host it uses the RFC5424 format and parses the messages correctly. However we have some hosts that are older and still using rsyslog which is using the RFC3164 format - those events do not parse correctly. My question is what is the best way to get syslog-ng to parse them? This is how they come out: {"TAGS":".source.test","SOURCEIP":"127.0.0.1","SOURCE":"test","SEQNUM":"26","PROGRAM":"info","PRIORITY":"notice","MESSAGE":" mig-agent 10430 - - - [info] refreshing agent environment","LEGACY_MSGHDR":"info ","HOST_FROM":" syslog-dev1.private.mdc1.mozilla.com","HOST":"sanvmadm1.ops.mdc1.mozilla.com","FILE_NAME":"/var/log/test.log","FACILITY":"user","DATE":"Jan 17 23:57:52","CATEGORY":"syslog"] Notice the Program says "info" and the mig-agent and pid are in the message key's value. This is a correctly parsed event that has those fields parsed properly: {"TAGS":".source.moz_net","SOURCEIP":"127.0.0.1","SOURCE":"moz_net","SEQNUM":"35","PROGRAM":"mig-agent","PRIORITY":"info","PID":"2698","MESSAGE":"- - - [info] Public IP retrieval failed through proxy http://proxy.dmz.scl3.mozilla.com:3128 - Get https://api.mig.mozilla.org/api/v1//ip: proxyconnect tcp: dial tcp 10.22.74.78:3128: i/o timeout","LEGACY_MSGHDR":"mig-agent[2698]: ","HOST_FROM":"localhost6.localdomain","HOST":" syslog-dev1.private.mdc1.mozilla.com","FACILITY":"daemon","DATE":"Jan 18 00:02:25","CATEGORY":"syslog"} destination d_amqp { amqp( vhost("/") host("localhost") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope selected_macros --scope nv_pairs)") persistent(no) username("rabbituser") password("*****") ); }; -- Alicia Smith @phrozyn Information Security Engineer asmith@mozilla.com
Hi, On Thu, Jan 18, 2018 at 8:47 PM, Alicia Smith <asmith@mozilla.com> wrote:
I'm using syslog-ng rpm version 3.12.1-2 on CentOS 7
When we receive events remotely from another CentOS 7 host it uses the RFC5424 format and parses the messages correctly. However we have some hosts that are older and still using rsyslog which is using the RFC3164 format - those events do not parse correctly.
My question is what is the best way to get syslog-ng to parse them?
Parsing is done before sending logs to the destination. Syslog protocols are usually parsed on the source driver side. You can use the syslog() source for RFC5424 messages and the network() source for RFC3164 (see at https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g... ).
destination d_amqp { amqp(
So, you use the AMQP destination. Good to know. Then I fix packaging for 3.13 :) The AMQP client library does not compile on Fedora due to openssl 1.1 and I also disabled it for RHEL to keep the spec file for the RPM clean and easy. I hope to publish an updated RPM later today which features AMQP for RHEL. Bye, CzP
participants (2)
-
Alicia Smith
-
Czanik, Péter