[syslog-ng] format-json incorrectly parsing some events

[Alicia Smith]

I'm using syslog-ng rpm  version 3.12.1-2 on CentOS 7

When we receive events remotely from another CentOS 7 host it uses the
RFC5424 format and parses the messages correctly.
However we have some hosts that are older and still using rsyslog which is
using the RFC3164 format - those events do not parse correctly.

My question is what is the best way to get syslog-ng to parse them?

This is how they come out:
{"TAGS":".source.test","SOURCEIP":"127.0.0.1","SOURCE":"test","SEQNUM":"26","PROGRAM":"info","PRIORITY":"notice","MESSAGE":"
  mig-agent       10430   - - - [info] refreshing agent
environment","LEGACY_MSGHDR":"info ","HOST_FROM":"
syslog-dev1.private.mdc1.mozilla.com","HOST":"sanvmadm1.ops.mdc1.mozilla.com","FILE_NAME":"/var/log/test.log","FACILITY":"user","DATE":"Jan
17 23:57:52","CATEGORY":"syslog"]
Notice the Program says "info" and the mig-agent and pid are in the message
key's value.

This is a correctly parsed event that has those fields parsed properly:
{"TAGS":".source.moz_net","SOURCEIP":"127.0.0.1","SOURCE":"moz_net","SEQNUM":"35","PROGRAM":"mig-agent","PRIORITY":"info","PID":"2698","MESSAGE":"-
- - [info] Public IP retrieval failed through proxy
http://proxy.dmz.scl3.mozilla.com:3128 - Get
https://api.mig.mozilla.org/api/v1//ip: proxyconnect tcp: dial tcp
10.22.74.78:3128: i/o timeout","LEGACY_MSGHDR":"mig-agent[2698]:
","HOST_FROM":"localhost6.localdomain","HOST":"
syslog-dev1.private.mdc1.mozilla.com","FACILITY":"daemon","DATE":"Jan 18
00:02:25","CATEGORY":"syslog"}



destination d_amqp {
    amqp(
        vhost("/")
        host("localhost")
        port(5672)
        exchange("eventtask")
        exchange-type("direct")
        routing-key("eventtask")
        body("$(format-json --scope selected_macros --scope nv_pairs)")
        persistent(no)
        username("rabbituser")
        password("*****")
    );
};



-- 

Alicia Smith
@phrozyn
Information Security Engineer
asmith at mozilla.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180118/a84024e6/attachment.html>