Syslog-ng + graylog2 destination
Hi, I’m playing with syslog-ng OSE and Graylog. In this project I try to find the best method to deliver logs from my central log server to Graylog. So far, it seems graylog2 destination looks the best, however I have a strange issue and I can’t find a solution It seems the other side permanenty drops the connections like this Oct 14 13:53:20 lumberjack syslog-ng[2174]: Casting error; value='', type-hint='int32' Oct 14 13:53:20 lumberjack syslog-ng[2174]: I/O error occurred while writing; fd='481', error='Connection reset by peer (104)' Oct 14 13:53:20 lumberjack syslog-ng[2174]: Syslog connection broken; fd='481', server='AF_INET(10.72.0.137:12201)', time_reopen='10' Oct 14 13:53:30 lumberjack syslog-ng[2174]: Syslog connection established; fd='501', server='AF_INET(10.72.0.137:12201)', local='AF_INET(0.0.0.0:0)’ My destination config is quite simple #Graylog native GELF destination destination d_graylog_gelf { graylog2( log_fifo_size(500000) host("10.72.0.137") transport (tcp) ); }; I’ve tried this with flow-control and w/o flow-control, so I think it must be some tuning issue either on Graylog side or at Syslog. What is interesting, is I don’t have such an issue if I use raw-tcp or syslog destinations toward Graylog Thx L:
Hi László, On Mon, Oct 14, 2019 at 01:58:20PM +0200, László Pál wrote:
Oct 14 13:53:20 lumberjack syslog-ng[2174]: Casting error; value='', type-hint='int32'
It seems one of the fields you're sending should be an integer, but it's an empty string instead. If you look at the destination's code, you'll see that it uses the following format: template-function "format-gelf" "$(format-json version='1.1' host='${HOST}' short_message='${MSG}' level=int(${LEVEL_NUM}) timestamp=int64(${R_UNIXTIME}) _program='${PROGRAM}' _pid=int(${PID}) _facility='${FACILITY}' _class='${.classifier.class}' --key .* --key _*)$(binary 0x00)"; I'm suspecting that one of your macros ($LEVEL_NUM or $PID) is empty, due to a problem in syslog-format parsing. It's probably $PID, as the other one is automatically set. If my theory is correct, try the following workaround: destination d_graylog_gelf { channel { rewrite { set("${PID:-0}", value('PID')); }; }; channel { destination { graylog2( log_fifo_size(500000) host("10.72.0.137") transport (tcp) ); }; }; }; This will set the PID to 0 if unset.
Thank you. It seems this workaround has improved the situation a bit, however I still can see the reset logs from Graylog, so some other things must be wrong. The problem is I have no idea how to figure out which of my message sources are sending in-proper messages. These are mostly routers, but some of the firwalls (ASA) also sends logs to central syslog. It seems GELF is very sensitive, so maybe in this case is better if I simply use syslog or json towards Graylog Laszlo
On 2019. Oct 14., at 15:09, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi László,
On Mon, Oct 14, 2019 at 01:58:20PM +0200, László Pál wrote:
Oct 14 13:53:20 lumberjack syslog-ng[2174]: Casting error; value='', type-hint='int32'
It seems one of the fields you're sending should be an integer, but it's an empty string instead. If you look at the destination's code, you'll see that it uses the following format:
template-function "format-gelf" "$(format-json version='1.1' host='${HOST}' short_message='${MSG}' level=int(${LEVEL_NUM}) timestamp=int64(${R_UNIXTIME}) _program='${PROGRAM}' _pid=int(${PID}) _facility='${FACILITY}' _class='${.classifier.class}' --key .* --key _*)$(binary 0x00)";
I'm suspecting that one of your macros ($LEVEL_NUM or $PID) is empty, due to a problem in syslog-format parsing. It's probably $PID, as the other one is automatically set.
If my theory is correct, try the following workaround:
destination d_graylog_gelf { channel { rewrite { set("${PID:-0}", value('PID')); }; }; channel { destination { graylog2( log_fifo_size(500000) host("10.72.0.137") transport (tcp) ); }; }; };
This will set the PID to 0 if unset.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Mon, Oct 14, 2019 at 03:50:42PM +0200, László Pál wrote:
Thank you. It seems this workaround has improved the situation a bit, however I still can see the reset logs from Graylog, so some other things must be wrong. The problem is I have no idea how to figure out which of my message sources are sending in-proper messages. These are mostly routers, but some of the firwalls (ASA) also sends logs to central syslog.
It seems GELF is very sensitive, so maybe in this case is better if I simply use syslog or json towards Graylog
It's already what happens behind the scenes, as graylog2() is just an SCL wrapper of network(). You could experiment other templates by overriding the default. Here's an example that sets default values for all macros (in case they're absent): destination d_graylog_gelf { graylog2( log_fifo_size(500000) host("10.72.0.137") transport (tcp) template("$(format-json version='1.1' host='${HOST:-none}' short_message='${MSG:-none}' level=int(${LEVEL_NUM:-0}) timestamp=int64(${R_UNIXTIME:-0}) _program='${PROGRAM:-none}' _pid=int(${PID:-0}) _facility='${FACILITY:-none}' _class='${.classifier.class:-none}' --key .* --key _*)$(binary 0x00)") ); };
participants (2)
-
Fabien Wernli
-
László Pál