On Tue, 2012-02-21 at 14:10 +0100, Ulrich David wrote:
Hi,
I have an ISC-DHCP who is logging on a local facility on a chrooted system :
#options { # chain_hostnames(no); # use_dns(no); # flush_lines(1); # stats_freq(3600); #}; #source dhcp_src { # unix-stream("/chroot/dhcp/dev/log" max-connections(256)); #};
My syslog-ng take logs from this source, filter on program and then put the logs on two destinations :
#filter f_chroot_l2 { # program(dhcpd); #}; # #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); }; #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); }; # #log { # source(dhcp_src); # filter(f_chroot_l2); # destination(dhcp_log); # destination(dhcp_sql); # flags(final ); #};
Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test).
When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too.
I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem.
I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment.
Well, I probably know the reason for this, however I agree that it is confusing. The history for the behaviour is this: - earlier syslog-ng prioritized all output operations over input ones - this meant that if any output driver had things to do, it inhibited all input drivers to accept further messages. - however with the threaded behaviour the same prioritization is very difficult to implement (and would probably cause a lot of performance penalties), so file destinations enable flow-control implicitly - this flow-control is not completely the same as using "flags(flow-control)" as when the file destination is suspended because of an error, it is not in effect. This is called "soft-flow-control" in the codebase. - flow-control is inherited to all parallel branches of a log statement. This means that if you used separate log statements, the program destination wouldn't be flow controlled. I remember having to readd this "inheritance" of flow-control (as it was removed first while implementing the threaded architecture), because without it flow-control wasn't in effect when filtering was used in one of the branches. Anyway, I'd have to give it some thought, since this case is clearly difficult to diagnose and understand. For now, use a separate log statement, that should settle it. -- Bazsi