[syslog-ng] syslog-ng blocking application

[Balazs Scheidler]

On Tue, 2012-02-21 at 14:10 +0100, Ulrich David wrote:
> Hi,
> 
> I have an ISC-DHCP who is logging on a local facility on a chrooted system :
> 
> #options { 
> #        chain_hostnames(no); 
> #        use_dns(no);
> #        flush_lines(1);
> #        stats_freq(3600); 
> #};
> #source dhcp_src {
> #    unix-stream("/chroot/dhcp/dev/log" max-connections(256));
> #};
> 
> My syslog-ng take logs from this source, filter on program and then put the logs on two destinations :
> 
> #filter f_chroot_l2 {
> #        program(dhcpd);
> #};
> #
> #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); };
> #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); };
> #
> #log { 
> #        source(dhcp_src);
> #        filter(f_chroot_l2);
> #        destination(dhcp_log);
> #        destination(dhcp_sql);
> #        flags(final );
> #};
> 
> Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test).
> 
> When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too.
> 
> I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem.
> 
> I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment.

Well, I probably know the reason for this, however I agree that it is
confusing.

The history for the behaviour is this:
  - earlier syslog-ng prioritized all output operations over input ones
  - this meant that if any output driver had things to do, it inhibited
all input drivers to accept further messages.
  - however with the threaded behaviour the same prioritization is very
difficult to implement (and would probably cause a lot of performance
penalties), so file destinations enable flow-control implicitly
  - this flow-control is not completely the same as using
"flags(flow-control)" as when the file destination is suspended because
of an error, it is not in effect. This is called "soft-flow-control" in
the codebase.
  - flow-control is inherited to all parallel branches of a log
statement.

This means that if you used separate log statements, the program
destination wouldn't be flow controlled.

I remember having to readd this "inheritance" of flow-control (as it was
removed first while implementing the threaded architecture), because
without it flow-control wasn't in effect when filtering was used in one
of the branches.

Anyway, I'd have to give it some thought, since this case is clearly
difficult to diagnose and understand.

For now, use a separate log statement, that should settle it.

-- 
Bazsi