[syslog-ng] syslog-ng blocking application

[Ulrich David]

Hi,

Thanks for your answer.

I tried to separate the destination in two log statements :
log { 
	source(dhcp_src);
	filter(f_chroot_l2);
	destination(dhcp_log);
};
log { 
	source(dhcp_src);
	filter(f_chroot_l2);
	destination(dhcp_sql);
	flags(final);
};

But i got the same result, blocking dhcp_sql destination blocks dhcp_log destination.
I have found a workaround, i throw my log in a unix socket and the file dhcpd.log together, then i re-take the socket to put in SQL database. (not so pretty)

New versions of syslog-ng have a different behavior which is better to not block inputs ?

My version of syslog-ng is :

dhcp-SEIC-3 syslog-ng # syslog-ng --version
syslog-ng 3.0.4
Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10
Compile-Date: Dec  3 2009 16:17:01
Enable-Threads: off
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-Sun-STREAMS: off
Enable-Sun-Door: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: on
Enable-SSL: on
Enable-SQL: off
Enable-Linux-Caps: off
Enable-Pcre: on


David Ulrich
---
Head of Systems, Quality and Security

e-mail: david.ulrich at netplus.pro
Direct: +41275657531
Groupe:  +41275657530
Réception:  +41275657511
Fax:  +41275657512

netplus.ch SA
Techno-pôle 3
CH-3960 Sierre

Le 16 mars 2012 à 13:24, Balazs Scheidler a écrit :

> On Tue, 2012-02-21 at 14:10 +0100, Ulrich David wrote:
>> Hi,
>> 
>> I have an ISC-DHCP who is logging on a local facility on a chrooted system :
>> 
>> #options { 
>> #        chain_hostnames(no); 
>> #        use_dns(no);
>> #        flush_lines(1);
>> #        stats_freq(3600); 
>> #};
>> #source dhcp_src {
>> #    unix-stream("/chroot/dhcp/dev/log" max-connections(256));
>> #};
>> 
>> My syslog-ng take logs from this source, filter on program and then put the logs on two destinations :
>> 
>> #filter f_chroot_l2 {
>> #        program(dhcpd);
>> #};
>> #
>> #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); };
>> #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); };
>> #
>> #log { 
>> #        source(dhcp_src);
>> #        filter(f_chroot_l2);
>> #        destination(dhcp_log);
>> #        destination(dhcp_sql);
>> #        flags(final );
>> #};
>> 
>> Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test).
>> 
>> When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too.
>> 
>> I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem.
>> 
>> I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment.
> 
> Well, I probably know the reason for this, however I agree that it is
> confusing.
> 
> The history for the behaviour is this:
>  - earlier syslog-ng prioritized all output operations over input ones
>  - this meant that if any output driver had things to do, it inhibited
> all input drivers to accept further messages.
>  - however with the threaded behaviour the same prioritization is very
> difficult to implement (and would probably cause a lot of performance
> penalties), so file destinations enable flow-control implicitly
>  - this flow-control is not completely the same as using
> "flags(flow-control)" as when the file destination is suspended because
> of an error, it is not in effect. This is called "soft-flow-control" in
> the codebase.
>  - flow-control is inherited to all parallel branches of a log
> statement.
> 
> This means that if you used separate log statements, the program
> destination wouldn't be flow controlled.
> 
> I remember having to readd this "inheritance" of flow-control (as it was
> removed first while implementing the threaded architecture), because
> without it flow-control wasn't in effect when filtering was used in one
> of the branches.
> 
> Anyway, I'd have to give it some thought, since this case is clearly
> difficult to diagnose and understand.
> 
> For now, use a separate log statement, that should settle it.
> 
> -- 
> Bazsi
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>