[syslog-ng] syslog-ng blocking application

Ulrich David david.ulrich at netplus.pro
Fri Mar 16 15:54:09 CET 2012


i don't have the possibility to test it with threads on, on gentoo it seems to be only on 3.2.5 activated.

But I have done a test on a 3.2.4 and having two difference log statements works (same syslog-ng.conf than 3.0.4). More surprising in one statement it works too.

# syslog-ng --version
syslog-ng 3.2.4
Installer-Version: 3.2.4
Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6
Compile-Date: Jul 13 2011 15:17:37
Enable-Threads: off
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-Sun-STREAMS: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: on
Enable-SSL: on
Enable-SQL: off
Enable-Linux-Caps: off
Enable-Pcre: on
Enable-Pacct: off


Is it possible there is just a small bug on 3.0.4 ?


David Ulrich
---
Head of Systems, Quality and Security

e-mail: david.ulrich at netplus.pro
Direct: +41275657531
Groupe:  +41275657530
Réception:  +41275657511
Fax:  +41275657512

netplus.ch SA
Techno-pôle 3
CH-3960 Sierre

Le 16 mars 2012 à 15:05, Martin Holste a écrit :

> What happens if you recompile with threads enabled and then add
> threads(yes); to your global config?  You should get multiple threads
> running for each log statement which should theoretically alleviate
> the blocking problem.
> 
> On Fri, Mar 16, 2012 at 8:51 AM, Ulrich David <david.ulrich at netplus.pro> wrote:
>> Hi,
>> 
>> Thanks for your answer.
>> 
>> I tried to separate the destination in two log statements :
>> log {
>>        source(dhcp_src);
>>        filter(f_chroot_l2);
>>        destination(dhcp_log);
>> };
>> log {
>>        source(dhcp_src);
>>        filter(f_chroot_l2);
>>        destination(dhcp_sql);
>>        flags(final);
>> };
>> 
>> But i got the same result, blocking dhcp_sql destination blocks dhcp_log destination.
>> I have found a workaround, i throw my log in a unix socket and the file dhcpd.log together, then i re-take the socket to put in SQL database. (not so pretty)
>> 
>> New versions of syslog-ng have a different behavior which is better to not block inputs ?
>> 
>> My version of syslog-ng is :
>> 
>> dhcp-SEIC-3 syslog-ng # syslog-ng --version
>> syslog-ng 3.0.4
>> Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10
>> Compile-Date: Dec  3 2009 16:17:01
>> Enable-Threads: off
>> Enable-Debug: off
>> Enable-GProf: off
>> Enable-Memtrace: off
>> Enable-Sun-STREAMS: off
>> Enable-Sun-Door: off
>> Enable-IPv6: on
>> Enable-Spoof-Source: off
>> Enable-TCP-Wrapper: on
>> Enable-SSL: on
>> Enable-SQL: off
>> Enable-Linux-Caps: off
>> Enable-Pcre: on
>> 
>> 
>> David Ulrich
>> ---
>> Head of Systems, Quality and Security
>> 
>> e-mail: david.ulrich at netplus.pro
>> Direct: +41275657531
>> Groupe:  +41275657530
>> Réception:  +41275657511
>> Fax:  +41275657512
>> 
>> netplus.ch SA
>> Techno-pôle 3
>> CH-3960 Sierre
>> 
>> Le 16 mars 2012 à 13:24, Balazs Scheidler a écrit :
>> 
>>> On Tue, 2012-02-21 at 14:10 +0100, Ulrich David wrote:
>>>> Hi,
>>>> 
>>>> I have an ISC-DHCP who is logging on a local facility on a chrooted system :
>>>> 
>>>> #options {
>>>> #        chain_hostnames(no);
>>>> #        use_dns(no);
>>>> #        flush_lines(1);
>>>> #        stats_freq(3600);
>>>> #};
>>>> #source dhcp_src {
>>>> #    unix-stream("/chroot/dhcp/dev/log" max-connections(256));
>>>> #};
>>>> 
>>>> My syslog-ng take logs from this source, filter on program and then put the logs on two destinations :
>>>> 
>>>> #filter f_chroot_l2 {
>>>> #        program(dhcpd);
>>>> #};
>>>> #
>>>> #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); };
>>>> #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); };
>>>> #
>>>> #log {
>>>> #        source(dhcp_src);
>>>> #        filter(f_chroot_l2);
>>>> #        destination(dhcp_log);
>>>> #        destination(dhcp_sql);
>>>> #        flags(final );
>>>> #};
>>>> 
>>>> Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test).
>>>> 
>>>> When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too.
>>>> 
>>>> I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem.
>>>> 
>>>> I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment.
>>> 
>>> Well, I probably know the reason for this, however I agree that it is
>>> confusing.
>>> 
>>> The history for the behaviour is this:
>>>  - earlier syslog-ng prioritized all output operations over input ones
>>>  - this meant that if any output driver had things to do, it inhibited
>>> all input drivers to accept further messages.
>>>  - however with the threaded behaviour the same prioritization is very
>>> difficult to implement (and would probably cause a lot of performance
>>> penalties), so file destinations enable flow-control implicitly
>>>  - this flow-control is not completely the same as using
>>> "flags(flow-control)" as when the file destination is suspended because
>>> of an error, it is not in effect. This is called "soft-flow-control" in
>>> the codebase.
>>>  - flow-control is inherited to all parallel branches of a log
>>> statement.
>>> 
>>> This means that if you used separate log statements, the program
>>> destination wouldn't be flow controlled.
>>> 
>>> I remember having to readd this "inheritance" of flow-control (as it was
>>> removed first while implementing the threaded architecture), because
>>> without it flow-control wasn't in effect when filtering was used in one
>>> of the branches.
>>> 
>>> Anyway, I'd have to give it some thought, since this case is clearly
>>> difficult to diagnose and understand.
>>> 
>>> For now, use a separate log statement, that should settle it.
>>> 
>>> --
>>> Bazsi
>>> 
>>> 
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>> 
>> 
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 



More information about the syslog-ng mailing list