[syslog-ng] syslog-ng blocking application

[Martin Holste]

What happens if you recompile with threads enabled and then add
threads(yes); to your global config?  You should get multiple threads
running for each log statement which should theoretically alleviate
the blocking problem.

On Fri, Mar 16, 2012 at 8:51 AM, Ulrich David <david.ulrich at netplus.pro> wrote:
> Hi,
>
> Thanks for your answer.
>
> I tried to separate the destination in two log statements :
> log {
>        source(dhcp_src);
>        filter(f_chroot_l2);
>        destination(dhcp_log);
> };
> log {
>        source(dhcp_src);
>        filter(f_chroot_l2);
>        destination(dhcp_sql);
>        flags(final);
> };
>
> But i got the same result, blocking dhcp_sql destination blocks dhcp_log destination.
> I have found a workaround, i throw my log in a unix socket and the file dhcpd.log together, then i re-take the socket to put in SQL database. (not so pretty)
>
> New versions of syslog-ng have a different behavior which is better to not block inputs ?
>
> My version of syslog-ng is :
>
> dhcp-SEIC-3 syslog-ng # syslog-ng --version
> syslog-ng 3.0.4
> Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10
> Compile-Date: Dec  3 2009 16:17:01
> Enable-Threads: off
> Enable-Debug: off
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-Sun-STREAMS: off
> Enable-Sun-Door: off
> Enable-IPv6: on
> Enable-Spoof-Source: off
> Enable-TCP-Wrapper: on
> Enable-SSL: on
> Enable-SQL: off
> Enable-Linux-Caps: off
> Enable-Pcre: on
>
>
> David Ulrich
> ---
> Head of Systems, Quality and Security
>
> e-mail: david.ulrich at netplus.pro
> Direct: +41275657531
> Groupe:  +41275657530
> Réception:  +41275657511
> Fax:  +41275657512
>
> netplus.ch SA
> Techno-pôle 3
> CH-3960 Sierre
>
> Le 16 mars 2012 à 13:24, Balazs Scheidler a écrit :
>
>> On Tue, 2012-02-21 at 14:10 +0100, Ulrich David wrote:
>>> Hi,
>>>
>>> I have an ISC-DHCP who is logging on a local facility on a chrooted system :
>>>
>>> #options {
>>> #        chain_hostnames(no);
>>> #        use_dns(no);
>>> #        flush_lines(1);
>>> #        stats_freq(3600);
>>> #};
>>> #source dhcp_src {
>>> #    unix-stream("/chroot/dhcp/dev/log" max-connections(256));
>>> #};
>>>
>>> My syslog-ng take logs from this source, filter on program and then put the logs on two destinations :
>>>
>>> #filter f_chroot_l2 {
>>> #        program(dhcpd);
>>> #};
>>> #
>>> #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); };
>>> #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); };
>>> #
>>> #log {
>>> #        source(dhcp_src);
>>> #        filter(f_chroot_l2);
>>> #        destination(dhcp_log);
>>> #        destination(dhcp_sql);
>>> #        flags(final );
>>> #};
>>>
>>> Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test).
>>>
>>> When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too.
>>>
>>> I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem.
>>>
>>> I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment.
>>
>> Well, I probably know the reason for this, however I agree that it is
>> confusing.
>>
>> The history for the behaviour is this:
>>  - earlier syslog-ng prioritized all output operations over input ones
>>  - this meant that if any output driver had things to do, it inhibited
>> all input drivers to accept further messages.
>>  - however with the threaded behaviour the same prioritization is very
>> difficult to implement (and would probably cause a lot of performance
>> penalties), so file destinations enable flow-control implicitly
>>  - this flow-control is not completely the same as using
>> "flags(flow-control)" as when the file destination is suspended because
>> of an error, it is not in effect. This is called "soft-flow-control" in
>> the codebase.
>>  - flow-control is inherited to all parallel branches of a log
>> statement.
>>
>> This means that if you used separate log statements, the program
>> destination wouldn't be flow controlled.
>>
>> I remember having to readd this "inheritance" of flow-control (as it was
>> removed first while implementing the threaded architecture), because
>> without it flow-control wasn't in effect when filtering was used in one
>> of the branches.
>>
>> Anyway, I'd have to give it some thought, since this case is clearly
>> difficult to diagnose and understand.
>>
>> For now, use a separate log statement, that should settle it.
>>
>> --
>> Bazsi
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>