[syslog-ng] Flow-control behaviour and Cached lines timestamping, plus a suspend error in 2.1beta1

[Behal, Pavel]

Dear Bazsi,

Thank you for the clarificatrion.

Regarding the point 2) - sorry, you are right. I have re-tested syslog-ng and it timestamps the events in buffer correctly. I think, my problem was related to the point 1) with enabled flow-control on system sources. I suspect, on Solaris 9, the generated messages got buffered in the OS, and were timestamped just after the reading process was waken-up.

Best Regards,

Pavel


--
Pavel Běhal
Hewlett-Packard

-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Balazs Scheidler
Sent: Thursday, May 29, 2008 7:38 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Flow-control behaviour and Cached lines timestamping, plus a suspend error in 2.1beta1

On Thu, 2008-05-29 at 13:03 +0000, Behal, Pavel wrote:
> Dear Bazsi,
>
> I have found two "features" and one bug I would like to discuss a little:
>
> 1) We use the syslog-ng 2.1alpha1 as a root system syslog. The problem is, we have had two destinations setup. First destination was the traditional write of system messages to the file like "/var/log/messages" etc. The second destination path was to the TCP stream to the central syslog server. And the problem is, we have used the flow-control flag on the central TCP path. But, when the central TCP syslog-ng becomes unavailable, after all the buffers filled up, the local system syslog-ng stopped processing any system messages. It did not write anything through the local file destinations until the TCP communication was re-established.
> Is this a correct behaviour? We have found, that to allow smooth processing of local system sources, we can not use flow-control flag on any destination paths it used..
>

Yes, this is correct behaviour. If syslog-ng continued reading from the local source (to feed the file), it'd have to put the read messages somewhere. Normally the memory based FIFO is used for this purpose, but once it fills up there's no other way, but to suspend reading.

The workaround is to increase the buffer size of the destination, and window size of the source to match the network outage you want to plan for.

The Premium Edition features a disk buffer, in which case you can use disk space to buffer data towards the TCP destination and until that fills up, the local messages file will correctly be written.

>
> 2) When messages are in syslog-ng buffer, they do not contain any
> timestamp. We have realized, that the timestamp is added at the time
> the syslog-ng flushes all the buffers to the disk. This is not very
> good in case you need to debug the messages ages and all of them have same and non-related timestamp.

Hmm. this does not sound true, every message is timestamped as it enters syslog-ng and this timestamp is kept throughout its life in syslog-ng.
(with disk buffers it is saved to disk and restored when it is possible to send the message out).


> Is it possible to modify the logic of syslog-ng to timestamp the
> messages as soon as possible - even before the buffer, at the
> collection time? We have found it important for log integrity reasons.

It works like this.

>
>
> 3) Directly in the 2.1beta1 there has been introduced some error with the "suspend writing a destination file when an I/O error ..." patch. During our tests we have found it continously thinks the destination is unavailable and writes strange messages into to the internal log:
>
> May 29 15:00:02 europa syslog-ng[16798]: I/O error occurred while writing; fd='7', error='Connection timed out (110)'
> May 29 15:00:02 europa syslog-ng[16798]: Connection broken; time_reopen='60'
> May 29 15:00:02 europa syslog-ng[16798]: Suspending write operation because of an I/O error; fd='7', time_reopen='60'

Right, I've found this problem in the meanwhile. This code was added to inject artifical errors to check that syslog-ng correctly retries I/O operations. This code was integrated to 2.1beta1.

I'm pushing out a git update today, and probably will release 2.1beta2 soon.

--
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html