[syslog-ng] Lot of log lost

[Balazs Scheidler]

On Wed, 2011-07-20 at 11:14 +0200, maxime.denier at orange-ftgroup.com
wrote:
> Hello,
>  
> I have recently installed syslog-ng OSE 3.1 as log collector and I
> face a problem.
> A great number of logs arrive on the server, but a little part of them
> arrive in the destination files, but all the destination files have
> some logs on it.
> I have enabled the verbose mode and I see this:
> Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file
> writer;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv'
> Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination
> files;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination
> files;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out,
> reaping;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/peony2/201107202400.csv'
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd;
> fd='31'
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out,
> reaping;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/decca2/201107202400.csv'
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd;
> fd='19'
> Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file
> writer;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/hyenne2/201107202400.csv'
> Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file
> writer;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/olive2/201107202400.csv'
> Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination
> files;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
> Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out,
> reaping;
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv'
> Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd;
> fd='24'
>  
> I haven't found information about the root cause of these timed out.

These only indicate that syslog-ng is properly garbage-collecting
destination files that receive no data. This is not an error, that's why
you only get this if you enable --debug / --verbose (i'm not sure which
one).

> This seems to be a writing problem.
> Before using syslog-ng, Logs were processed by a application owned by
> the firewall publisher on the same type of hardware without this great
> number of log lost.

Is this UDP? syslog-ng doesn't increase udp receive buffer sizes unless
explicitly told so, using the so-rcvbuf() option on the udp source. you
probably need to increase that.

But I'd recommend not using udp, as that can cause a lot of lost
messages easily. (if you create a simple loop that sends udp frames to
the syslog receiver, you can easily see as much as 97% messages lost!)
easy DoS.

>  
> If any body have already faced this problem and have a solution, it
> would be wonderful.

There was a guy on this list, who has published charts and numbers how
he had to tune the various buffering options. I forgot his name, but if
you google syslog-ng, udp buffer size, you'll probably find it.


-- 
Bazsi