Hello, I have recently installed syslog-ng OSE 3.1 as log collector and I face a problem. A great number of logs arrive on the server, but a little part of them arrive in the destination files, but all the destination files have some logs on it. I have enabled the verbose mode and I see this: Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv' Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/peony2/201107202400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='31' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/decca2/201107202400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='19' Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/hyenne2/201107202400.csv' Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/olive2/201107202400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd; fd='24' I haven't found information about the root cause of these timed out. This seems to be a writing problem. Before using syslog-ng, Logs were processed by a application owned by the firewall publisher on the same type of hardware without this great number of log lost. If any body have already faced this problem and have a solution, it would be wonderful. Regards, Maxime Denier ******************************************************************************** IMPORTANT.Les informations contenues dans ce message electronique y compris les fichiers attaches sont strictement confidentielles et peuvent etre protegees par la loi. Ce message electronique est destine exclusivement au(x) destinataire(s) mentionne(s) ci-dessus. Si vous avez recu ce message par erreur ou s il ne vous est pas destine, veuillez immediatement le signaler a l expediteur et effacer ce message et tous les fichiers eventuellement attaches. Toute lecture, exploitation ou transmission des informations contenues dans ce message est interdite. Tout message electronique est susceptible d alteration. A ce titre, le Groupe France Telecom decline toute responsabilite notamment s il a ete altere, deforme ou falsifie. De meme, il appartient au destinataire de s assurer de l absence de tout virus. IMPORTANT.This e-mail message and any attachments are strictly confidential and may be protected by law. This message is intended only for the named recipient(s) above. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. Any unauthorized view, usage or disclosure ofthis message is prohibited. Since e-mail messages may not be reliable, France Telecom Group shall not be liable for any message if modified, changed or falsified. Additionally the recipient should ensure they are actually virus free. ********************************************************************************
On Wed, 2011-07-20 at 11:14 +0200, maxime.denier@orange-ftgroup.com wrote:
Hello,
I have recently installed syslog-ng OSE 3.1 as log collector and I face a problem. A great number of logs arrive on the server, but a little part of them arrive in the destination files, but all the destination files have some logs on it. I have enabled the verbose mode and I see this: Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv' Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/peony2/201107202400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='31' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/decca2/201107202400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='19' Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/hyenne2/201107202400.csv' Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/olive2/201107202400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd; fd='24'
I haven't found information about the root cause of these timed out.
These only indicate that syslog-ng is properly garbage-collecting destination files that receive no data. This is not an error, that's why you only get this if you enable --debug / --verbose (i'm not sure which one).
This seems to be a writing problem. Before using syslog-ng, Logs were processed by a application owned by the firewall publisher on the same type of hardware without this great number of log lost.
Is this UDP? syslog-ng doesn't increase udp receive buffer sizes unless explicitly told so, using the so-rcvbuf() option on the udp source. you probably need to increase that. But I'd recommend not using udp, as that can cause a lot of lost messages easily. (if you create a simple loop that sends udp frames to the syslog receiver, you can easily see as much as 97% messages lost!) easy DoS.
If any body have already faced this problem and have a solution, it would be wonderful.
There was a guy on this list, who has published charts and numbers how he had to tune the various buffering options. I forgot his name, but if you google syslog-ng, udp buffer size, you'll probably find it. -- Bazsi
Hi, I did some work with UDP buffer testing and wrote it into my Wiki at http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.1#UDP_Buffers The other guy that Baszi refers to did some nice graphs, but it was a temporary URL. At the time, I thought the email thread would be useful, so I copied it to text and saved it locally. Here's that thread: --- I've run a series of tests against our log server, using loggen-generated logging rates of 100, 1000, 2000, 4000, 8000, 16000, and 32000 messages/sec, and measured the results for each rate using socket buffer sizes ranging from 128KB to 16M. The results show, essentially, what buffer size you need to meet a target rate of message logging. So I'm happy now, and I'm getting the sort of rates I expect to be able to support. I figured other folks might like the data. I've put the raw data online as well as a graph of the results: http://drop.io/syslog_ng The graph shows so_rcvbuf() size along the X axis, and packet loss along the Y axis.
From Baszi: Hmm. the numbers you are seeing are indeed low, with sufficient buffer sizes I could get up to the 20k message/sec range with syslog-ng, although it's been a while I last tested it.
What I'd recommend is to calculate how much _bytes_ the message rate you are generating means. If you generate 2000 messages, 300 byte each (loggen default IIRC), that's 600000 bytes every second. syslog-ng is single threaded, thus the latency to write to the disk applies. This means that it may take some time for syslog-ng to care about its source, if it is busy writing out messages. This is the #1 reason why I want to work on multithreading. With a flow controlled source, syslog-ng is able to do about 70-75k msg/sec. But not with UDP. In order to improve the numbers, I'd: 1) increase the receive buffer rate to 3-5 seconds (e.g. 3-5MB, not just 0.5) 2) increase log_fetch_limit() to a larger value, this controls how much messages syslog-ng fetches in each poll iteration. Increase this to 3-500 3) increase log_fifo_size() for the destination, by taking the fetch_limit values for each sources feeding the destination (so if you have two sources, each with 1000 fetch limit, then the destination queue should be _at least_ 2000, preferably rounded to the next order of magnitude (e.g. with 2x1000 fetch-limits, increase fifo to 10000) You haven't included in your email whether syslog-ng itself is dropping messages, or the kernel. netstat drop counts or syslog-ng statistics should help decide that. ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Wed, Jul 20, 2011 at 7:11 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2011-07-20 at 11:14 +0200, maxime.denier@orange-ftgroup.com wrote:
Hello,
I have recently installed syslog-ng OSE 3.1 as log collector and I face a problem. A great number of logs arrive on the server, but a little part of them arrive in the destination files, but all the destination files have some logs on it. I have enabled the verbose mode and I see this: Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file writer;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv'
Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination files;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination files;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/peony2/201107202400.csv'
Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='31' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/decca2/201107202400.csv'
Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='19' Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file writer;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/hyenne2/201107202400.csv'
Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file writer;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/olive2/201107202400.csv'
Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination files;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out, reaping;
template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv'
Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd; fd='24'
I haven't found information about the root cause of these timed out.
These only indicate that syslog-ng is properly garbage-collecting destination files that receive no data. This is not an error, that's why you only get this if you enable --debug / --verbose (i'm not sure which one).
This seems to be a writing problem. Before using syslog-ng, Logs were processed by a application owned by the firewall publisher on the same type of hardware without this great number of log lost.
Is this UDP? syslog-ng doesn't increase udp receive buffer sizes unless explicitly told so, using the so-rcvbuf() option on the udp source. you probably need to increase that.
But I'd recommend not using udp, as that can cause a lot of lost messages easily. (if you create a simple loop that sends udp frames to the syslog receiver, you can easily see as much as 97% messages lost!) easy DoS.
If any body have already faced this problem and have a solution, it would be wonderful.
There was a guy on this list, who has published charts and numbers how he had to tune the various buffering options. I forgot his name, but if you google syslog-ng, udp buffer size, you'll probably find it.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Clayton Dukes
-
maxime.denier＠orange-ftgroup.com