No subject
Mon Feb 21 10:55:38 CET 2011
Hmm. the numbers you are seeing are indeed low, with sufficient buffer
sizes I could get up to the 20k message/sec range with syslog-ng,
although it's been a while I last tested it.
What I'd recommend is to calculate how much _bytes_ the message rate you
are generating means.
If you generate 2000 messages, 300 byte each (loggen default IIRC),
that's 600000 bytes every second. syslog-ng is single threaded, thus the
latency to write to the disk applies. This means that it may take some
time for syslog-ng to care about its source, if it is busy writing out
messages. This is the #1 reason why I want to work on multithreading.
With a flow controlled source, syslog-ng is able to do about 70-75k
msg/sec. But not with UDP.
In order to improve the numbers, I'd:
1) increase the receive buffer rate to 3-5 seconds (e.g. 3-5MB, not just
0.5)
2) increase log_fetch_limit() to a larger value, this controls how much
messages syslog-ng fetches in each poll iteration. Increase this to
3-500
3) increase log_fifo_size() for the destination, by taking the
fetch_limit values for each sources feeding the destination (so if you
have two sources, each with 1000 fetch limit, then the destination queue
should be _at least_ 2000, preferably rounded to the next order of
magnitude (e.g. with 2x1000 fetch-limits, increase fifo to 10000)
You haven't included in your email whether syslog-ng itself is dropping
messages, or the kernel. netstat drop counts or syslog-ng statistics
should help decide that.
______________________________________________________________
Clayton Dukes
______________________________________________________________
On Wed, Jul 20, 2011 at 7:11 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> On Wed, 2011-07-20 at 11:14 +0200, maxime.denier at orange-ftgroup.com
> wrote:
> > Hello,
> >
> > I have recently installed syslog-ng OSE 3.1 as log collector and I
> > face a problem.
> > A great number of logs arrive on the server, but a little part of them
> > arrive in the destination files, but all the destination files have
> > some logs on it.
> > I have enabled the verbose mode and I see this:
> > Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file
> > writer;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',
> filename='/var/logs/zidane2/201107202400.csv'
> > Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination
> > files;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
> > Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination
> > files;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
> > Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out,
> > reaping;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',
> filename='/var/logs/peony2/201107202400.csv'
> > Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd;
> > fd='31'
> > Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out,
> > reaping;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',
> filename='/var/logs/decca2/201107202400.csv'
> > Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd;
> > fd='19'
> > Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file
> > writer;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',
> filename='/var/logs/hyenne2/201107202400.csv'
> > Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file
> > writer;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',
> filename='/var/logs/olive2/201107202400.csv'
> > Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination
> > files;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'
> > Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out,
> > reaping;
> >
> template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',
> filename='/var/logs/zidane2/201107202400.csv'
> > Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd;
> > fd='24'
> >
> > I haven't found information about the root cause of these timed out.
>
> These only indicate that syslog-ng is properly garbage-collecting
> destination files that receive no data. This is not an error, that's why
> you only get this if you enable --debug / --verbose (i'm not sure which
> one).
>
> > This seems to be a writing problem.
> > Before using syslog-ng, Logs were processed by a application owned by
> > the firewall publisher on the same type of hardware without this great
> > number of log lost.
>
> Is this UDP? syslog-ng doesn't increase udp receive buffer sizes unless
> explicitly told so, using the so-rcvbuf() option on the udp source. you
> probably need to increase that.
>
> But I'd recommend not using udp, as that can cause a lot of lost
> messages easily. (if you create a simple loop that sends udp frames to
> the syslog receiver, you can easily see as much as 97% messages lost!)
> easy DoS.
>
> >
> > If any body have already faced this problem and have a solution, it
> > would be wonderful.
>
> There was a guy on this list, who has published charts and numbers how
> he had to tune the various buffering options. I forgot his name, but if
> you google syslog-ng, udp buffer size, you'll probably find it.
>
>
> --
> Bazsi
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
--0015175885129c7e9b04a87f4a6c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hi,<div>I did some work with UDP buffer testing and wrote it into my Wiki a=
t=A0<a href=3D"http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.1=
#UDP_Buffers">http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.1#=
UDP_Buffers</a></div>
<div><br></div><div>The other guy that Baszi refers to did some nice graphs=
, but it was a temporary URL.</div><div>At the time, I thought the email th=
read would be useful, so I copied it to text and saved it locally. Here'=
;s that thread:</div>
<div><br></div><div>---</div><div><div>I've run a series of tests again=
st our log server, using</div><div>loggen-generated logging rates of 100, 1=
000, 2000, 4000, 8000, 16000,</div><div>and 32000 messages/sec, and measure=
d the results for each rate using</div>
<div>socket buffer sizes ranging from 128KB to 16M. =A0The results show,</d=
iv><div>essentially, what buffer size you need to meet a target rate of</di=
v><div>message logging.</div><div><br></div><div>So I'm happy now, and =
I'm getting the sort of rates I expect to be</div>
<div>able to support. =A0I figured other folks might like the data. =A0I=
9;ve put</div><div>the raw data online as well as a graph of the results:</=
div><div><br></div><div>=A0<a href=3D"http://drop.io/syslog_ng">http://drop=
.io/syslog_ng</a></div>
<div><br></div><div>The graph shows so_rcvbuf() size along the X axis, and =
packet loss</div><div>along the Y axis.</div><div><br></div><div><br></div>=
<div>From Baszi:</div><div>Hmm. the numbers you are seeing are indeed low, =
with sufficient buffer</div>
<div>sizes I could get up to the 20k message/sec range with syslog-ng,</div=
><div>although it's been a while I last tested it.</div><div><br></div>=
<div>What I'd recommend is to calculate how much _bytes_ the message ra=
te you</div>
<div>are generating means.</div><div><br></div><div>If you generate 2000 me=
ssages, 300 byte each (loggen default IIRC),</div><div>that's 600000 by=
tes every second. syslog-ng is single threaded, thus the</div><div>latency =
to write to the disk applies. This means that it may take some</div>
<div>time for syslog-ng to care about its source, if it is busy writing out=
</div><div>messages. This is the #1 reason why I want to work on multithrea=
ding.</div><div>With a flow controlled source, syslog-ng is able to do abou=
t 70-75k</div>
<div>msg/sec. But not with UDP.</div><div><br></div><div>In order to improv=
e the numbers, I'd:</div><div><br></div><div>1) increase the receive bu=
ffer rate to 3-5 seconds (e.g. 3-5MB, not just</div><div>0.5)</div><div>
<br></div><div>2) increase log_fetch_limit() to a larger value, this contro=
ls how much</div><div>messages syslog-ng fetches in each poll iteration. In=
crease this to</div><div>3-500</div><div><br></div><div>3) increase log_fif=
o_size() for the destination, by taking the</div>
<div>fetch_limit values for each sources feeding the destination (so if you=
</div><div>have two sources, each with 1000 fetch limit, then the destinati=
on queue</div><div>should be _at least_ 2000, preferably rounded to the nex=
t order of</div>
<div>magnitude (e.g. with 2x1000 fetch-limits, increase fifo to 10000)</div=
><div><br></div><div>You haven't included in your email whether syslog-=
ng itself is dropping</div><div>messages, or the kernel. netstat drop count=
s or syslog-ng statistics</div>
<div>should help decide that.</div><div><br></div><div><br></div>__________=
____________________________________________________ <br><br>Clayton Dukes<=
br>______________________________________________________________<br>
<br><br><div class=3D"gmail_quote">On Wed, Jul 20, 2011 at 7:11 AM, Balazs =
Scheidler <span dir=3D"ltr"><<a href=3D"mailto:bazsi at balabit.hu">bazsi at b=
alabit.hu</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
On Wed, 2011-07-20 at 11:14 +0200, <a href=3D"mailto:maxime.denier at orange-f=
tgroup.com">maxime.denier at orange-ftgroup.com</a><br>
wrote:<br>
<div><div></div><div class=3D"h5">> Hello,<br>
><br>
> I have recently installed syslog-ng OSE 3.1 as log collector and I<br>
> face a problem.<br>
> A great number of logs arrive on the server, but a little part of them=
<br>
> arrive in the destination files, but all the destination files have<br=
>
> some logs on it.<br>
> I have enabled the verbose mode and I see this:<br>
> Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file=
<br>
> writer;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',=
filename=3D'/var/logs/zidane2/201107202400.csv'<br>
> Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination<br=
>
> files;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'<=
br>
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination<br=
>
> files;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'<=
br>
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out,<br>
> reaping;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',=
filename=3D'/var/logs/peony2/201107202400.csv'<br>
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd;<br>
> fd=3D'31'<br>
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out,<br>
> reaping;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',=
filename=3D'/var/logs/decca2/201107202400.csv'<br>
> Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd;<br>
> fd=3D'19'<br>
> Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file=
<br>
> writer;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',=
filename=3D'/var/logs/hyenne2/201107202400.csv'<br>
> Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file=
<br>
> writer;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',=
filename=3D'/var/logs/olive2/201107202400.csv'<br>
> Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination<br=
>
> files;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv'<=
br>
> Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out,<br>
> reaping;<br>
> template=3D'/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED=
_TIME.YEAR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv',=
filename=3D'/var/logs/zidane2/201107202400.csv'<br>
> Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd;<br>
> fd=3D'24'<br>
><br>
> I haven't found information about the root cause of these timed ou=
t.<br>
<br>
</div></div>These only indicate that syslog-ng is properly garbage-collecti=
ng<br>
destination files that receive no data. This is not an error, that's wh=
y<br>
you only get this if you enable --debug / --verbose (i'm not sure which=
<br>
one).<br>
<div class=3D"im"><br>
> This seems to be a writing problem.<br>
> Before using syslog-ng, Logs were processed by a application owned by<=
br>
> the firewall publisher on the same type of hardware without this great=
<br>
> number of log lost.<br>
<br>
</div>Is this UDP? syslog-ng doesn't increase udp receive buffer sizes =
unless<br>
explicitly told so, using the so-rcvbuf() option on the udp source. you<br>
probably need to increase that.<br>
<br>
But I'd recommend not using udp, as that can cause a lot of lost<br>
messages easily. (if you create a simple loop that sends udp frames to<br>
the syslog receiver, you can easily see as much as 97% messages lost!)<br>
easy DoS.<br>
<div class=3D"im"><br>
><br>
> If any body have already faced this problem and have a solution, it<br=
>
> would be wonderful.<br>
<br>
</div>There was a guy on this list, who has published charts and numbers ho=
w<br>
he had to tune the various buffering options. I forgot his name, but if<br>
you google syslog-ng, udp buffer size, you'll probably find it.<br>
<font color=3D"#888888"><br>
<br>
--<br>
Bazsi<br>
<br>
<br>
___________________________________________________________________________=
___<br>
Member info: <a href=3D"https://lists.balabit.hu/mailman/listinfo/syslog-ng=
" target=3D"_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>=
<br>
Documentation: <a href=3D"http://www.balabit.com/support/documentation/?pro=
duct=3Dsyslog-ng" target=3D"_blank">http://www.balabit.com/support/document=
ation/?product=3Dsyslog-ng</a><br>
FAQ: <a href=3D"http://www.balabit.com/wiki/syslog-ng-faq" target=3D"_blank=
">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</font></blockquote></div><br></div>
--0015175885129c7e9b04a87f4a6c--
More information about the syslog-ng
mailing list