<div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>There are a couple of things I have in mind to improve this, but sometimes it's easier to be more interactive than email. </div><div><br></div><div>We can chat on this discord <a href="https://discord.gg/qmq53uBm2c">https://discord.gg/qmq53uBm2c</a> and try to find me (bazsi77 is my nick there). I am usually found on Reddit and Reddit chat as well <a href="https://www.reddit.com/user/bazsi771/">https://www.reddit.com/user/bazsi771/</a></div><div><br></div><div>Anyway, couple of ideas/questions:</div><div><br></div><div>* what's your EPS rate? I can see 200000 messages on a 5 minute interval, which translates to 666 EPS, does that sounds right?</div><div>* your UDP settings look legit: flow control is enabled, but you have a high log-iw-size(). This means that flow control should not clog you down if the destination does not accept messages for more than 375 seconds (250000/666)</div><div>* you have use-dns() enabled, which means that syslog-ng will do a DNS lookup for incoming logs, which might clog the system while names are being resolved. DNS is cached though.</div><div>* you have reliable(yes) set in the queue, you should be fine with reliable(no) and that should improve disk buffering performance a lot</div><div>* you could implement UDP source load balancing, by using so-reuseport() and you could even add eBPF load balancing. But 666 messages a second is not a disaster, so a single source code work.</div><div><br></div><div>I would look at more metrics and over time. It is important to see what happens on that syslog server. Since your UDP settings seem to be ok, I'd check these:</div><div><br></div><div>1) does your destination consume messages at the required rate? I'd start recording the destination stats and see if the processed counter is moving or not</div><div>2) delay metrics, how much time a message spends in the disk buffer.</div><div>3) number of messages stored in the disk buffer and how they change over time</div><div><br></div></div><div>BTW: There are a few blog posts about some of these on the <a href="http://axoflow.com">axoflow.com</a> website.</div><div><br></div><div>Balazs</div><div><br></div><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Feb 11, 2025 at 2:27 PM Taavi Ansper <<a href="mailto:taavi.ansper@cyber.ee">taavi.ansper@cyber.ee</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi<br>
<br>
I am running a syslog server that is having problems getting all of the data via UDP. I have gone through most of the tips for making the performance <br>
better, but something is still amiss.<br>
<br>
1. The syslog-ng server is running in an VM. 8vcpu and 8gb RAM. Debian 12.<br>
<br>
2. After restarting or starting the server the UDP errors are tamed.<br>
<br>
nstat -asz | grep Error<br>
UdpInErrors 0 0.0<br>
UdpRcvbufErrors 0 0.0<br>
<br>
After 15 minutes or so the errors start to appear. (This amount of errors is after some time.)<br>
<br>
UdpInErrors 6734175 0.0<br>
UdpRcvbufErrors 6734175 0.0<br>
<br>
After some time it starts to receive the packets again. I can see all of this clearly in my graylog graphics that sucks that log data in. (added <br>
picture) <a href="https://imgur.com/a/eeRpovh" rel="noreferrer" target="_blank">https://imgur.com/a/eeRpovh</a><br>
<br>
It goes like a sinewave up and down every 15 minutes or so.<br>
<br>
My config looks like this.<br>
<br>
options {<br>
ts-format(iso);<br>
frac-digits(6);<br>
keep-timestamp(no);<br>
chain-hostnames(no);<br>
keep-hostname(no);<br>
use-dns(yes);<br>
use-fqdn(yes);<br>
create-dirs(yes);<br>
dir-owner(root);<br>
dir-group(adm);<br>
dir-perm(0750);<br>
owner(root);<br>
group(adm);<br>
perm(0640);<br>
log-fifo-size(100000);<br>
};<br>
<br>
source clients {<br>
network(<br>
transport(udp)<br>
so-rcvbuf(32MiB)<br>
log-iw-size(250k)<br>
log-fetch-limit(10k)<br>
);<br>
network(<br>
transport(tcp)<br>
max-connections(100)<br>
so-rcvbuf(32MiB)<br>
log-iw-size(250k)<br>
log-fetch-limit(10k)<br>
);<br>
};<br>
<br>
source forward {<br>
network(<br>
port(6514)<br>
transport(tcp)<br>
keep-timestamp(yes)<br>
keep-hostname(yes)<br>
);<br>
};<br>
<br>
destination file {<br>
file("/srv/log/${HOST}/${FACILITY}.log");<br>
};<br>
<br>
destination forward {<br>
network(<br>
"192.168.X.X"<br>
port(6514)<br>
transport(tcp)<br>
so-keepalive(yes)<br>
disk-buffer(<br>
mem-buf-size(512M)<br>
disk-buf-size(2048M)<br>
reliable(yes)<br>
)<br>
);<br>
};<br>
<br>
destination graylog {<br>
network(<br>
"192.168.X.X"<br>
port(5140)<br>
flags(syslog-protocol)<br>
transport(tcp)<br>
so-keepalive(yes)<br>
disk-buffer(<br>
mem-buf-size(512M)<br>
disk-buf-size(2048M)<br>
reliable(yes)<br>
)<br>
);<br>
};<br>
<br>
filter graylog {<br>
host("^gw-example-")<br>
};<br>
<br>
<br>
log {<br>
source(clients);<br>
destination(file);<br>
destination(forward);<br>
flags(flow-control);<br>
};<br>
<br>
log {<br>
source(forward);<br>
destination(file);<br>
flags(flow-control);<br>
};<br>
<br>
log {<br>
source(clients);<br>
filter(graylog);<br>
destination(graylog);<br>
flags(flow-control);<br>
};<br>
<br>
My rmem_max is set like so:<br>
<br>
sysctl net.core.rmem_max<br>
net.core.rmem_max = 33445532<br>
<br>
Any suggestions?<br>
<br>
Best regards<br>
-- <br>
Taavi Ansper<br>
<a href="mailto:taavi.ansper@cyber.ee" target="_blank">taavi.ansper@cyber.ee</a><br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Bazsi</div></div>