[syslog-ng] syslog-ng 3.38 UDP errors and missing messages

[Balazs Scheidler]

Hi,

There are a couple of things I have in mind to improve this, but sometimes
it's easier to be more interactive than email.

We can chat on this discord https://discord.gg/qmq53uBm2c and try to find
me (bazsi77 is my nick there). I am usually found on Reddit and Reddit chat
as well https://www.reddit.com/user/bazsi771/

Anyway, couple of ideas/questions:

* what's your EPS rate? I can see 200000 messages on a 5 minute interval,
which translates to 666 EPS, does that sounds right?
* your UDP settings look legit: flow control is enabled, but you have a
high log-iw-size(). This means that flow control should not clog you down
if the destination does not accept messages for more than 375 seconds
(250000/666)
* you have use-dns() enabled, which means that syslog-ng will do a DNS
lookup for incoming logs, which might clog the system while names are being
resolved. DNS is cached though.
* you have reliable(yes) set in the queue, you should be fine with
reliable(no) and that should improve disk buffering performance a lot
* you could implement UDP source load balancing, by using so-reuseport()
and you could even add eBPF load balancing. But 666 messages a second is
not a disaster, so a single source code work.

I would look at more metrics and over time. It is important to see what
happens on that syslog server. Since your UDP settings seem to be ok, I'd
check these:

1) does your destination consume messages at the required rate? I'd start
recording the destination stats  and see if the processed counter is moving
or not
2) delay metrics, how much time a message spends in the disk buffer.
3) number of messages stored in the disk buffer and how they change over
time

BTW: There are a few blog posts about some of these on the axoflow.com
website.

Balazs

On Tue, Feb 11, 2025 at 2:27 PM Taavi Ansper <taavi.ansper at cyber.ee> wrote:

> Hi
>
> I am running a syslog server that is having problems getting all of the
> data via UDP. I have gone through most of the tips for making the
> performance
> better, but something is still amiss.
>
> 1. The syslog-ng server is running in an VM. 8vcpu and 8gb RAM. Debian 12.
>
> 2. After restarting or starting the server the UDP errors are tamed.
>
> nstat -asz | grep Error
> UdpInErrors                     0            0.0
> UdpRcvbufErrors                 0            0.0
>
> After 15 minutes or so the errors start to appear. (This amount of errors
> is after some time.)
>
> UdpInErrors                     6734175            0.0
> UdpRcvbufErrors                 6734175            0.0
>
> After some time it starts to receive the packets again. I can see all of
> this clearly in my graylog graphics that sucks that log data in. (added
> picture) https://imgur.com/a/eeRpovh
>
> It goes like a sinewave up and down every 15 minutes or so.
>
> My config looks like this.
>
> options {
>    ts-format(iso);
>    frac-digits(6);
>    keep-timestamp(no);
>    chain-hostnames(no);
>    keep-hostname(no);
>    use-dns(yes);
>    use-fqdn(yes);
>    create-dirs(yes);
>    dir-owner(root);
>    dir-group(adm);
>    dir-perm(0750);
>    owner(root);
>    group(adm);
>    perm(0640);
>    log-fifo-size(100000);
> };
>
> source clients {
>    network(
>      transport(udp)
>      so-rcvbuf(32MiB)
>      log-iw-size(250k)
>      log-fetch-limit(10k)
>    );
>    network(
>      transport(tcp)
>      max-connections(100)
>      so-rcvbuf(32MiB)
>      log-iw-size(250k)
>      log-fetch-limit(10k)
>    );
> };
>
> source forward {
>    network(
>      port(6514)
>      transport(tcp)
>      keep-timestamp(yes)
>      keep-hostname(yes)
>    );
> };
>
> destination file {
>    file("/srv/log/${HOST}/${FACILITY}.log");
> };
>
> destination forward {
>    network(
>      "192.168.X.X"
>      port(6514)
>      transport(tcp)
>      so-keepalive(yes)
>      disk-buffer(
>        mem-buf-size(512M)
>        disk-buf-size(2048M)
>        reliable(yes)
>      )
>    );
> };
>
> destination graylog {
>    network(
>      "192.168.X.X"
>      port(5140)
>      flags(syslog-protocol)
>      transport(tcp)
>      so-keepalive(yes)
>      disk-buffer(
>        mem-buf-size(512M)
>        disk-buf-size(2048M)
>        reliable(yes)
>      )
>    );
> };
>
> filter graylog {
>    host("^gw-example-")
> };
>
>
> log {
>    source(clients);
>    destination(file);
>    destination(forward);
>    flags(flow-control);
> };
>
> log {
>    source(forward);
>    destination(file);
>    flags(flow-control);
> };
>
> log {
>    source(clients);
>    filter(graylog);
>    destination(graylog);
>    flags(flow-control);
> };
>
> My rmem_max is set like so:
>
> sysctl net.core.rmem_max
> net.core.rmem_max = 33445532
>
> Any suggestions?
>
> Best regards
> --
> Taavi Ansper
> taavi.ansper at cyber.ee
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-- 
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20250211/82e19b2e/attachment.htm>