Hi,

There are a couple of things I have in mind to improve this, but sometimes it's easier to be more interactive than email. 

We can chat on this discord https://discord.gg/qmq53uBm2c and try to find me (bazsi77 is my nick there). I am usually found on Reddit and Reddit chat as well https://www.reddit.com/user/bazsi771/

Anyway, couple of ideas/questions:

* what's your EPS rate? I can see 200000 messages on a 5 minute interval, which translates to 666 EPS, does that sounds right?
* your UDP settings look legit: flow control is enabled, but you have a high log-iw-size(). This means that flow control should not clog you down if the destination does not accept messages for more than 375 seconds (250000/666)
* you have use-dns() enabled, which means that syslog-ng will do a DNS lookup for incoming logs, which might clog the system while names are being resolved. DNS is cached though.
* you have reliable(yes) set in the queue, you should be fine with reliable(no) and that should improve disk buffering performance a lot
* you could implement UDP source load balancing, by using so-reuseport() and you could even add eBPF load balancing. But 666 messages a second is not a disaster, so a single source code work.

I would look at more metrics and over time. It is important to see what happens on that syslog server. Since your UDP settings seem to be ok, I'd check these:

1) does your destination consume messages at the required rate? I'd start recording the destination stats  and see if the processed counter is moving or not
2) delay metrics, how much time a message spends in the disk buffer.
3) number of messages stored in the disk buffer and how they change over time

BTW: There are a few blog posts about some of these on the axoflow.com website.

Balazs

On Tue, Feb 11, 2025 at 2:27 PM Taavi Ansper <taavi.ansper@cyber.ee> wrote:
Hi

I am running a syslog server that is having problems getting all of the data via UDP. I have gone through most of the tips for making the performance
better, but something is still amiss.

1. The syslog-ng server is running in an VM. 8vcpu and 8gb RAM. Debian 12.

2. After restarting or starting the server the UDP errors are tamed.

nstat -asz | grep Error
UdpInErrors                     0            0.0
UdpRcvbufErrors                 0            0.0

After 15 minutes or so the errors start to appear. (This amount of errors is after some time.)

UdpInErrors                     6734175            0.0
UdpRcvbufErrors                 6734175            0.0

After some time it starts to receive the packets again. I can see all of this clearly in my graylog graphics that sucks that log data in. (added
picture) https://imgur.com/a/eeRpovh

It goes like a sinewave up and down every 15 minutes or so.

My config looks like this.

options {
   ts-format(iso);
   frac-digits(6);
   keep-timestamp(no);
   chain-hostnames(no);
   keep-hostname(no);
   use-dns(yes);
   use-fqdn(yes);
   create-dirs(yes);
   dir-owner(root);
   dir-group(adm);
   dir-perm(0750);
   owner(root);
   group(adm);
   perm(0640);
   log-fifo-size(100000);
};

source clients {
   network(
     transport(udp)
     so-rcvbuf(32MiB)
     log-iw-size(250k)
     log-fetch-limit(10k)
   );
   network(
     transport(tcp)
     max-connections(100)
     so-rcvbuf(32MiB)
     log-iw-size(250k)
     log-fetch-limit(10k)
   );
};

source forward {
   network(
     port(6514)
     transport(tcp)
     keep-timestamp(yes)
     keep-hostname(yes)
   );
};

destination file {
   file("/srv/log/${HOST}/${FACILITY}.log");
};

destination forward {
   network(
     "192.168.X.X"
     port(6514)
     transport(tcp)
     so-keepalive(yes)
     disk-buffer(
       mem-buf-size(512M)
       disk-buf-size(2048M)
       reliable(yes)
     )
   );
};

destination graylog {
   network(
     "192.168.X.X"
     port(5140)
     flags(syslog-protocol)
     transport(tcp)
     so-keepalive(yes)
     disk-buffer(
       mem-buf-size(512M)
       disk-buf-size(2048M)
       reliable(yes)
     )
   );
};

filter graylog {
   host("^gw-example-")
};


log {
   source(clients);
   destination(file);
   destination(forward);
   flags(flow-control);
};

log {
   source(forward);
   destination(file);
   flags(flow-control);
};

log {
   source(clients);
   filter(graylog);
   destination(graylog);
   flags(flow-control);
};

My rmem_max is set like so:

sysctl net.core.rmem_max
net.core.rmem_max = 33445532

Any suggestions?

Best regards
--
Taavi Ansper
taavi.ansper@cyber.ee

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



--
Bazsi