[syslog-ng] syslog-ng 3.38 UDP errors and missing messages

[Taavi Ansper]

Hi

I am running a syslog server that is having problems getting all of the data via UDP. I have gone through most of the tips for making the performance 
better, but something is still amiss.

1. The syslog-ng server is running in an VM. 8vcpu and 8gb RAM. Debian 12.

2. After restarting or starting the server the UDP errors are tamed.

nstat -asz | grep Error
UdpInErrors                     0            0.0
UdpRcvbufErrors                 0            0.0

After 15 minutes or so the errors start to appear. (This amount of errors is after some time.)

UdpInErrors                     6734175            0.0
UdpRcvbufErrors                 6734175            0.0

After some time it starts to receive the packets again. I can see all of this clearly in my graylog graphics that sucks that log data in. (added 
picture) https://imgur.com/a/eeRpovh

It goes like a sinewave up and down every 15 minutes or so.

My config looks like this.

options {
   ts-format(iso);
   frac-digits(6);
   keep-timestamp(no);
   chain-hostnames(no);
   keep-hostname(no);
   use-dns(yes);
   use-fqdn(yes);
   create-dirs(yes);
   dir-owner(root);
   dir-group(adm);
   dir-perm(0750);
   owner(root);
   group(adm);
   perm(0640);
   log-fifo-size(100000);
};

source clients {
   network(
     transport(udp)
     so-rcvbuf(32MiB)
     log-iw-size(250k)
     log-fetch-limit(10k)
   );
   network(
     transport(tcp)
     max-connections(100)
     so-rcvbuf(32MiB)
     log-iw-size(250k)
     log-fetch-limit(10k)
   );
};

source forward {
   network(
     port(6514)
     transport(tcp)
     keep-timestamp(yes)
     keep-hostname(yes)
   );
};

destination file {
   file("/srv/log/${HOST}/${FACILITY}.log");
};

destination forward {
   network(
     "192.168.X.X"
     port(6514)
     transport(tcp)
     so-keepalive(yes)
     disk-buffer(
       mem-buf-size(512M)
       disk-buf-size(2048M)
       reliable(yes)
     )
   );
};

destination graylog {
   network(
     "192.168.X.X"
     port(5140)
     flags(syslog-protocol)
     transport(tcp)
     so-keepalive(yes)
     disk-buffer(
       mem-buf-size(512M)
       disk-buf-size(2048M)
       reliable(yes)
     )
   );
};

filter graylog {
   host("^gw-example-")
};


log {
   source(clients);
   destination(file);
   destination(forward);
   flags(flow-control);
};

log {
   source(forward);
   destination(file);
   flags(flow-control);
};

log {
   source(clients);
   filter(graylog);
   destination(graylog);
   flags(flow-control);
};

My rmem_max is set like so:

sysctl net.core.rmem_max
net.core.rmem_max = 33445532

Any suggestions?

Best regards
-- 
Taavi Ansper
taavi.ansper at cyber.ee