Hello everyone, we’ve been experiencing packet loss on syslog-ng collectors (these are open stack VMs), lately we have increased ingestion and that refelcted packet loss ( especially UDP). Tuning parameters didn’t help. If anyone here on the list has experienced a similar issue please share your resolution or work around. We are at 3.9.1, we are planning to update to 3.17.x for better data handling/other bug fixes. As an alternate approach we have increased the VM capacity ( no of VMs). If there is any that I missed, please point. Any thoughts? Thank you, Vijay Amrut. -- Thanks, Vijay Amrut.
Hi Vijay, We're running 3.9.1 too with UDP. Have you tried increasing so-rcvbuf?
Thank you, Fabien. I have tuned parameters at OS level, these being VMs didn't help much, can this be set at app level? if so, how? On Wed, Sep 12, 2018 at 7:09 AM Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Vijay,
We're running 3.9.1 too with UDP. Have you tried increasing so-rcvbuf?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Thanks, Vijay Amrut.
Have you been able to isolate what is dropping the packets? If it is the kernel / driver (netstat -su) then tuning the app probably won't help much. You might look at "pinning" the resources if you can do that in your virtual environment. If you have load balancers available, you might be better off to simply point the UDP syslogs at a virtual IP and have the load balancer distribute across a pool of syslog servers. Good luck, Jim On Wed, Sep 12, 2018 at 1:27 PM vijay amruth <vijayamruth@gmail.com> wrote:
Thank you, Fabien. I have tuned parameters at OS level, these being VMs didn't help much, can this be set at app level? if so, how?
On Wed, Sep 12, 2018 at 7:09 AM Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Vijay,
We're running 3.9.1 too with UDP. Have you tried increasing so-rcvbuf?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Thanks, Vijay Amrut.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Wed, Sep 12, 2018 at 10:27:40AM -0700, vijay amruth wrote:
Thank you, Fabien. I have tuned parameters at OS level, these being VMs didn't help much, can this be set at app level? if so, how?
Yes, you can set it in the source driver options [1]. However you may also have to raise the kernel's hard limit (net.core.rmem_max) to at least the same. -- [1] https://www.syslog-ng.com/technical-documents/doc/syslog-ng-premium-edition/...
participants (3)
-
Fabien Wernli
-
Jim Hendrick
-
vijay amruth