I am trying to send the output from an application log to a LogLogic device, and it appears we are getting UDP drops repeatedly. Any ideas on what I can do to fix them? Aug 25 15:24:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6376', processed='center(received)=6376', processed='destination(d_messages)=6374', processed='destination(d_bo ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=1', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=1', processed='destination(d_cons)=0', processed='des tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=2', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0' Aug 25 15:34:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6378', processed='center(received)=6378', processed='destination(d_messages)=6374', processed='destination(d_bo ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=2', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=2', processed='destination(d_cons)=0', processed='des tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=4', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0' Jamie
Hello, On Thu, Aug 25, 2011 at 10:38 PM, Aldrich, Jamie S <JSAldrich@pier1.com> wrote:
I am trying to send the output from an application log to a LogLogic device, and it appears we are getting UDP drops repeatedly. Any ideas on what I can do to fix them?
Aug 25 15:24:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6376', processed='center(received)=6376', processed='destination(d_messages)=6374', processed='destination(d_bo
ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=1', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=1', processed='destination(d_cons)=0', processed='des
tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=2', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0'
Aug 25 15:34:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6378', processed='center(received)=6378', processed='destination(d_messages)=6374', processed='destination(d_bo
ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=2', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=2', processed='destination(d_cons)=0', processed='des
tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=4', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0'
I don't see any proof of dropped messages Are you sure that your syslog-ng config is OK? It's somewhat unexpected that a server only gets 2 log messages in 10 mins so I'm more or less sure that some vital log sources are missing from your config. Regards, Sandor
We are not writing these specific logs to the /var/adm/messages, but to a LogLogic devices. Here is the syslog-ng.conf file part that handles these logs. source s_file { file("/psfs_logs/APPSRV_current.LOG" flags(no-parse)); }; destination d_messages{ udp("10.13.33.11"); }; log { source(s_file); destination(d_messages); }; Jamie -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Sandor Geller Sent: Friday, August 26, 2011 4:05 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] UDP Dropping packets Hello, On Thu, Aug 25, 2011 at 10:38 PM, Aldrich, Jamie S <JSAldrich@pier1.com> wrote:
I am trying to send the output from an application log to a LogLogic device, and it appears we are getting UDP drops repeatedly. Any ideas on what I can do to fix them?
Aug 25 15:24:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6376', processed='center(received)=6376', processed='destination(d_messages)=6374', processed='destination(d_bo
ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=1', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=1', processed='destination(d_cons)=0', processed='des
tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=2', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0'
Aug 25 15:34:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6378', processed='center(received)=6378', processed='destination(d_messages)=6374', processed='destination(d_bo
ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=2', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=2', processed='destination(d_cons)=0', processed='des
tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=4', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0'
I don't see any proof of dropped messages Are you sure that your syslog-ng config is OK? It's somewhat unexpected that a server only gets 2 log messages in 10 mins so I'm more or less sure that some vital log sources are missing from your config. Regards, Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, 2011-08-26 at 08:56 -0500, Aldrich, Jamie S wrote:
We are not writing these specific logs to the /var/adm/messages, but to a LogLogic devices. Here is the syslog-ng.conf file part that handles these logs.
source s_file { file("/psfs_logs/APPSRV_current.LOG" flags(no-parse)); };
destination d_messages{ udp("10.13.33.11"); };
log { source(s_file); destination(d_messages); };
UDP is not reliable, and you could be surprised how much it is unreliable. I've seen udp transports to drop over 90% of the traffic. It can be improved somewhat by increasing the receive buffer size (so_rcvbuf() option in syslog-ng, but kernel limits may have to be adjusted as well). Google for udp receive buffer syslog-ng, and you'll get a number of pages that describe the issue. -- Bazsi
Would using "tcp" be a worse option? Jamie -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Sunday, August 28, 2011 1:29 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] UDP Dropping packets On Fri, 2011-08-26 at 08:56 -0500, Aldrich, Jamie S wrote:
We are not writing these specific logs to the /var/adm/messages, but to a LogLogic devices. Here is the syslog-ng.conf file part that handles these logs.
source s_file { file("/psfs_logs/APPSRV_current.LOG" flags(no-parse)); };
destination d_messages{ udp("10.13.33.11"); };
log { source(s_file); destination(d_messages); };
UDP is not reliable, and you could be surprised how much it is unreliable. I've seen udp transports to drop over 90% of the traffic. It can be improved somewhat by increasing the receive buffer size (so_rcvbuf() option in syslog-ng, but kernel limits may have to be adjusted as well). Google for udp receive buffer syslog-ng, and you'll get a number of pages that describe the issue. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
While I've certainly seen UDP drop issues, setting log_fifo_size(30000) has allowed my commodity systems to receive over 50k events/sec without drops on standard UDP, so unless you are handling an extreme amount of logging or logging to a high-latency destination (e.g. SQL), UDP should do just fine. On Mon, Aug 29, 2011 at 10:48 AM, Aldrich, Jamie S <JSAldrich@pier1.com> wrote:
Would using "tcp" be a worse option?
Jamie
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Sunday, August 28, 2011 1:29 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] UDP Dropping packets
On Fri, 2011-08-26 at 08:56 -0500, Aldrich, Jamie S wrote:
We are not writing these specific logs to the /var/adm/messages, but to a LogLogic devices. Here is the syslog-ng.conf file part that handles these logs.
source s_file { file("/psfs_logs/APPSRV_current.LOG" flags(no-parse)); };
destination d_messages{ udp("10.13.33.11"); };
log { source(s_file); destination(d_messages); };
UDP is not reliable, and you could be surprised how much it is unreliable. I've seen udp transports to drop over 90% of the traffic.
It can be improved somewhat by increasing the receive buffer size (so_rcvbuf() option in syslog-ng, but kernel limits may have to be adjusted as well).
Google for udp receive buffer syslog-ng, and you'll get a number of pages that describe the issue.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Is there a max size you can set the log_fifo_size? Jamie -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Martin Holste Sent: Monday, August 29, 2011 11:56 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] UDP Dropping packets While I've certainly seen UDP drop issues, setting log_fifo_size(30000) has allowed my commodity systems to receive over 50k events/sec without drops on standard UDP, so unless you are handling an extreme amount of logging or logging to a high-latency destination (e.g. SQL), UDP should do just fine. On Mon, Aug 29, 2011 at 10:48 AM, Aldrich, Jamie S <JSAldrich@pier1.com> wrote:
Would using "tcp" be a worse option?
Jamie
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Sunday, August 28, 2011 1:29 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] UDP Dropping packets
On Fri, 2011-08-26 at 08:56 -0500, Aldrich, Jamie S wrote:
We are not writing these specific logs to the /var/adm/messages, but to a LogLogic devices. Here is the syslog-ng.conf file part that handles these logs.
source s_file { file("/psfs_logs/APPSRV_current.LOG" flags(no-parse)); };
destination d_messages{ udp("10.13.33.11"); };
log { source(s_file); destination(d_messages); };
UDP is not reliable, and you could be surprised how much it is unreliable. I've seen udp transports to drop over 90% of the traffic.
It can be improved somewhat by increasing the receive buffer size (so_rcvbuf() option in syslog-ng, but kernel limits may have to be adjusted as well).
Google for udp receive buffer syslog-ng, and you'll get a number of pages that describe the issue.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Aldrich, Jamie S
-
Balazs Scheidler
-
Martin Holste
-
Sandor Geller