I apologize for what seems to be repost of a rather similar problem but I having looked through the archives and unable to find answers. I am running syslog-ng 3.0.4 on Solaris 10 x86 (64bit). I have about 200 hosts, all running over udp. I have 1 heavy hitter, which is my firewall. I puts about 1500 messages a minute. It seems that syslog-ng is able to handle this amount of traffic but I am having trouble figuring out why I can't seem to make it work that way. I started to notice that I was dropping udp packets by running netstat -s |grep udpInOverflows. I tweaked the udp buffers by setting them to their max ndd -set /dev/udp udp_max_buf 1073741824 ndd -set /dev/udp udp_recv_hiwt 65536 I was still losing packets until I started to tweek my syslog-ng.conf and added the so_rcvbuf entries. The problem is, the logfiles do not reflect that all of the messages are making it. I only know this because the firewall is also logging to another standalone solaris server running standard syslogd and the syslog-ng's firewall's logs are still only getting 1/3 of the logs. Here is my syslog-ng.conf @version:3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on SunOS. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # 20000925 gb@sysfive.com # Global Options options { flush_lines (2000); time_reopen (10); time_sleep (50); log_fifo_size (100000); log_fetch_limit (50000); chain_hostnames (yes); use_dns (yes); use_fqdn (no); keep_hostname (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire(87600); create_dirs (yes); owner(splunk); group(Unix_users); dir_group(Unix_users); perm(0640); dir_perm(0750); stats_level(1); }; ######################################################## # Sources # Sun Remote port source s_sun { udp(ip("xxx.xxx.xxx.101") port(514) so_rcvbuf(1073741824)); }; # Hp Remote port source s_hp { udp(ip("xxx.xxx.xxx.102") port(514) so_rcvbuf(1073741824)); }; # Other Remote port source s_other { udp(ip("xxx.xxx.xxx.103") port(514)); }; # ESX Remote port source s_esx { udp(ip("xxx.xxx.xxx.104") port(514) so_rcvbuf(1073741824)); }; # Linux Remote port source s_linux { udp(ip("xxx.xxx.xxx.105") port(514) so_rcvbuf(1073741824)); }; # Switch Remote port source s_switch { udp(ip("xxx.xxx.xxx.106") port(514) so_rcvbuf(1073741824) ); }; source s_local { internal(); }; ######################################################## # Destinations # Sun destinations destination d_sun_cons { file("/shares/logfiles/syslog/sun/$HOST/console"); }; destination d_sun_mesg { file("/shares/logfiles/syslog/sun/$HOST/messages"); }; destination d_sun_syslog { file("/shares/logfiles/syslog/sun/$HOST/syslog"); }; destination d_sun_auth { file("/shares/logfiles/syslog/sun/$HOST/auth"); }; destination d_sun_audit { file("/shares/logfiles/syslog/sun/$HOST/audit.log"); }; # HP destinations destination d_hp_cons { file("/shares/logfiles/syslog/hp/$HOST/console"); }; destination d_hp_mail { file("/shares/logfiles/syslog/hp/$HOST/mail.log"); }; destination d_hp_syslog { file("/shares/logfiles/syslog/hp/$HOST/syslog.log"); }; # SAN destinations destination d_switch { file("/shares/logfiles/syslog/switches/$HOST/syslog.log"); }; # Other destinations destination d_other { file("/shares/logfiles/syslog/other_devices/$HOST/syslog.log"); }; # ESX destinations destination d_esx_cons { file("/shares/logfiles/syslog/esx/$HOST/console"); }; destination d_esx_mesg { file("/shares/logfiles/syslog/esx/$HOST/messages"); }; destination d_esx_mail { file("/shares/logfiles/syslog/esx/$HOST/maillog"); }; destination d_esx_auth { file("/shares/logfiles/syslog/esx/$HOST/secure"); }; destination d_esx_cron { file("/shares/logfiles/syslog/esx/$HOST/cron.log"); }; # Linux destinations destination d_linux_cons { file("/shares/logfiles/syslog/linux/$HOST/console"); }; destination d_linux_mesg { file("/shares/logfiles/syslog/linux/$HOST/messages"); }; destination d_linux_mail { file("/shares/logfiles/syslog/linux/$HOST/maillog"); }; destination d_linux_auth { file("/shares/logfiles/syslog/linux/$HOST/secure"); }; destination d_linux_cron { file("/shares/logfiles/syslog/linux/$HOST/cron.log"); }; destination d_local { file("/shares/logfiles/syslog/syslog-ng.log"); }; ######################################################## # Filtering # Sun Remote Filtering filter f_sun_mesg { level(err) or level(crit) or (facility(kern)) or (facility(daemon)) or (facility(mail) and level(crit)); }; filter f_sun_syslog { facility(mail); }; filter f_sun_auth { facility(auth); }; filter f_sun_audit { facility(13); }; filter f_sun_cons { level(crit) or level(alert) or (facility (auth)) or level(emerg); }; # Hp Remote Filtering filter f_hp_mail { facility(mail); }; filter f_hp_syslog { level(info); }; filter f_hp_cons { level(alert) or level(emerg); }; # SAN Remote Filtering #filter f_san { level(info); }; # ESX Remote Filtering filter f_esx_cons { level(emerg); }; filter f_esx_mesg { level(info) or facility(auth) or facility(local6) or facility(local5) or facility(local7) or (facility(mail) and level(crit)); }; filter f_esx_auth { facility(auth); }; filter f_esx_mail { facility(mail); }; filter f_esx_cron { facility(cron); }; # Linux Remote Filtering filter f_linux_cons { level(emerg); }; filter f_linux_mesg { level(info) or facility(auth) or facility(local6) or facility(local5) or facility(local7) or (facility(mail) and level(crit)); }; filter f_linux_auth { facility(auth); }; filter f_linux_mail { facility(mail); }; filter f_linux_cron { facility(cron); }; ######################################################## # Logging # Sun Remote Logging log { source(s_sun); filter(f_sun_cons); destination(d_sun_cons); flags(flow-control); }; log { source(s_sun); filter(f_sun_mesg); destination(d_sun_mesg); flags(flow-control); }; log { source(s_sun); filter(f_sun_syslog); destination(d_sun_syslog); flags(flow-control); }; log { source(s_sun); filter(f_sun_auth); destination(d_sun_auth); flags(flow-control); }; log { source(s_sun); filter(f_sun_audit); destination(d_sun_audit); flags(flow-control); }; # Hp Remote Logging log { source(s_hp); filter(f_hp_mail); destination(d_hp_mail); flags(flow-control); }; log { source(s_hp); filter(f_hp_syslog); destination(d_hp_syslog); flags(flow-control); }; log { source(s_hp); filter(f_hp_cons); destination(d_hp_cons); flags(flow-control); }; # SAN log { source(s_switch); destination(d_switch); flags(flow-control); }; # SAN log { source(s_other); destination(d_other); flags(flow-control); }; # ESX Remote Logging log { source(s_esx); filter(f_esx_cons); destination(d_esx_cons); flags(flow-control); }; log { source(s_esx); filter(f_esx_mesg); destination(d_esx_mesg); flags(flow-control); }; log { source(s_esx); filter(f_esx_mail); destination(d_esx_mail); flags(flow-control); }; log { source(s_esx); filter(f_esx_auth); destination(d_esx_auth); flags(flow-control); }; log { source(s_esx); filter(f_esx_cron); destination(d_esx_cron); flags(flow-control); }; # Linux Remote Logging log { source(s_linux); filter(f_linux_cons); destination(d_linux_cons); flags(flow-control); }; log { source(s_linux); filter(f_linux_mesg); destination(d_linux_mesg); flags(flow-control); }; log { source(s_linux); filter(f_linux_mail); destination(d_linux_mail); flags(flow-control); }; log { source(s_linux); filter(f_linux_auth); destination(d_linux_auth); flags(flow-control); }; log { source(s_linux); filter(f_linux_cron); destination(d_linux_cron); flags(flow-control); }; log { source(s_local); destination(d_local); }; Thanks for any help you can offer. Brian Pontius
On Wed, 2010-01-20 at 17:05 -0500, Pontius, Brian D CIV NAVSISA wrote:
I apologize for what seems to be repost of a rather similar problem but I having looked through the archives and unable to find answers.
I am running syslog-ng 3.0.4 on Solaris 10 x86 (64bit). I have about 200 hosts, all running over udp. I have 1 heavy hitter, which is my firewall. I puts about 1500 messages a minute. It seems that syslog-ng is able to handle this amount of traffic but I am having trouble figuring out why I can't seem to make it work that way.
I started to notice that I was dropping udp packets by running netstat -s |grep udpInOverflows.
I tweaked the udp buffers by setting them to their max ndd -set /dev/udp udp_max_buf 1073741824 ndd -set /dev/udp udp_recv_hiwt 65536
I was still losing packets until I started to tweek my syslog-ng.conf and added the so_rcvbuf entries. The problem is, the logfiles do not reflect that all of the messages are making it. I only know this because the firewall is also logging to another standalone solaris server running standard syslogd and the syslog-ng's firewall's logs are still only getting 1/3 of the logs.
But what was the result of your tweaks? did the msg rate increase? I guess the options you've quoted above will only increase the maximum possible size, that the OS permits for applications. It doesn't immediately increase receive buffer size. -- Bazsi
Increasing the udp_max_buf and udp_recv_hiwt to their max alone did not stop the increase of the udpInOverflows. Only after I added the so_rcvbuf entries to some of my sources, did the increase of udpInOverflows stop increasing. Unfortunately, it still seems like I am losing messages because my firewall logs ARE STILL 1/3 of the second firewall syslog server. Syslog-ng still says that it is not dropping messages. Therefore I am stumped. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Saturday, January 23, 2010 8:02 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Messages dropped... On Wed, 2010-01-20 at 17:05 -0500, Pontius, Brian D CIV NAVSISA wrote:
I apologize for what seems to be repost of a rather similar problem but I having looked through the archives and unable to find answers.
I am running syslog-ng 3.0.4 on Solaris 10 x86 (64bit). I have about 200 hosts, all running over udp. I have 1 heavy hitter, which is my firewall. I puts about 1500 messages a minute. It seems that syslog-ng is able to handle this amount of traffic but I am having trouble figuring out why I can't seem to make it work that way.
I started to notice that I was dropping udp packets by running netstat -s |grep udpInOverflows.
I tweaked the udp buffers by setting them to their max ndd -set /dev/udp udp_max_buf 1073741824 ndd -set /dev/udp udp_recv_hiwt 65536
I was still losing packets until I started to tweek my syslog-ng.conf and added the so_rcvbuf entries. The problem is, the logfiles do not reflect that all of the messages are making it. I only know this because the firewall is also logging to another standalone solaris server running standard syslogd and the syslog-ng's firewall's logs are still only getting 1/3 of the logs.
But what was the result of your tweaks? did the msg rate increase? I guess the options you've quoted above will only increase the maximum possible size, that the OS permits for applications. It doesn't immediately increase receive buffer size. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Balazs Scheidler
-
Pontius, Brian D CIV NAVSISA