Solaris 10 non-global zone & syslog-ng performance question.
Folks, I'm a newbie to syslog-ng. Everything is working in my environment, running 2.0.8. But, when I try to stress the server I drop messages. I'm running from another physical server: ./loggen -D -s 111 -r 2000 -I 1 <hostname> 514 ; date average rate = 1818.00 msg/sec, count=1818 Thu Mar 6 18:07:17 MST 2008 The syslog-ng server is in a Solaris 10 non-global zone; my auth.info is going to /var/adm/ssh_DAY.log. date ; tail /var/adm/ssh_06.log | awk '/runid/ {print $9}' | sort -u | xargs -i egrep {} /var/adm/ssh_06.log | wc -l Thu Mar 6 18:07:23 MST 2008 1233 So, I'm dropping packets. I've stopped syslog-ng and started standard syslog. Syslog logs all my messages to the file when using a rate of 2000/sec. As you can from above, in my syslog-ng environment I lose about a 3rd. When I use a rate of 1000/sec, it is not quite as bad, but I do drop a significant amount of messages. I've read: http://www.l3jane.net/doc/server/syslog-ng/#tuning I've toyed with log_fifo_size (using values like 1000, 5000, 10000, 50000) log_fetch_limit (using values like 100, 200, 2000) sync (using values like 20, 40, 50, 100) I have "use_dns (no)" - turning that off first. Also, I commented out all but two sources, the one filter, the one log line. source s_ip148 { udp (ip(<x>.<y>.<z>.148)); }; source s_ip248 { udp (ip(<x>.<y>.<z>.248)); }; <x>, <y>, <z> are obviously replace in our environment with the octets for the subnet. Before trying standard syslog, I was running snoop in my global zone to ensure all the UDP packets were arriving. The count always matched the count shown by loggen. Since I can see all the messages with standard syslog in this Sol 10 non-global zone, I've ruled out: * the zone being an issue. * UDP tuning for the Solaris 10 host. * The internal hard drive i/o speed from being an issue. Looking at the log messages, I'm not hitting some cap (e.g. 1233) and then not logging any more. Random gaps (both of frequency and size) are seen throughout. Can someone give me some pointers? Thanks in advance, Jim
On Thu, 2008-03-06 at 18:35 -0700, Surlow, Jim wrote:
Folks,
I’m a newbie to syslog-ng. Everything is working in my environment, running 2.0.8. But, when I try to stress the server I drop messages.
I’m running from another physical server:
./loggen -D -s 111 -r 2000 -I 1 <hostname> 514 ; date
average rate = 1818.00 msg/sec, count=1818
Thu Mar 6 18:07:17 MST 2008
Since I can see all the messages with standard syslog in this Sol 10 non-global zone, I’ve ruled out:
· the zone being an issue.
· UDP tuning for the Solaris 10 host.
· The internal hard drive i/o speed from being an issue.
Looking at the log messages, I’m not hitting some cap (e.g. 1233) and then not logging any more. Random gaps (both of frequency and size) are seen throughout.
Still, you should increase your UDP receive buffer. Syslog-ng has more latency wrt reading the socket buffer, so that's why you can see more drops. This is not a throughput issue in itself if you have large enough buffers (e.g. the message rate might be higher than syslogd, but the latency between message receival might be long enough to overflow your UDP receive buffers). So so_rcvbuf() is the option you need. Be sure that you also check what the default OS limit is. -- Bazsi
Thanks. I got this working. A note to make: http://www.syslog.org/syslog-ng/v2/ Tables 3.2 & 3.9 have the description & defaults switched half way through the table. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Monday, March 10, 2008 5:44 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Solaris 10 non-global zone & syslog-ngperformance question. On Thu, 2008-03-06 at 18:35 -0700, Surlow, Jim wrote:
Folks,
I’m a newbie to syslog-ng. Everything is working in my environment, running 2.0.8. But, when I try to stress the server I drop messages.
I’m running from another physical server:
./loggen -D -s 111 -r 2000 -I 1 <hostname> 514 ; date
average rate = 1818.00 msg/sec, count=1818
Thu Mar 6 18:07:17 MST 2008
Since I can see all the messages with standard syslog in this Sol 10 non-global zone, I’ve ruled out:
· the zone being an issue.
· UDP tuning for the Solaris 10 host.
· The internal hard drive i/o speed from being an issue.
Looking at the log messages, I’m not hitting some cap (e.g. 1233) and then not logging any more. Random gaps (both of frequency and size) are seen throughout.
Still, you should increase your UDP receive buffer. Syslog-ng has more latency wrt reading the socket buffer, so that's why you can see more drops. This is not a throughput issue in itself if you have large enough buffers (e.g. the message rate might be higher than syslogd, but the latency between message receival might be long enough to overflow your UDP receive buffers). So so_rcvbuf() is the option you need. Be sure that you also check what the default OS limit is. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Balazs Scheidler
-
Surlow, Jim