June 2001 - syslog-ng - lists.balabit.hu

RE: [syslog-ng]newbie Solaris question
by T. Esting 25 Jun '01

25 Jun '01

Drew, et al - Thanks for writing back and for the sync() parameter suggestion. I tried setting sync(0) and that does indeed work for low-volume situations. However, the penalty for doing it this way seems to be exacted during high-volume situations, e.g. several hundred messages per second. I have a rather simple test program that I run while looking at my output file: #include <syslog.h> #include <unistd.h> main( int argc, char **argv) { if (argc == 2) { int i, loop = 0; loop = atoi(argv[1]); if (loop != 0) { openlog( "EER-TEST: ", LOG_PID, LOG_LOCAL5); for (i = 1; i <= loop; i++) { syslog( LOG_INFO, "testing, 1, 2, 3 ... %d", i); } } } } This attempts to simulate an nmap scan or DoS-type attack against my firewall; I am hoping to lose as few log messages as possible in these scenarios. With the stock Solaris syslog (brain-dead as it may be), I am able to run this program with arguments of 1000 or 2000 and not lose any messages. After setting sync(0) so that I wouldn't lose any messages in low-volume situations, I ran the same tests and, although syslog-ng kept up with a burst of 100 or 200 messages, 1000 turned out to be way too much; over 300 messages were lost. I'm hoping to find the sweet spot for tuning syslog-ng, since it obviously has many superior features to the standard syslog daemon that ships with Solaris, but I'm running into some roadblocks. TIA. -------------------- What do you mean you can't get it to output every message it receives? I have used syslog-ng on Solaris for 2 years with very few problems and I log over 1000 devices. Do you know if it is receiving messages? You're not running syslogd as well are you? In low volumes the defaults for garbage collection should be sufficient. How many hosts are you logging? Try setting sync(0). Regards, Drew -----Original Message----- From: T. Esting [mailto:T_Esting@excite.com] Tuesday, June 12, 2001 6:16 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng]newbie Solaris question I've been trying to get both 1.4.11 and 1.5.7 to run on my Solaris system, and have met with limited success. Compiling and installing were fairly easy, as was creating a configuration file. I understand from the docs that the mark() feature still doesn't work, and from the mailing list that sync_freq() should actually be sync(). I am having trouble, however, with flushing output in low-volume scenarios. Here is the configuration file I am using: options { use_dns(no); log_fifo_size(XXX); gc_idle_threshold(YYY); gc_busy_threshold(ZZZ); use_time_recvd(yes); }; source sun { sun-streams ( "/dev/log" door("/etc/.syslog_door")); internal(); }; destination testout { file ( "/var/adm/testout.txt" sync(WWW)); } ; log { source (sun); destination (testout); }; I've tried various values for WWW, XXX, YYY, and ZZZ, even setting WWW to 1. I have not been able to influence syslog-ng to write out every message it receives. Does anyone have any pointers on what I might want to try next to get syslog-ng to more reliably receive and log messages on Solaris? Thanks. _______________________________________________________ Send a cool gift with your E-Card http://www.bluemountain.com/giftcenter/

2 1

RE: [syslog-ng]newbie Solaris question
by Hamilton, Andrew Mr RAYTHEON 5 SIG CMD 25 Jun '01

25 Jun '01

That helps some. What is your log_fifo_size() set to? I assume you are looking at doing this over the network, yes? In the original email you said you were looking at low-volume situations so the suggestions we made were based on that. For higher volumes you may need to do other things and that depends on your set-up. Of course I/O system calls are very expensive performance wise and a heavy amount of disk activity is going to slow you down. You will have to raise your sync some to buffer your messages and slow down the writes. The sync number is the number of lines buffered before write. If you are sending 1000 lines per second try raising your sync to 1000 and writing every second or so. You'll have to play with the numbers. Some of your performance issues are going to be hardware related, slow disks, etc. If you are logging to the machine that you are running the program on log_fifo_size isn't going to help. That is for slow networks, not slow disks. If you are writing your messages over the network and you want to slow down your network writes you raise your log_fifo_size on the client. There is some memory usage cost associated with that but you can save a considerable amount of slowdown on your log system. Regards, Drew -----Original Message----- From: T. Esting [mailto:T_Esting@excite.com] Sent: Saturday, June 23, 2001 5:38 PM To: HamiltonA(a)hq.5sigcmd.army.mil Cc: syslog-ng(a)lists.balabit.hu Subject: RE: [syslog-ng]newbie Solaris question Drew, et al - Thanks for writing back and for the sync() parameter suggestion. I tried setting sync(0) and that does indeed work for low-volume situations. However, the penalty for doing it this way seems to be exacted during high-volume situations, e.g. several hundred messages per second. I have a rather simple test program that I run while looking at my output file: #include <syslog.h> #include <unistd.h> main( int argc, char **argv) { if (argc == 2) { int i, loop = 0; loop = atoi(argv[1]); if (loop != 0) { openlog( "EER-TEST: ", LOG_PID, LOG_LOCAL5); for (i = 1; i <= loop; i++) { syslog( LOG_INFO, "testing, 1, 2, 3 ... %d", i); } } } } This attempts to simulate an nmap scan or DoS-type attack against my firewall; I am hoping to lose as few log messages as possible in these scenarios. With the stock Solaris syslog (brain-dead as it may be), I am able to run this program with arguments of 1000 or 2000 and not lose any messages. After setting sync(0) so that I wouldn't lose any messages in low-volume situations, I ran the same tests and, although syslog-ng kept up with a burst of 100 or 200 messages, 1000 turned out to be way too much; over 300 messages were lost. I'm hoping to find the sweet spot for tuning syslog-ng, since it obviously has many superior features to the standard syslog daemon that ships with Solaris, but I'm running into some roadblocks. TIA. -------------------- What do you mean you can't get it to output every message it receives? I have used syslog-ng on Solaris for 2 years with very few problems and I log over 1000 devices. Do you know if it is receiving messages? You're not running syslogd as well are you? In low volumes the defaults for garbage collection should be sufficient. How many hosts are you logging? Try setting sync(0). Regards, Drew -----Original Message----- From: T. Esting [mailto:T_Esting@excite.com] Tuesday, June 12, 2001 6:16 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng]newbie Solaris question I've been trying to get both 1.4.11 and 1.5.7 to run on my Solaris system, and have met with limited success. Compiling and installing were fairly easy, as was creating a configuration file. I understand from the docs that the mark() feature still doesn't work, and from the mailing list that sync_freq() should actually be sync(). I am having trouble, however, with flushing output in low-volume scenarios. Here is the configuration file I am using: options { use_dns(no); log_fifo_size(XXX); gc_idle_threshold(YYY); gc_busy_threshold(ZZZ); use_time_recvd(yes); }; source sun { sun-streams ( "/dev/log" door("/etc/.syslog_door")); internal(); }; destination testout { file ( "/var/adm/testout.txt" sync(WWW)); } ; log { source (sun); destination (testout); }; I've tried various values for WWW, XXX, YYY, and ZZZ, even setting WWW to 1. I have not been able to influence syslog-ng to write out every message it receives. Does anyone have any pointers on what I might want to try next to get syslog-ng to more reliably receive and log messages on Solaris? Thanks. _______________________________________________________ Send a cool gift with your E-Card http://www.bluemountain.com/giftcenter/

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: RE: [syslog-ng]newbie Solaris question that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0

Mail message not delivered
by MAILER-DAEMON＠colltech.com 25 Jun '01

25 Jun '01

Your message, titled: [syslog-ng]Mail message not delivered that you sent to rdees(a)colltech.com has not been delivered. The user at that address is no longer available. The following alias addresses are also no longer available. dees_randy(a)colltech.com randy_dees(a)colltech.com Please update your address book Thank You

1 0