How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases? -- Florin Andrei mailto:florin@linuxstart.com http://members.linuxstart.com/~florin/ tel: +40-93-261162
On Mon, 8 May 2000, Florin Andrei wrote:
How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases?
For me, it eats up the cpu and a lot of ram (~40M) after a week or so. The old syslogd didn't do that. (~20 routers and 5-6 servers logging to it with $PROGRAM & $HOST expansion, no dns, k7-500, u2w disk) greetings endre
How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases?
For me, it eats up the cpu and a lot of ram (~40M) after a week or so. The old syslogd didn't do that.
I find that syslog-ng tends to eat MUCH MUCH less system performance than syslogd, as long as I have cron restart it once a day. This is with enough inbound logging to generate somewhere around 200 megs a day. -- A.L.Lambert -------------------------------------------------------------------------- If what they've been doing hasn't solved the problem, tell them to do something else. -- Gerald Weinberg, "The Secrets of Consulting" --------------------------------------------------------------------------
On Mon, 8 May 2000, Florin Andrei wrote:
How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases?
I find that syslog-ng drops approximately 95% of all log entries when the system generates them quickly. This is using the tcp protocol and writing to a file on the remote machine. This has kept me from trying to implement syslog-ng in production. -jwb
How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases?
I find that syslog-ng drops approximately 95% of all log entries when the system generates them quickly. This is using the tcp protocol and writing to a file on the remote machine.
This has kept me from trying to implement syslog-ng in production.
You can finetune garbage collection parameters using the gc_idle_threshold and gc_busy_threshold options. idle threshold should be set to a low value, busy_threshold should be set to a high value (about the number of messages in a burst) options { gc_idle_threshold(100); gc_busy_threshold(5000); }; -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
On Mon, 8 May 2000, Balazs Scheidler wrote:
How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases?
I find that syslog-ng drops approximately 95% of all log entries when the system generates them quickly. This is using the tcp protocol and writing to a file on the remote machine.
This has kept me from trying to implement syslog-ng in production.
You can finetune garbage collection parameters using the gc_idle_threshold and gc_busy_threshold options.
idle threshold should be set to a low value, busy_threshold should be set to a high value (about the number of messages in a burst)
options { gc_idle_threshold(100); gc_busy_threshold(5000); };
I tried your suggestion before. I does help with transient message bursts, but at steady state syslog-ng eventually gets overwhelmed. My application does not generate short, separated bursts of messages. It creates long, sustained streams of messages. This is where I see syslog-ng dropping (most) log entries. You can try for yourself by dumping short messages to an input named pipe in a tight loop. -jwb
How syslog-ng deals with high-volume logging? The old syslogd is very bad in this respect; if you feed it with lots of logs, the system slows down horribly. So, how does syslog-ng behaves in such cases?
I find that syslog-ng drops approximately 95% of all log entries when the system generates them quickly. This is using the tcp protocol and writing to a file on the remote machine.
This has kept me from trying to implement syslog-ng in production.
You can finetune garbage collection parameters using the gc_idle_threshold and gc_busy_threshold options.
idle threshold should be set to a low value, busy_threshold should be set to a high value (about the number of messages in a burst)
options { gc_idle_threshold(100); gc_busy_threshold(5000); };
I tried your suggestion before. I does help with transient message bursts, but at steady state syslog-ng eventually gets overwhelmed. My application does not generate short, separated bursts of messages. It creates long, sustained streams of messages. This is where I see syslog-ng dropping (most) log entries. You can try for yourself by dumping short messages to an input named pipe in a tight loop.
Then you may need to increase the output fifo size using the log_fifo_size() option. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
participants (5)
-
A.L.
-
Balazs Scheidler
-
Florin Andrei
-
Hirling Endre
-
Jeffrey W. Baker