syslog-ng buffer and reload
Hi, I'm using the disk queue to buffer the writes to elasticsearch. I notices the following behaviour: When reloading the configuration (`syslog-ng-ctl reload`), syslog-ng stops processing incoming messages (they appear as `dropped` in stats) and starts emptying the queue. It only starts accepting new messages when the queue is completely empty. I understand this is probably an expected behaviour, but in the following scenario (I just experienced) it poses a problem: 1. some application goes bananas logging at zillions of events per second 2. syslog-ng queue starts filling up 3. crazy app identified: I modify syslog-ng.conf in order to filter out the app 4. syslog-ng-ctl reload 5. syslog-ng starts dropping all new messages and emptying the queue 6. I have to wait for the queue to be empty (which can take a long time) Wouldn't it be saner to continue accepting messages when intercepting the HUP?
+1 (make sense) -- Jorge Pereira On Mon, Mar 27, 2017 at 7:52 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
I'm using the disk queue to buffer the writes to elasticsearch. I notices the following behaviour:
When reloading the configuration (`syslog-ng-ctl reload`), syslog-ng stops processing incoming messages (they appear as `dropped` in stats) and starts emptying the queue. It only starts accepting new messages when the queue is completely empty.
I understand this is probably an expected behaviour, but in the following scenario (I just experienced) it poses a problem:
1. some application goes bananas logging at zillions of events per second 2. syslog-ng queue starts filling up 3. crazy app identified: I modify syslog-ng.conf in order to filter out the app 4. syslog-ng-ctl reload 5. syslog-ng starts dropping all new messages and emptying the queue 6. I have to wait for the queue to be empty (which can take a long time)
Wouldn't it be saner to continue accepting messages when intercepting the HUP?
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
hi, your use case is a special one: you know that you won't need any of the messages stored in the diskbuffer after a reload. Maybe a --clear-diskqueue/--start-with-empty-diskqueue switch to the reload command could solve this issue. But with this option you will lose all your messages that are stored in the diskqueue. What do you think? regards, Laszlo Budai On Monday, March 27, 2017, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
I'm using the disk queue to buffer the writes to elasticsearch. I notices the following behaviour:
When reloading the configuration (`syslog-ng-ctl reload`), syslog-ng stops processing incoming messages (they appear as `dropped` in stats) and starts emptying the queue. It only starts accepting new messages when the queue is completely empty.
I understand this is probably an expected behaviour, but in the following scenario (I just experienced) it poses a problem:
1. some application goes bananas logging at zillions of events per second 2. syslog-ng queue starts filling up 3. crazy app identified: I modify syslog-ng.conf in order to filter out the app 4. syslog-ng-ctl reload 5. syslog-ng starts dropping all new messages and emptying the queue 6. I have to wait for the queue to be empty (which can take a long time)
Wouldn't it be saner to continue accepting messages when intercepting the HUP?
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Tue, Mar 28, 2017 at 09:05:08AM +0200, Budai, László wrote:
your use case is a special one: you know that you won't need any of the messages stored in the diskbuffer after a reload.
True enough
Maybe a --clear-diskqueue/--start-with-empty-diskqueue switch to the reload command could solve this issue. But with this option you will lose all your messages that are stored in the diskqueue. What do you think?
That would certainly be an improvement :-) What would be even better (maybe the "syslog-ng as a command line" could help) is to apply a filter to what shall be deleted: --purge-queue --drop-from-queue-matching 'not message("spurious message");' but I fear I'm asking for too much ;-D
Hi, On Tue, Mar 28, 2017 at 9:17 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Tue, Mar 28, 2017 at 09:05:08AM +0200, Budai, László wrote:
your use case is a special one: you know that you won't need any of the messages stored in the diskbuffer after a reload.
True enough
Maybe a --clear-diskqueue/--start-with-empty-diskqueue switch to the reload command could solve this issue. But with this option you will lose all your messages that are stored in the diskqueue. What do you think?
That would certainly be an improvement :-)
What would be even better (maybe the "syslog-ng as a command line" could help) is to apply a filter to what shall be deleted:
--purge-queue --drop-from-queue-matching 'not message("spurious message");'
but I fear I'm asking for too much ;-D
no, I like your idea :) If it's not a problem to stop syslog-ng then drop some messages from the diskqueue and then start syslog-ng. L.
On Tue, Mar 28, 2017 at 10:19:39AM +0200, Budai, László wrote:
no, I like your idea :) If it's not a problem to stop syslog-ng then drop some messages from the diskqueue and then start syslog-ng.
What about: # disables queue but does not remove file # e.g. by doing `mv syslog-ng-00000.qf syslog-ng-00000.qf.disabled` syslog-ng-ctl --mv-away-queue --reload # syslog-ng continues immediately with an empty queue # so sysadmin has time to curate the queue file # offline work on disabled queue file syslog-ng --cli-mode [… do some queue filtering on syslog-ng-00000.qf.disabled…] # now we're happy with old queue, tell running syslog-ng to process it syslog-ng-ctl --process-old-queue=syslog-ng-00000.qf.disabled Would that be doable?
+1 (--process-old-queue) as a separate process or thread when reloaded. On Tue, Mar 28, 2017 at 4:37 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Tue, Mar 28, 2017 at 10:19:39AM +0200, Budai, László wrote:
no, I like your idea :) If it's not a problem to stop syslog-ng then drop some messages from the diskqueue and then start syslog-ng.
What about:
# disables queue but does not remove file # e.g. by doing `mv syslog-ng-00000.qf syslog-ng-00000.qf.disabled` syslog-ng-ctl --mv-away-queue --reload
# syslog-ng continues immediately with an empty queue # so sysadmin has time to curate the queue file
# offline work on disabled queue file syslog-ng --cli-mode [… do some queue filtering on syslog-ng-00000.qf.disabled…]
# now we're happy with old queue, tell running syslog-ng to process it syslog-ng-ctl --process-old-queue=syslog-ng-00000.qf.disabled
Would that be doable?
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Budai, László
-
Fabien Wernli
-
Jorge Pereira
-
Scot