Queue size in syslog-ng?
Hi all, In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages). What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down? -- Cheers, Abhijeet R http://blog.abhijeetr.com
We have a lot of endpoints that log, sometimes, lots of data. Does anyone know of a good, professionally hosted syslog solution, so we can direct the traffic there and make the headaches of collecting it turn into a monthly bill? Thx
You need to configure log_fifo_size. I've set "log_fifo_size(1000000)" to get sure nothing get lost. Make sure you have some memory left free on the server :) Daniel -----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Abhijeet R Gesendet: Montag, 19. November 2012 11:43 An: syslog-ng@lists.balabit.hu Betreff: [syslog-ng] Queue size in syslog-ng? Hi all, In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages). What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down? -- Cheers, Abhijeet R http://blog.abhijeetr.com ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Random tuning. Just make it bigger! On Thu, Nov 22, 2012 at 1:55 PM, Daniel Neubacher <daniel.neubacher@xing.com
wrote:
You need to configure log_fifo_size. I've set "log_fifo_size(1000000)" to get sure nothing get lost. Make sure you have some memory left free on the server :)
Daniel
-----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Abhijeet R Gesendet: Montag, 19. November 2012 11:43 An: syslog-ng@lists.balabit.hu Betreff: [syslog-ng] Queue size in syslog-ng?
Hi all,
In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages).
What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down?
-- Cheers, Abhijeet R http://blog.abhijeetr.com
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Best regards, Koldaev Anton
How many messages per second can syslog-ng handle? Does anyone have any number. Lets suppose, I have a 16GB RAM, 8 core 2.5Ghz box. On Thu 22 Nov 2012 08:01:27 PM IST, Anton Koldaev wrote:
Random tuning. Just make it bigger!
On Thu, Nov 22, 2012 at 1:55 PM, Daniel Neubacher <daniel.neubacher@xing.com <mailto:daniel.neubacher@xing.com>> wrote:
You need to configure log_fifo_size. I've set "log_fifo_size(1000000)" to get sure nothing get lost. Make sure you have some memory left free on the server :)
Daniel
-----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu>] Im Auftrag von Abhijeet R Gesendet: Montag, 19. November 2012 11:43 An: syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu> Betreff: [syslog-ng] Queue size in syslog-ng?
Hi all,
In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages).
What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down?
-- Cheers, Abhijeet R http://blog.abhijeetr.com
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Best regards, Koldaev Anton
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Cheers, Abhijeet R http://blog.abhijeetr.com
shadyabhi <abhijeet.1989@gmail.com> writes:
How many messages per second can syslog-ng handle? Does anyone have any number. Lets suppose, I have a 16GB RAM, 8 core 2.5Ghz box.
That depends on a lot of things. If you have a single source, a single destination and no filtering or rewriting, that'll be reasonably fast, but will use only one core. However, since this simple forwarding is not CPU intensive, your bottleneck won't be the CPU. (most likely the end point will be, especially if its a disk) If you do filtering or rewriting, that can slow things down considerably. Using multiple sources or destinations also affects performance in various ways. Therefore, unfortunately, there is no definitive answer to your question, as it depends on a whole lot of variables. -- |8]
Nice useless comment... With 30k logs per second it is in fact a small queue but enough for a restart of the logstash services. Von: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Anton Koldaev Gesendet: Donnerstag, 22. November 2012 15:31 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] Queue size in syslog-ng? Random tuning. Just make it bigger! On Thu, Nov 22, 2012 at 1:55 PM, Daniel Neubacher <daniel.neubacher@xing.com<mailto:daniel.neubacher@xing.com>> wrote: You need to configure log_fifo_size. I've set "log_fifo_size(1000000)" to get sure nothing get lost. Make sure you have some memory left free on the server :) Daniel -----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] Im Auftrag von Abhijeet R Gesendet: Montag, 19. November 2012 11:43 An: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Betreff: [syslog-ng] Queue size in syslog-ng? Hi all, In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages). What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down? -- Cheers, Abhijeet R http://blog.abhijeetr.com ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -- Best regards, Koldaev Anton
Hi, as others have suggested, you can increase the log_fifo_size option, which will increase the memory queue of syslog-ng. Depending on the message rate, this can prevent losing messages if the consumer is down, but if anything happens to your syslog-ng server in the meantime, these messages will be lost. The commercial version of syslog-ng, syslog-ng Premium Edition can buffer the messages to the hard disk in such case to decrese the risk of losing messages. Robert On 11/19/2012 11:42 AM, Abhijeet R wrote:
Hi all,
In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages).
What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down?
participants (7)
-
Abhijeet R
-
Anton Koldaev
-
Daniel Neubacher
-
Fekete Robert
-
Gergely Nagy
-
Rory Toma
-
shadyabhi