Coming from Graylog to Syslog-NG
Dear, I've implemented Graylog to receive around 5000-6000 logs/sec, but I have several problems: INPUT greater than OUTPUT so the journal increase a lot and the system crash, mainly. Now I want to use Syslog-NG, because I used it in the past but for 100-200 logs/sec. Here is my questions: 1) If I use Syslog-NG with a flat text file to receive 5000-6000 logs/sec, using s fast disk from my storage, do you recommneded to me this option ??? 2) As an extra benefit, what is the best and simple way to have a graphical view of all the logs ??? Thanks a lot, and maybe I'll ask you again about this topic in accordance with your comments. Roberto
Hi,
1) If I use Syslog-NG with a flat text file to receive 5000-6000 logs/sec, using s fast disk from my storage, do you recommneded to me this option ???
The file destination of syslog-ng is capable of handling 5000-6000 logs/sec. But if you have problems, you can open an issue on Github. 2) As an extra benefit, what is the best and simple way to have a
graphical view of all the logs ???
AFAIK there are no tools for visualizing logs in case of file destination. However, if logs could be forwarded using ElasticSearch destination, Kibana could be used to visualize logs. Also, if you configure a mongodb destination, you can use mojology (https://github.com/algernon/mojology).
On Wed, Jul 27, 2016 at 2:31 PM, Noémi Ványi <sitbackandwait@gmail.com> wrote:
Hi,
1) If I use Syslog-NG with a flat text file to receive 5000-6000 logs/sec, using s fast disk from my storage, do you recommneded to me this option ???
The file destination of syslog-ng is capable of handling 5000-6000 logs/sec. But if you have problems, you can open an issue on Github.
it should be possible to do 100k msg/sec or even more if you have templated destination files (e.g. /var/log/${PROGRAM}.log) we have seen syslog-ng chewing 500-600k msg/sec when writing to a distributed set of files.
2) As an extra benefit, what is the best and simple way to have a
graphical view of all the logs ???
AFAIK there are no tools for visualizing logs in case of file destination. However, if logs could be forwarded using ElasticSearch destination, Kibana could be used to visualize logs. Also, if you configure a mongodb destination, you can use mojology (https://github.com/algernon/mojology).
Thanks Balazs, so I can grow up the received Logs/Sec without any problem, using text files!!! Regards, 2016-07-27 9:52 GMT-03:00 Scheidler, Balázs <balazs.scheidler@balabit.com>:
On Wed, Jul 27, 2016 at 2:31 PM, Noémi Ványi <sitbackandwait@gmail.com> wrote:
Hi,
1) If I use Syslog-NG with a flat text file to receive 5000-6000 logs/sec, using s fast disk from my storage, do you recommneded to me this option ???
The file destination of syslog-ng is capable of handling 5000-6000 logs/sec. But if you have problems, you can open an issue on Github.
it should be possible to do 100k msg/sec or even more if you have templated destination files (e.g. /var/log/${PROGRAM}.log) we have seen syslog-ng chewing 500-600k msg/sec when writing to a distributed set of files.
2) As an extra benefit, what is the best and simple way to have a graphical view of all the logs ???
AFAIK there are no tools for visualizing logs in case of file destination. However, if logs could be forwarded using ElasticSearch destination, Kibana could be used to visualize logs. Also, if you configure a mongodb destination, you can use mojology (https://github.com/algernon/mojology).
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Wed, Jul 27, 2016 at 10:06:07AM -0300, Roberto Carna wrote:
Thanks Balazs, so I can grow up the received Logs/Sec without any problem, using text files!!!
I would't worry too much on syslog-ng's performance, but more on what your hardware can achieve :-)
Dear Noemi, thanks for your support. Please just a last question: Which is aproximately the Logs/Sec limit in the File destination ??? Which destination do I have to use in order to reach the maximum number of Logs/Sec ??? I'm thinking in the future, because now I receive 5000-6000 logs/sec, but maybe later this number will increase. Special thanks. 2016-07-27 9:31 GMT-03:00 Noémi Ványi <sitbackandwait@gmail.com>:
Hi,
1) If I use Syslog-NG with a flat text file to receive 5000-6000 logs/sec, using s fast disk from my storage, do you recommneded to me this option ???
The file destination of syslog-ng is capable of handling 5000-6000 logs/sec. But if you have problems, you can open an issue on Github.
2) As an extra benefit, what is the best and simple way to have a graphical view of all the logs ???
AFAIK there are no tools for visualizing logs in case of file destination. However, if logs could be forwarded using ElasticSearch destination, Kibana could be used to visualize logs. Also, if you configure a mongodb destination, you can use mojology (https://github.com/algernon/mojology).
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Fabien Wernli
-
Noémi Ványi
-
Roberto Carna
-
Scheidler, Balázs