- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Elasticsearch destination
by Jim Hendrick 31 Oct '14

31 Oct '14

Or reman or "R" .. Hey .. this might actually be a thing! Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/31/2014 7:43 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Elasticsearch destination Hi, On Fri, Oct 31, 2014 at 07:05:47AM -0400, Jim Hendrick wrote: > area. That is one of the reasons I am trying to have syslog-ng do as > much as possible so I can remove "L" and only use "E" and "K" :-) yeah, let's call it the SNEK stack, or SNERK when using Riemann too :o) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 28 Oct '14

28 Oct '14

Thanks!I am on vacation this week but I look forward to trying out these suggestions. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/06/2014 11:42 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Mon, Oct 6, 2014 at 9:36 AM, Radu Gheorghe <radu.gheorghe(a)sematext.com> wrote: Hi Jim, With rabbitmq you have the advantage that you can install the RabbitMQ river and have Elasticsearch pull logs from Rabbit instead of having another [moving] piece pull logs from Rabbit and push them to ES. So you'd have a simpler setup that also makes sure ES isn't overwhelmed (because ES is pulling). Another point that can be an advantage is that with Redis you are limited to the RAM memory available on the machine (all your data need to fit in memory) , with RabbitMQ you have disk persistence that can help in cases where you need to stop ES consumption by any reason. There are some problems with this approach: - the river only runs on one node at a time, which may become a bottleneck - rivers are deprecated (or will be) so the ES side isn't actively maintained. I've seen failover issues (node running the river goes down, another node should start the river but doesn't) which needed river delete + recreate to kick the process in again I didn't know that, I will check it out. Logstash started by recommending RabbitMQ as the queue between two Logstash instances, but now moved to Redis. Apparently the reason is that Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from the guide: "Previous versions of this guide used AMQP via RabbitMQ. Due to the complexity of AMQP as well as performance issues related to the Bunny driver we use, we're now recommending Redis instead." Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 4

improving SQLite writing performances (WAS: Re: inserts into a sqlite3 database are not delayed)
by Davide Alberani 27 Oct '14

27 Oct '14

On Wed, Oct 15, 2014 at 10:56 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: > > It immediately runs the query, flush_lines() controls the transaction size, > which can be enabled using > flags(explicit-commits). Ok, I see and I can confirm noticeable improvements on a system under heavy I/O and CPU load using explicit-commits (or better: I see it gets a lot worse disabling it, since I already had it on ;)) Plus, I'm seeing minor improvements with session_statements set to "PRAGMA synchronous=OFF; PRAGMA count_changes=OFF;" Beside this, has anyone found other ways to improve writing performances using syslog-ng with sqlite? On (physical) systems with high load, I've noticed that the system is much more responsive if the elevator is set to noop or deadline instead of cfq. But I guess this is highly dependent on the specific circumstances of the system. Thanks! -- Davide Alberani <davide.alberani(a)gmail.com> [PGP KeyID: 0x465BFD47] http://www.mimante.net/

2 1

errors running configure for incubator on Ubuntu 12.04
by Russell Fulton 25 Oct '14

25 Oct '14

rful011@secmontst01:~/src/syslog-ng-incubator$ ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane… yes [ snip ] checking whether g++ accepts -g... yes checking dependency style of g++... gcc3 checking for inline... inline ./configure: line 4534: syntax error near unexpected token `shared' ./configure: line 4534: `LT_INIT(shared dlopen)’ I have looked at configure and config.log to try and figure out what it is actually checking for. ‘inline’ is to generic to google for :( any ideas what is wrong? Russell

2 1

syslog-ng 3.6.1 is released
by Czanik, Péter 22 Oct '14

22 Oct '14

Hi, We are proud to announce that culminating almost a year’s worth of development, syslog-ng 3.6.1 has been released! This is the first stable release from the 3.6 branch, the successor to 3.5, which was originally released in November, 2013. Read more about the highlights of this release at https://czanik.blogs.balabit.com/2014/10/syslog-ng-3-6-1-is-released/ or all the technical details at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.6.1 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng as "shipper" into ELK stack
by Jim Hendrick 20 Oct '14

20 Oct '14

Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana It works very well using the redis destination in syslog-ng, although I am having performance problems with logstash & elasticsearch default configurations keeping up. (I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on) Not so much a specific question (I'll be working on heap & thread settings and am pretty confident I can get it to handle at least this moderate load) but I was wondering if anyone else is working in this area. Also, in this configuration logstash is simply "parsing" the data it pulls from redis and sending it into elasticsearch. Seems like something syslog-ng might be able to do directly. Is anyone aware of any plans to implement an elasticsearch destination? Feel free to contact me on or off list if you want to discuss this. Thanks!! Jim

3 3

inserts into a sqlite3 database are not delayed
by Davide Alberani 15 Oct '14

15 Oct '14

Hi, I'm using a SQLite 3 database as a destination, but it seems that the options to delay the insert of new rows are ignored. The destination is something like: destination sqlite_db { sql(type(sqlite3) flags('explicit-commits') flush_timeout(60000) log_fifo_size(1000) flush_lines(1000) retry_sql_inserts(2) database("/tmp/logs.db") [...table, columns, values, indexes definitions...] }; Every time a log arrives, it's immediately written into the database, while I expected to have it delayed accordingly to flush_timeout and flush_lines. Currently I'm using syslog-ng OSE 3.4.2, but looking at the code of the latest version doesn't seem to have changed much. SQLite version is 3.8.4. libdbi 0.8.4 libdbi-drivers 0.8.3 libdbi-dbd-sqlite 0.8.3 libol 0.3.16 libevtlog 0.2.13 Any idea about how to fix it? Thanks in advance, -- Davide Alberani <davide.alberani(a)gmail.com> [PGP KeyID: 0x465BFD47] http://www.mimante.net/

2 1

Reg:Can two versions of Syslog-ng used on the same server
by Justin Kala 07 Oct '14

07 Oct '14

Can two versions of syslog-ng r reside on the same physical server. If so ,how do I handle the incoming messages.How do I configure syslog-ng.conf Please suggest -- Kaladhar

2 2

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 06 Oct '14

06 Oct '14

Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue.

3 2

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 03 Oct '14

03 Oct '14

Thanks! I will. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/03/2014 4:12 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack Hi Jim, On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote: > syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana We've been using the following stack for over a year: syslog-ng ==> logstash ==> elasticsearch For various reasons, one being performance, we recently switched to: syslog-ng ==> elasticsearch This was done thanks to the syslog-ng-incubator perl module. I've set up a small github repository where you can see our configuration [1]. > (I topped out today sending ~7000 events per second, and saw an insane > amount of swapping going on) I've had tremendous issues with LS when the workload was darting up. Since we switched to perl, we still have issues, but they're certainly not performance related: with a single perl destination we could easily keep up 10k events per second on a mediumish virtual machine. > Is anyone aware of any plans to implement an elasticsearch destination? The upcoming 3.6 version will ship with a "native" elasticsearch destination, which currently however is only a wrapper script. I'd highly appreciate if you could test a similar config to ours, in order to share some experience. Cheers [1] https://github.com/faxm0dem/syslog_ng-elasticsearch ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0