October 2014 - syslog-ng - lists.balabit.hu

syslog-ng as "shipper" into ELK stack
by Jim Hendrick 20 Oct '14

20 Oct '14

Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana It works very well using the redis destination in syslog-ng, although I am having performance problems with logstash & elasticsearch default configurations keeping up. (I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on) Not so much a specific question (I'll be working on heap & thread settings and am pretty confident I can get it to handle at least this moderate load) but I was wondering if anyone else is working in this area. Also, in this configuration logstash is simply "parsing" the data it pulls from redis and sending it into elasticsearch. Seems like something syslog-ng might be able to do directly. Is anyone aware of any plans to implement an elasticsearch destination? Feel free to contact me on or off list if you want to discuss this. Thanks!! Jim

3 3

inserts into a sqlite3 database are not delayed
by Davide Alberani 15 Oct '14

15 Oct '14

Hi, I'm using a SQLite 3 database as a destination, but it seems that the options to delay the insert of new rows are ignored. The destination is something like: destination sqlite_db { sql(type(sqlite3) flags('explicit-commits') flush_timeout(60000) log_fifo_size(1000) flush_lines(1000) retry_sql_inserts(2) database("/tmp/logs.db") [...table, columns, values, indexes definitions...] }; Every time a log arrives, it's immediately written into the database, while I expected to have it delayed accordingly to flush_timeout and flush_lines. Currently I'm using syslog-ng OSE 3.4.2, but looking at the code of the latest version doesn't seem to have changed much. SQLite version is 3.8.4. libdbi 0.8.4 libdbi-drivers 0.8.3 libdbi-dbd-sqlite 0.8.3 libol 0.3.16 libevtlog 0.2.13 Any idea about how to fix it? Thanks in advance, -- Davide Alberani <davide.alberani(a)gmail.com> [PGP KeyID: 0x465BFD47] http://www.mimante.net/

2 1

Reg:Can two versions of Syslog-ng used on the same server
by Justin Kala 07 Oct '14

07 Oct '14

Can two versions of syslog-ng r reside on the same physical server. If so ,how do I handle the incoming messages.How do I configure syslog-ng.conf Please suggest -- Kaladhar

2 2

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 06 Oct '14

06 Oct '14

Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue.

3 2

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 03 Oct '14

03 Oct '14

Thanks! I will. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/03/2014 4:12 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack Hi Jim, On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote: > syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana We've been using the following stack for over a year: syslog-ng ==> logstash ==> elasticsearch For various reasons, one being performance, we recently switched to: syslog-ng ==> elasticsearch This was done thanks to the syslog-ng-incubator perl module. I've set up a small github repository where you can see our configuration [1]. > (I topped out today sending ~7000 events per second, and saw an insane > amount of swapping going on) I've had tremendous issues with LS when the workload was darting up. Since we switched to perl, we still have issues, but they're certainly not performance related: with a single perl destination we could easily keep up 10k events per second on a mediumish virtual machine. > Is anyone aware of any plans to implement an elasticsearch destination? The upcoming 3.6 version will ship with a "native" elasticsearch destination, which currently however is only a wrapper script. I'd highly appreciate if you could test a similar config to ours, in order to share some experience. Cheers [1] https://github.com/faxm0dem/syslog_ng-elasticsearch ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Question on parsing
by Richards, James L - DOA 03 Oct '14

03 Oct '14

So I have a scenario I am having difficulties with. I have an IDS sensor (suricata), and it is generating a log-file at /log_file_dir/fast.log And I would like to parse this log and send it off to a remote syslog server. I have put the following in my syslog-ng.conf: source s_log_server { file("/log_file_dir/fast.log " program_override("snort")); }; added a destination for the remote server: destination d_log_server { udp ("fqdn.of.primary.elsa.box" port(514)); }; Then in the log{ section I have put this: Destination (d_log_server); Logs are making it to the remote box, but in an unparsed format... How do I get this to trigger a parser in syslog-ng? Thanks much, Jim

1 0

File Source limits in OSE and PE?
by Doug McClure 03 Oct '14

03 Oct '14

Are there limits to the number of unique files that can be monitored with an installation of OSE or PE? For example, could I monitor 500 unique log files on a given server? Tks! Doug

2 2

exclude pattern from logging
by fRANz 01 Oct '14

01 Oct '14

Hello guys, I'm trying to config syslog-ng to ignore and don't log some strings. I'm using syslog-ng-3.2.5-4.el6.x86_64 on a CentOS 6.5: the config is pretty simple: # cat /etc/syslog-ng/syslog-ng.conf @version:3.2 options { check_hostname(yes); keep_hostname(yes); stats_freq(0); chain_hostnames(no); }; source inputs { internal(); unix-stream("/dev/log"); udp(); tcp(max_connections(100)); }; destination logpile { file("/logs/$HOST/$YEAR/$MONTH/$DAY/$FACILITY" owner(root) group(root) perm(0600) create_dirs(yes) dir_perm(0700)); }; #filter vmware_filter { match("Section for VMware ESX" value ("MESSAGE")); }; #filter vmware_filter { message("Section for VMware ESX"); }; filter vmware_filter { not message("Section for VMware ESX"); }; #log { source(inputs); filter(vmware_filter); flags(final); }; #log { source(inputs); destination(logpile); }; log { source(inputs); filter(vmware_filter); destination(logpile); }; but I cannot find the right key. Wha'ts wrong in this config? You can see old configuration applied. Thanks for any tip. -f

2 4

Syslog-NG.conf to Fork to Two Log Aggregators
by wiskbroom＠hotmail.com 01 Oct '14

01 Oct '14

Hello; I have syslog clients that I would like to configure to send log-data to a middle-man/intermediary syslog-NG server. Once received on the intermediary, I want to immediately fork that data onto a different log-server, not syslog-NG; satisfying a requirement to feed two systems. The reason for the fork is because the non-syslog-NG-server is running a proprietary logging system, and it must, at least for now, be capable of seeing *most* of my logs. It, the non-syslog-NG-server, is incapable of retransmitting to my syslog-NG server, nor would I trust it to do so. My questions to the list are, 1. Has anyone successfully done something similar? 2. Any recommendations/gotchas I should be aware of? 3. Can I also configure syslog-NG to also resend Splunk data? Or do I have to run a Splunk Univ Forwarder configured similarly to my intermediary syslog-NG server to achieve that? (Yes, I know, OT question, sorry...) Thank you in advance, .vp

2 1