- syslog-ng - lists.balabit.hu

Syslog-ng Drops Logs
by Yalin Aksoy 07 Nov '14

07 Nov '14

We have lots of traffic going through syslog-ng in our system (5000 logs per second), so some logs are dropped because of 'log_fifo_size()'. I've look around the web and found flush-lines and flush-timeout methods, but also it failed in every configuration. Related parts of my syslog-ng conf looks like that at the moment. " log_fifo_size(4096); flush_lines(100); flush_timeout(1000); " If I increase fifo size to ~ 16000 ,syslog-ng consumes too much memory for my system to operate. Is there any other way to stop that leak and what is the best practice to use? Thanks in advance. -Yalin

3 3

insider 2014-11: 3.6.1 released; JSON; SSB 4LTS; TLS mutual auth; PE 5F2;
by Czanik, Péter 06 Nov '14

06 Nov '14

Dear syslog-ng users, This is the 38th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. FEATURED NEWS syslog-ng 3.6.1 released ------------------------ We are proud to announce that culminating almost a year’s worth of development, syslog-ng 3.6.1 has been released! This is the first stable release from the 3.6 branch, the successor to 3.5, which was originally released in November, 2013. Read more about the highlights of this release at https://czanik.blogs.balabit.com/2014/10/syslog-ng-3-6-1-is-released/ or all the technical details at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.6.1 Sending JSON formatted messages with syslog-ng ---------------------------------------------- The use of name-value pairs greatly enhances the usability of collected logs. There are many ways in syslog-ng to turn log messages into name-value pairs: parsers. From this post you can learn how to parse log messages using syslog-ng and send the resulting name-value pairs in JSON format to Loggly. Read our guest blog at https://www.loggly.com/blog/sending-json-format-logs-syslog-ng/ This article focuses on JSON and Loggly, but once your logs are parsed, name-value pairs can also be sent to Graphite, Riemann, Redis, AMQP, MongoDB, and so on. SSB 4LTS is released -------------------- One of the products building on the foundations of the open source syslog-ng is SSB (syslog-ng Store Box). It provides a complete log management solution with a web interface based on syslog-ng. The new long term support release comes with many new features and performance improvements. Read more at https://jluby.blogs.balabit.com/2014/10/30/syslog-ng-store-box-4lts-release… TLS encryption and mutual authentication ---------------------------------------- Configuring TLS encryption and mutual authentication can be a challenging job as it needs both strong OpenSSL and syslog-ng skills. This tutorial provides step by step instructions about the necessary steps: creating certificates using OpenSSL and configuring syslog-ng to use them. The tutorial was originally published on linux.com: http://www.linux.com/community/blogs/129-servers/790912-tls-encryption-and-… and now also available on balabit.com at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guid… syslog-ng PE 5F2 released ------------------------- The second feature release of syslog-ng PE is released. The primary focus is data privacy: anonymization and pseudonymization of incoming log messages. Other features include journal support, SMTP destination. Read more about it at https://jluby.blogs.balabit.com/2014/10/28/introducing-syslog-ng-premium-ed… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Launching command with a certain value after extracting it from patterndb
by C. L. Martinez 05 Nov '14

05 Nov '14

Hi all, Is it possible to trigger a command after extracting a field using patterndb? For example I have the following log: Nov 4 15:18:10 myserver01 info ftps[876]: Rule Allow <ALLOW>: - MAP user:mytest IP:1.1.1.1 With patterndb, I can extract field user with for example, a value of $user. Can I trigger a command like "cat $user >> /tmp/users.log" without calling a script?? Thanks.

3 3

How can I disable SSLv3 in syslog-ng 3.3.2 client config to sovle CVE-2014-3566(SSLv3 Fallback Vulnerabilit)?
by bluebenben 05 Nov '14

05 Nov '14

Hi guys In my project I am using syslog-ng as syslog client and send log via TLS. We all know that recently there is one new security flaw which is Poodle(CVE-2014-3566 - SSLv3 Fallback Vulnerability) This requires disabling SSLv3 I have checked admin guide of syslog-ng 3.3.2 but I am able to find the option Could you please let me know the way? Alternatively I think I may achieve the object by disable SSLv3 ciphers used by syslog-ng client original ciphers used by us is ALL:!SSLv2:!MEDIUM:!LOW:!EXP:!ADH:!ECDH:!PSK:!MD5:@STRENGTH I may change it to ALL:!SSLv3:!SSLv2:!MEDIUM:!LOW:!EXP:!ADH:!ECDH:!PSK:!MD5:@STRENGTH Bug this will make syslog-ng only supports TLS1.2 and cause negative impact to interoperability Thanks Jason

2 1

Elasticsearch destination
by Russell Fulton 31 Oct '14

31 Oct '14

Hi We are already using the open source version of syslog-ng and I am about to set up some elastic search instances and would much prefer to feed data direct from syslog-ng rather than go through logstash (I already have a heap of patterndb parsers and performance should be way better!) I have spent an hour or so with Google and have found various references to elastic search destination being available but I can find no mention of it in the release notes for 3.6.1. I have also downloaded the the tarball and unpacked it but could not find any evidence of the module , nore is there any mention of it in the manual. As of now what is the recommended way of getting parsed data from OS syslog-ng into ES? Thanks, Russell

6 17

Re: [syslog-ng] Elasticsearch destination
by Jim Hendrick 31 Oct '14

31 Oct '14

If we use syslog-ng, hadoop, redis, elasticsearch and kibana we could call it the SHREK stack ... Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/31/2014 7:43 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Elasticsearch destination Hi, On Fri, Oct 31, 2014 at 07:05:47AM -0400, Jim Hendrick wrote: > area. That is one of the reasons I am trying to have syslog-ng do as > much as possible so I can remove "L" and only use "E" and "K" :-) yeah, let's call it the SNEK stack, or SNERK when using Riemann too :o) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

5 4

Re: [syslog-ng] Elasticsearch destination
by Jim Hendrick 31 Oct '14

31 Oct '14

Or reman or "R" .. Hey .. this might actually be a thing! Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/31/2014 7:43 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Elasticsearch destination Hi, On Fri, Oct 31, 2014 at 07:05:47AM -0400, Jim Hendrick wrote: > area. That is one of the reasons I am trying to have syslog-ng do as > much as possible so I can remove "L" and only use "E" and "K" :-) yeah, let's call it the SNEK stack, or SNERK when using Riemann too :o) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 28 Oct '14

28 Oct '14

Thanks!I am on vacation this week but I look forward to trying out these suggestions. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/06/2014 11:42 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Mon, Oct 6, 2014 at 9:36 AM, Radu Gheorghe <radu.gheorghe(a)sematext.com> wrote: Hi Jim, With rabbitmq you have the advantage that you can install the RabbitMQ river and have Elasticsearch pull logs from Rabbit instead of having another [moving] piece pull logs from Rabbit and push them to ES. So you'd have a simpler setup that also makes sure ES isn't overwhelmed (because ES is pulling). Another point that can be an advantage is that with Redis you are limited to the RAM memory available on the machine (all your data need to fit in memory) , with RabbitMQ you have disk persistence that can help in cases where you need to stop ES consumption by any reason. There are some problems with this approach: - the river only runs on one node at a time, which may become a bottleneck - rivers are deprecated (or will be) so the ES side isn't actively maintained. I've seen failover issues (node running the river goes down, another node should start the river but doesn't) which needed river delete + recreate to kick the process in again I didn't know that, I will check it out. Logstash started by recommending RabbitMQ as the queue between two Logstash instances, but now moved to Redis. Apparently the reason is that Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from the guide: "Previous versions of this guide used AMQP via RabbitMQ. Due to the complexity of AMQP as well as performance issues related to the Bunny driver we use, we're now recommending Redis instead." Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 4

improving SQLite writing performances (WAS: Re: inserts into a sqlite3 database are not delayed)
by Davide Alberani 27 Oct '14

27 Oct '14

On Wed, Oct 15, 2014 at 10:56 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: > > It immediately runs the query, flush_lines() controls the transaction size, > which can be enabled using > flags(explicit-commits). Ok, I see and I can confirm noticeable improvements on a system under heavy I/O and CPU load using explicit-commits (or better: I see it gets a lot worse disabling it, since I already had it on ;)) Plus, I'm seeing minor improvements with session_statements set to "PRAGMA synchronous=OFF; PRAGMA count_changes=OFF;" Beside this, has anyone found other ways to improve writing performances using syslog-ng with sqlite? On (physical) systems with high load, I've noticed that the system is much more responsive if the elevator is set to noop or deadline instead of cfq. But I guess this is highly dependent on the specific circumstances of the system. Thanks! -- Davide Alberani <davide.alberani(a)gmail.com> [PGP KeyID: 0x465BFD47] http://www.mimante.net/

2 1

errors running configure for incubator on Ubuntu 12.04
by Russell Fulton 25 Oct '14

25 Oct '14

rful011@secmontst01:~/src/syslog-ng-incubator$ ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane… yes [ snip ] checking whether g++ accepts -g... yes checking dependency style of g++... gcc3 checking for inline... inline ./configure: line 4534: syntax error near unexpected token `shared' ./configure: line 4534: `LT_INIT(shared dlopen)’ I have looked at configure and config.log to try and figure out what it is actually checking for. ‘inline’ is to generic to google for :( any ideas what is wrong? Russell

2 1