December 2016 - syslog-ng - lists.balabit.hu

Insider 2016-12: latest RPMs; heat maps; Elasticsearch 5;
by Czanik, Péter 16 Dec '16

16 Dec '16

Dear syslog-ng users, This is the 53rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why and how to install the latest syslog-ng RPMs ------------------------------------------------ The syslog-ng application is included in all major Linux distributions. Learn why it is still worth using unofficial repositories providing the latest version of syslog-ng. This blog provides instructions for RPM distributions, like RHEL or SUSE: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Creating heat maps using syslog-ng and Kibana --------------------------------------------- Using the key=value parser of syslog-ng you can easily extract IP addresses from different firewall log messages. The GeoIP parser can add the geolocation belonging to the IP address and Kibana can display the results on a map. Learn how to do this from https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Getting started with Elasticsearch 5 ------------------------------------ For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. As Elastic Stack 5.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0 on RHEL/CentOS 7: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… Fedora and openSUSE now feature syslog-ng 3.8.1 ----------------------------------------------- The second half of November brought us two exciting new Linux distribution releases: openSUSE Leap 42.2 and Fedora 25. Both of them are based on the RPM packaging format and cover everything from embedded through desktops to servers. While there are considerable differences: both feature the latest syslog-ng release, version 3.8.1: https://www.balabit.com/blog/fedora-opensuse-what-is-common-in-the-latest-r… NEW RELEASES - syslog-ng PE 7.0: https://www.balabit.com/blog/syslog-ng-premium-edition-7-released/ - SSB 4.6: https://www.balabit.com/blog/syslog-ng-store-box-goes-to-the-cloud/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Syslog-ng Crashes on CentOS Linux release 7.2.1511 (Core) - Segfaults
by Traiano Welcome 13 Dec '16

13 Dec '16

Hi List syslog-ng 3.5.6 fails to start up on a newly installed CentOS 7 server: Environment details: OS: 3.10.0-327.10.1.el7.x86_64 Distro: CentOS Linux release 7.2.1511 (Core) Syslog-ng: syslog-ng-3.5.6-3.el7.x86_64 I'm doing a fresh install on CentOS linux and installing syslog ng from epel-release repo. syslog-ng fails to start up via systemd as follows: systemctl restart syslog-ng Job for syslog-ng.service failed because a fatal signal was delivered to the control process. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. Looking at journalctl -xe output: - Unit syslog-ng.service has failed. -- -- The result is failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Unit syslog-ng.service entered failed state. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Dec 12 09:40:40 ip-172-31-23-167 kernel: syslog-ng[9920]: segfault at 0 ip (null) sp 00007fff5bbba5b8 error 14 in syslog-ng[400000+2000] Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service: main process exited, code=killed, status=11/SEGV Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. Would anyone know how to fix this, or if this is a bug? Thanks in advance, Traiano

2 1

why and how to install the latest syslog-ng RPMs
by Czanik, Péter 13 Dec '16

13 Dec '16

Hi, There are two returning questions about syslog-ng RPMs: - where to find the latest packages? - why to use unofficial packages if syslog-ng is available in distros? I combined the answers into a single blog: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

character handling in kv-parser
by David Boles (dboles) 13 Dec '16

13 Dec '16

I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal". When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings. Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact? Thanks, - db

4 8

TimeStamps
by David Campeau 08 Dec '16

08 Dec '16

Hello, Using a syslog-ng sever to filter syslogs before forwarding. I'm being asked to not change the timestamp in the syslog message. I've tried the "keep-timestamp(yes);" option in the syslog-ng.conf, but there's no change in the timestamp. There must be an option I'm missing? Example of the change: Dec 07 15:08:57 <<<< Not filtered by syslog-ng 1 2016-12-07T15:07:32-06:00 <<< Filtering currently with syslog-ng Thank you for looking Best Regards, David

2 2

elasticsearch 5 on rhe 6.4
by Russell Fulton 08 Dec '16

08 Dec '16

Hi I had syslog-ng 3.8.1 (built from tar ball) working nicely with ES 2.x running on the local machine as a pilot. Last week I got two new machines to run ES on with lots of memory and SSD. I installed ES 5.x on them and upgraded ES to 5.x on the original pilot machine. So I now have a cluster of 3. Back to the original machine that has the feed from syslog-ng I was puzzled by the stream of error messages regarding 2.x client that was trying to connect. It took me a while to figure out that this must be the syslog-ng plug in. Stopping syslog-ng stopped the errors. A web search quickly found: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… So I set about reinstalling syslog-ng from Peter’s repos — there were some issues but I got there without too much problem. (see notes at the end). My config file matched the blog example pretty closely - I need to add the http transport for ES 5.x but that was the only change, When I tried to start the new version of syslog-ng I get an error saying that it can not find the elasticsearch2 plugin. I can’t find any reference to insgtalling the plugin in the post. Did I miss something. My old config also had @module mod-java line but this does not appear to make any difference. I have gone back to the 3.8 manual but can not find anything about having to install the ES plugin so I am thoroughly puzzled. Any ideas what is wrong? Russell Problems with the syslog-ng epel 6 repos: since I was on RHE 6 I figured I needed the epel6 rather than the epel7 repo so I wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-6/czan… but when I tried to install Yum said https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-6-x…: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found” After a little poking I figured that the url should be /syslog-ng38eple6/ and that worked.

2 1

how to stop the processing if the filter return true?
by Jorge Pereira 07 Dec '16

07 Dec '16

Hi folks, Basically, in my setup, I received a jSON and the all usual (default) syslog packets in my server. But, I have a simple filter "f_blacklist_network_by_clientip" based in a jSON field. everything works well, the problem is because when my filter "f_blacklist_network_by_clientip" return OK, the junction() goes to the next statement processing the d_default_handler() destination. My doubt is: how to stop the processing when my filter f_blacklist_network_by_clientip() returns OK? *<SNIP>* log { source(s_internet); junction { channel { filter(f_wb_access_log); parser(p_msg2json); filter(f_blacklist_network_by_clientip); destination(d_wb_access_log); flags(final); }; # Default destination channel { * # DOUBT: how to don't processing if the f_blacklist_network_by_clientip return OK?* destination(d_default_handler); }; }; }; filter f_wb_access_log { program("wb_access"); }; parser p_msg2json { json-parser( marker("") prefix("j.")); }; filter f_blacklist_network_by_clientip { not match("^127\.0\.", value("j.clientip")); not match("^172\.16\.", value("j.clientip")); not match("^172\.26\.", value("j.clientip")); not match("^172\.31\.", value("j.clientip")); * # if match, stop the processing and don't jump to "channel { destination(d_default_handler); };* }; destination d_wb_access_log { file("/var/log/syslog-ng/wb/${j.webapp_domain:-invalid_gw_access}_access.log" create_dirs(yes) template("${MSG}\n") ); }; destination d_default_handler { file("/var/log/syslog-ng/servers/${HOST}/${FACILITY:-invalid_facility}.log" create_dirs(yes) ); }; *</SNIP>*

2 1