- syslog-ng - lists.balabit.hu

Unofficial syslog-ng-3.9.1 Debian/Ubuntu packages
by Laszlo Budai 21 Dec '16

21 Dec '16

Hi, syslog-ng 3.9.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 * Ubuntu 16.10 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng-3.9.1 has been released
by Laszlo Budai 21 Dec '16

21 Dec '16

3.9.1 Features * Improve parsing performance in case of keep-timestamp(no), earlier the timestamp was parsed and then dropped, now we don't parse it, which is a 2x performance improvement in reception speed. * TLS based transports will publish the peer's certificate in a set of name-value pairs, as follows: * .tls.x509_cn - X.509 common name * .tls.x509_o - X.509 organization string * .tls.x509_ou - X.509 organizational unit * Improve performance of the tcp() source, due to a bug, syslog-ng attempted to apply position tracking to messages coming over a TCP transport, which is used for file position tracking and causing performance degradation. This bug is fixed, causing performance to be increased. (#1195<https://github.com/balabit/syslog-ng/pull/1195>) * Make it possible to configure the listen-backlog() for any stream based transports (unix-stream and tcp). Earlier this was hard-wired at 256 connections, now can be tuned using an option. For example: tcp(port(6514) listen-backlog(2048)); * Add a groupunset() rewrite rule that pairs up with groupset() but instead of setting values it unsets them. (#1235<https://github.com/balabit/syslog-ng/pull/1235>) * Add support for Elastic Shield (#1228<https://github.com/balabit/syslog-ng/pull/1228>) and SearchGuard (#1223<https://github.com/balabit/syslog-ng/pull/1223>) * kv-parser() is now able to cope with unquoted values with an embedded space in them, it also trims whitespace from keys/values and is in general more reliable in extracting key-value pairs from arbitrary log messages. * Improve performance for java based destinations. (#1243<https://github.com/balabit/syslog-ng/pull/1243>) * Add prefix() option to add-contextual-data() Bugfixes * Fix a potential crash in the file destination, in case it is a template based filename and time-reap() is elapsed. (#1183<https://github.com/balabit/syslog-ng/pull/1183>) * Fix a potential ACK problem within syslog-ng that can cause input windows to overflow queue sizes over time, effectively causing message drops that shouldn't occur. (#1230<https://github.com/balabit/syslog-ng/pull/1230>) * Fix a heap corruption bug in the DNS cache, in case the maximum number of DNS cache entries is reached. (#1218<https://github.com/balabit/syslog-ng/pull/1218>) * Fix timestamp for suppression messages. (#1233<https://github.com/balabit/syslog-ng/pull/1233>) * Fix add-contextual-data() to support CRLF line endings in its CSV input files. * Fixed key() option parsing in riemann() destinations. * Find libsystemd-journal related functions in both libsystemd-journal.so and libsystemd.so, as recent systemd versions bundled all systemd related libs into the same library. * Fixed the build-time detection of system-wide installed librabbitmq, libmongoc and libcap. * Fix the file source to repeatedly check for unexisting files, as a bug caused syslog-ng to stop after two attempts previously. (#841<https://github.com/balabit/syslog-ng/issues/841>) * The performance testing tool "loggen" crashed if it was used to generate messages on multiple threads over TLS. This was now fixed. (#1182<https://github.com/balabit/syslog-ng/issues/1182>) * Fix an issue in the syslog-parser() parser, so that timestamps parsed earlier in the log path are properly overwritten. Earlier a time-zone setting may have remained in the timestamp in case the first timestamp did contain a timezone and then the one parsed by syslog-parser() didn't. (#1206<https://github.com/balabit/syslog-ng/pull/1206>) * Due to a compilation issue, tcp-keepalive-time(), tcp-keepalive-intvl() and tcp-keepalive-probes() were not working, now they are again. (#1214<https://github.com/balabit/syslog-ng/pull/1214>) * The --disable-shm-counters option is now passed to mongo-c-driver to work around a minor security issue (#1219<https://github.com/balabit/syslog-ng/pull/1219>). https://jira.mongodb.org/plugins/servlet/mobile#issue/CDRIVER-1691/comment/… * Fix compilation issues on FreeBSD. (#1252<https://github.com/balabit/syslog-ng/pull/1252>) * Add support to month names in all caps in syslog timestamps. At least one device seems to generate these. (#1263<https://github.com/balabit/syslog-ng/pull/1263>) * The options() option to java destination can now accept numbers and not just strings. * Fix a memory leak in the java destination driver, that may affect java based destinations like ElasticSearch, Kafka & HDFS. Other changes * HDFS was updated to 2.7.3 * Elasticsearch was updated to 2.4.0 * Support was added for OpenSSL 1.1.x (#1281<https://github.com/balabit/syslog-ng/pull/1281>) #Notes to the Developers * We started to standardize our tests on the criterion unit testing framework, please submit all new tests using this framework. Patches to convert existing ones are also welcome. https://github.com/Snaipe/Criterion * We also added a configuration file for astyle and accompanying make targets to check/reformat the source code to meet syslog-ng's style. * debian/ directory has been removed from the "master" branch and is now maintained in a separate "release" branch. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Lászlo Várady, 0xddaa, Balázs Scheidler, Tamás Nagy, László Budai, Fabien Wernli, Viktor Juhász, Kyle Manna, Michael Wimpy, Noémi Ványi, Attila Szalay, Tibor Bodnár, Zoltán Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [https://avatars3.githubusercontent.com/u/986981?v=3&s=400]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> github.com syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. regards, Laszlo Budai

1 0

how to install the latest stable syslog-ng on Debian / Ubuntu
by Czanik, Péter 21 Dec '16

21 Dec '16

Hi, As a follow up to my RPM blog last week, here comes an updated version of my blog tailored to Debian and Ubuntu users: how to install the latest stable syslog-ng packages. https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Parsing Messages for Elasticsearch
by Jentz, Tim 21 Dec '16

21 Dec '16

Hi, I get a lot of name="value" formated messages which I want to receive in syslog-ng and pass them into elasticsearch via the elasticsearch2 module. The message passing works fine, however I'm not able to parse the messages for elasticsearch yet. What I do get in elasticsearch is the whole syslog as a string in the message, what I actually want is all the keys as fields with the value. My elasticsearch destination is configured as followed: destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elastic-5-lib/lib") client-mode("http") cluster("ng") index("ng-${YEAR}.${MONTH}.${DAY}") type("syslog") cluster-url("http://172.18.1.5:9200/") template("$(format-json --scope nv-pairs --exclude R_DATE --key ISODATE)\n") flush-limit("100") concurrent-requests("10") disk-buffer( disk-buf-size(500000000) dir("/opt/disk-buffer") reliable(yes) ) ); }; I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all? Thanks Tim

3 2

Insider 2016-12: latest RPMs; heat maps; Elasticsearch 5;
by Czanik, Péter 16 Dec '16

16 Dec '16

Dear syslog-ng users, This is the 53rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why and how to install the latest syslog-ng RPMs ------------------------------------------------ The syslog-ng application is included in all major Linux distributions. Learn why it is still worth using unofficial repositories providing the latest version of syslog-ng. This blog provides instructions for RPM distributions, like RHEL or SUSE: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Creating heat maps using syslog-ng and Kibana --------------------------------------------- Using the key=value parser of syslog-ng you can easily extract IP addresses from different firewall log messages. The GeoIP parser can add the geolocation belonging to the IP address and Kibana can display the results on a map. Learn how to do this from https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Getting started with Elasticsearch 5 ------------------------------------ For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. As Elastic Stack 5.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0 on RHEL/CentOS 7: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… Fedora and openSUSE now feature syslog-ng 3.8.1 ----------------------------------------------- The second half of November brought us two exciting new Linux distribution releases: openSUSE Leap 42.2 and Fedora 25. Both of them are based on the RPM packaging format and cover everything from embedded through desktops to servers. While there are considerable differences: both feature the latest syslog-ng release, version 3.8.1: https://www.balabit.com/blog/fedora-opensuse-what-is-common-in-the-latest-r… NEW RELEASES - syslog-ng PE 7.0: https://www.balabit.com/blog/syslog-ng-premium-edition-7-released/ - SSB 4.6: https://www.balabit.com/blog/syslog-ng-store-box-goes-to-the-cloud/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Syslog-ng Crashes on CentOS Linux release 7.2.1511 (Core) - Segfaults
by Traiano Welcome 13 Dec '16

13 Dec '16

Hi List syslog-ng 3.5.6 fails to start up on a newly installed CentOS 7 server: Environment details: OS: 3.10.0-327.10.1.el7.x86_64 Distro: CentOS Linux release 7.2.1511 (Core) Syslog-ng: syslog-ng-3.5.6-3.el7.x86_64 I'm doing a fresh install on CentOS linux and installing syslog ng from epel-release repo. syslog-ng fails to start up via systemd as follows: systemctl restart syslog-ng Job for syslog-ng.service failed because a fatal signal was delivered to the control process. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. Looking at journalctl -xe output: - Unit syslog-ng.service has failed. -- -- The result is failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Unit syslog-ng.service entered failed state. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Dec 12 09:40:40 ip-172-31-23-167 kernel: syslog-ng[9920]: segfault at 0 ip (null) sp 00007fff5bbba5b8 error 14 in syslog-ng[400000+2000] Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service: main process exited, code=killed, status=11/SEGV Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. Would anyone know how to fix this, or if this is a bug? Thanks in advance, Traiano

2 1

why and how to install the latest syslog-ng RPMs
by Czanik, Péter 13 Dec '16

13 Dec '16

Hi, There are two returning questions about syslog-ng RPMs: - where to find the latest packages? - why to use unofficial packages if syslog-ng is available in distros? I combined the answers into a single blog: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

character handling in kv-parser
by David Boles (dboles) 13 Dec '16

13 Dec '16

I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal". When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings. Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact? Thanks, - db

4 8

TimeStamps
by David Campeau 08 Dec '16

08 Dec '16

Hello, Using a syslog-ng sever to filter syslogs before forwarding. I'm being asked to not change the timestamp in the syslog message. I've tried the "keep-timestamp(yes);" option in the syslog-ng.conf, but there's no change in the timestamp. There must be an option I'm missing? Example of the change: Dec 07 15:08:57 <<<< Not filtered by syslog-ng 1 2016-12-07T15:07:32-06:00 <<< Filtering currently with syslog-ng Thank you for looking Best Regards, David

2 2

elasticsearch 5 on rhe 6.4
by Russell Fulton 08 Dec '16

08 Dec '16

Hi I had syslog-ng 3.8.1 (built from tar ball) working nicely with ES 2.x running on the local machine as a pilot. Last week I got two new machines to run ES on with lots of memory and SSD. I installed ES 5.x on them and upgraded ES to 5.x on the original pilot machine. So I now have a cluster of 3. Back to the original machine that has the feed from syslog-ng I was puzzled by the stream of error messages regarding 2.x client that was trying to connect. It took me a while to figure out that this must be the syslog-ng plug in. Stopping syslog-ng stopped the errors. A web search quickly found: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… So I set about reinstalling syslog-ng from Peter’s repos — there were some issues but I got there without too much problem. (see notes at the end). My config file matched the blog example pretty closely - I need to add the http transport for ES 5.x but that was the only change, When I tried to start the new version of syslog-ng I get an error saying that it can not find the elasticsearch2 plugin. I can’t find any reference to insgtalling the plugin in the post. Did I miss something. My old config also had @module mod-java line but this does not appear to make any difference. I have gone back to the 3.8 manual but can not find anything about having to install the ES plugin so I am thoroughly puzzled. Any ideas what is wrong? Russell Problems with the syslog-ng epel 6 repos: since I was on RHE 6 I figured I needed the epel6 rather than the epel7 repo so I wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-6/czan… but when I tried to install Yum said https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-6-x…: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found” After a little poking I figured that the url should be /syslog-ng38eple6/ and that worked.

2 1