December 2016 - syslog-ng - lists.balabit.hu

RPM packages from git HEAD
by Czanik, Péter 23 Jan '17

23 Jan '17

Hi, Recently I started to create RPM packages from git HEAD. Obviously this is not for production, but helps to check, if a problem reported for 3.8.1 or earlier still exists. It is available for openSUSE, Fedora, RHEL 7 and SLES 11/12. I don't create RHEL 6 packages, as that would require maintaining a third spec file and repo, but if there are too many people trying to convince me, I can do it next year :) And the good news is, that I learned right after publishing my blog, that 3.8.2 is nearing and expected to arrive this year! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

5 6

unofficial syslog-ng 3.9.1 RPM packages
by Czanik, Péter 13 Jan '17

13 Jan '17

Hi, My unofficial syslog-ng 3.9.1 RPM packages are ready. They are now available from my "githead" repositories. This way I did not have to go through setting up separate 3.9 repositories over a very slow Internet connection and 3.9.1 is git HEAD right now anyway :) Supported distributions are: openSUSE Leap 42.1 and 42.2 SLES 11 and 12 Fedora 24, 25, 26 (rawhide) RHEL / CentOS 7 Instructions are available at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 1

Enable TLS encryption
by Sathish Sundaravel 09 Jan '17

09 Jan '17

Hi Team I have a Syslog Ng server running on Solaris and receiving events from native Solaris syslog clients. How do I enable UDP traffic pass through via TLS and how to configure client certificate for native Solaris syslog agent. We don't have rsyslog.

6 10

Adding specific information from Syslog-ng to Redis
by johnsc301＠gmail.com 07 Jan '17

07 Jan '17

I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); }; I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis. I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters. Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example). I did find this list of parameters: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this? Sent from Mail for Windows 10

4 3

Re: [syslog-ng] When is 3.8.2 scheduled?
by Balazs Scheidler 23 Dec '16

23 Dec '16

Well, you would want to backport 3.8.2 into existing distro releases? Because openssl 1.1 is I thought relevant to the bleeding edge. On Dec 23, 2016 08:06, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: Hi, Which is problematic in Linux distributions, where I have 3.8.1 (openSUSE, Fedora). There are tons of bugfixes, but while I can easily push 3.8.2 as an update, 3.9.1 is not an option as it also requires a config file modification. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Fri, Dec 23, 2016 at 8:01 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: > We decided to skip 3.8.2 and go straight for 3.9.1. > > On Dec 21, 2016 08:19, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: > >> Hi, >> >> Looking at the 3.8/master branch I see, that openssl 1.1 support is >> not included. I'd need that for Fedora 26 support. Could you add it? >> >> Thanks, >> Peter Czanik (CzP) <peter.czanik(a)balabit.com> >> Balabit / syslog-ng upstream >> https://www.balabit.com/blog/author/peterczanik/ >> https://twitter.com/PCzanik >> >> >> On Tue, Dec 20, 2016 at 8:54 AM, Laszlo Budai <laszlo.budai(a)outlook.com> >> wrote: >> > hi, >> > >> > well... searchguard could be part of 3.8.2, >> > so we can continue with this state. >> > >> > >> > >> > >> > _____________________________ >> > From: Scheidler, Balázs <balazs.scheidler(a)balabit.com> >> > Sent: Tuesday, December 20, 2016 8:48 AM >> > Subject: Re: [syslog-ng] When is 3.8.2 scheduled? >> > To: Syslog-ng users' and developers' mailing list >> > <syslog-ng(a)lists.balabit.hu> >> > >> > >> > I have just pushed a preliminary NEWS.md update on the 3.8/master >> branch. >> > Note that as it currently stands SearchGuard is part of 3.8/master. >> > >> > @lbudai: what do you think, shall we continue with this state, or reset >> the >> > branch to an earlier point and push these changes to 3.9 instead? >> > >> > -- >> > Bazsi >> > >> > On Mon, Dec 19, 2016 at 5:39 PM, Laszlo Budai <laszlo.budai(a)outlook.com >> > >> > wrote: >> >> >> >> Hi, >> >> >> >> I think it won't be part of 3.8.2 , but 3.9. >> >> >> >> Our plan is to release both versions in this year. >> >> >> >> regards, >> >> Laszlo Budai >> >> >> >> >> >> >> >> >> >> On Mon, Dec 19, 2016 at 5:36 PM +0100, "Fabien Wernli" < >> wernli(a)in2p3.fr> >> >> wrote: >> >> >> >>> Dear list,Are there plans for a 3.8.2 release?The reason I'm asking is >> >>> that I'm writing a blog post on Searchguardintegration, and this >> relies on >> >>> unreleased >> >>> features.Thanks!____________________________________________ >> __________________________________Member >> >>> info: https://lists.balabit.hu/mailman/listinfo/syslog-ngDocumenta >> tion: >> >>> http://www.balabit.com/support/documentation/?product=syslog-ngFAQ: >> >>> http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> >> >> >> ____________________________________________________________ >> __________________ >> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> >> Documentation: >> >> http://www.balabit.com/support/documentation/?product=syslog-ng >> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> > >> > >> > >> > >> > ____________________________________________________________ >> __________________ >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > Documentation: >> > http://www.balabit.com/support/documentation/?product=syslog-ng >> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > >> > >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support >> /documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product= > syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 3

When is 3.8.2 scheduled?
by Fabien Wernli 23 Dec '16

23 Dec '16

Dear list, Are there plans for a 3.8.2 release? The reason I'm asking is that I'm writing a blog post on Searchguard integration, and this relies on unreleased features. Thanks!

5 6

Unofficial syslog-ng-3.9.1 Debian/Ubuntu packages
by Laszlo Budai 21 Dec '16

21 Dec '16

Hi, syslog-ng 3.9.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 * Ubuntu 16.10 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng-3.9.1 has been released
by Laszlo Budai 21 Dec '16

21 Dec '16

3.9.1 Features * Improve parsing performance in case of keep-timestamp(no), earlier the timestamp was parsed and then dropped, now we don't parse it, which is a 2x performance improvement in reception speed. * TLS based transports will publish the peer's certificate in a set of name-value pairs, as follows: * .tls.x509_cn - X.509 common name * .tls.x509_o - X.509 organization string * .tls.x509_ou - X.509 organizational unit * Improve performance of the tcp() source, due to a bug, syslog-ng attempted to apply position tracking to messages coming over a TCP transport, which is used for file position tracking and causing performance degradation. This bug is fixed, causing performance to be increased. (#1195<https://github.com/balabit/syslog-ng/pull/1195>) * Make it possible to configure the listen-backlog() for any stream based transports (unix-stream and tcp). Earlier this was hard-wired at 256 connections, now can be tuned using an option. For example: tcp(port(6514) listen-backlog(2048)); * Add a groupunset() rewrite rule that pairs up with groupset() but instead of setting values it unsets them. (#1235<https://github.com/balabit/syslog-ng/pull/1235>) * Add support for Elastic Shield (#1228<https://github.com/balabit/syslog-ng/pull/1228>) and SearchGuard (#1223<https://github.com/balabit/syslog-ng/pull/1223>) * kv-parser() is now able to cope with unquoted values with an embedded space in them, it also trims whitespace from keys/values and is in general more reliable in extracting key-value pairs from arbitrary log messages. * Improve performance for java based destinations. (#1243<https://github.com/balabit/syslog-ng/pull/1243>) * Add prefix() option to add-contextual-data() Bugfixes * Fix a potential crash in the file destination, in case it is a template based filename and time-reap() is elapsed. (#1183<https://github.com/balabit/syslog-ng/pull/1183>) * Fix a potential ACK problem within syslog-ng that can cause input windows to overflow queue sizes over time, effectively causing message drops that shouldn't occur. (#1230<https://github.com/balabit/syslog-ng/pull/1230>) * Fix a heap corruption bug in the DNS cache, in case the maximum number of DNS cache entries is reached. (#1218<https://github.com/balabit/syslog-ng/pull/1218>) * Fix timestamp for suppression messages. (#1233<https://github.com/balabit/syslog-ng/pull/1233>) * Fix add-contextual-data() to support CRLF line endings in its CSV input files. * Fixed key() option parsing in riemann() destinations. * Find libsystemd-journal related functions in both libsystemd-journal.so and libsystemd.so, as recent systemd versions bundled all systemd related libs into the same library. * Fixed the build-time detection of system-wide installed librabbitmq, libmongoc and libcap. * Fix the file source to repeatedly check for unexisting files, as a bug caused syslog-ng to stop after two attempts previously. (#841<https://github.com/balabit/syslog-ng/issues/841>) * The performance testing tool "loggen" crashed if it was used to generate messages on multiple threads over TLS. This was now fixed. (#1182<https://github.com/balabit/syslog-ng/issues/1182>) * Fix an issue in the syslog-parser() parser, so that timestamps parsed earlier in the log path are properly overwritten. Earlier a time-zone setting may have remained in the timestamp in case the first timestamp did contain a timezone and then the one parsed by syslog-parser() didn't. (#1206<https://github.com/balabit/syslog-ng/pull/1206>) * Due to a compilation issue, tcp-keepalive-time(), tcp-keepalive-intvl() and tcp-keepalive-probes() were not working, now they are again. (#1214<https://github.com/balabit/syslog-ng/pull/1214>) * The --disable-shm-counters option is now passed to mongo-c-driver to work around a minor security issue (#1219<https://github.com/balabit/syslog-ng/pull/1219>). https://jira.mongodb.org/plugins/servlet/mobile#issue/CDRIVER-1691/comment/… * Fix compilation issues on FreeBSD. (#1252<https://github.com/balabit/syslog-ng/pull/1252>) * Add support to month names in all caps in syslog timestamps. At least one device seems to generate these. (#1263<https://github.com/balabit/syslog-ng/pull/1263>) * The options() option to java destination can now accept numbers and not just strings. * Fix a memory leak in the java destination driver, that may affect java based destinations like ElasticSearch, Kafka & HDFS. Other changes * HDFS was updated to 2.7.3 * Elasticsearch was updated to 2.4.0 * Support was added for OpenSSL 1.1.x (#1281<https://github.com/balabit/syslog-ng/pull/1281>) #Notes to the Developers * We started to standardize our tests on the criterion unit testing framework, please submit all new tests using this framework. Patches to convert existing ones are also welcome. https://github.com/Snaipe/Criterion * We also added a configuration file for astyle and accompanying make targets to check/reformat the source code to meet syslog-ng's style. * debian/ directory has been removed from the "master" branch and is now maintained in a separate "release" branch. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Lászlo Várady, 0xddaa, Balázs Scheidler, Tamás Nagy, László Budai, Fabien Wernli, Viktor Juhász, Kyle Manna, Michael Wimpy, Noémi Ványi, Attila Szalay, Tibor Bodnár, Zoltán Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [https://avatars3.githubusercontent.com/u/986981?v=3&s=400]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> github.com syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. regards, Laszlo Budai

1 0

how to install the latest stable syslog-ng on Debian / Ubuntu
by Czanik, Péter 21 Dec '16

21 Dec '16

Hi, As a follow up to my RPM blog last week, here comes an updated version of my blog tailored to Debian and Ubuntu users: how to install the latest stable syslog-ng packages. https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Parsing Messages for Elasticsearch
by Jentz, Tim 21 Dec '16

21 Dec '16

Hi, I get a lot of name="value" formated messages which I want to receive in syslog-ng and pass them into elasticsearch via the elasticsearch2 module. The message passing works fine, however I'm not able to parse the messages for elasticsearch yet. What I do get in elasticsearch is the whole syslog as a string in the message, what I actually want is all the keys as fields with the value. My elasticsearch destination is configured as followed: destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elastic-5-lib/lib") client-mode("http") cluster("ng") index("ng-${YEAR}.${MONTH}.${DAY}") type("syslog") cluster-url("http://172.18.1.5:9200/") template("$(format-json --scope nv-pairs --exclude R_DATE --key ISODATE)\n") flush-limit("100") concurrent-requests("10") disk-buffer( disk-buf-size(500000000) dir("/opt/disk-buffer") reliable(yes) ) ); }; I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all? Thanks Tim

3 2