- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Host variable substitution in syslog-ng.conf include statement
by Broome, John P - OASAM OCIO CTR 03 Jan '17

03 Jan '17

Robert, Thanks for your reply, but we are constrained to version 3.2 for now (has to be in the RHEL EPEL). I did not see this feature discussed in the 3.2 admin guide. Is it an undocumented feature of version 3.2, or did support for system environment variables come in a later version? Thank you, JB

2 1

Host variable substitution in syslog-ng.conf include statement
by Broome, John P - OASAM OCIO CTR 03 Jan '17

03 Jan '17

I am supporting a large number of syslog-ng hosts and we manage the configuration files with gitlab. I have been trying to figure out how to reference a variable based on hostname in the syslog-ng.conf file on each machine. For example, I would like to be able to have an include statement based on hostname - this would allow our common repository to have an exception set of filters/rules for each host but to use the exact same syslog-ng.conf file everywhere: ################################## syslog-ng.conf ########################################### @version: 3.2 Include "`hostname`.conf" ... Is there a way to accomplish this in version 3.2 (if not, does it appear in a later version)? We are restricted to EPEL6 included versions of syslog-ng, so that's why we're at version 3.2 - JB

2 1

Re: [syslog-ng] When is 3.8.2 scheduled?
by Balazs Scheidler 23 Dec '16

23 Dec '16

Well, you would want to backport 3.8.2 into existing distro releases? Because openssl 1.1 is I thought relevant to the bleeding edge. On Dec 23, 2016 08:06, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: Hi, Which is problematic in Linux distributions, where I have 3.8.1 (openSUSE, Fedora). There are tons of bugfixes, but while I can easily push 3.8.2 as an update, 3.9.1 is not an option as it also requires a config file modification. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Fri, Dec 23, 2016 at 8:01 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: > We decided to skip 3.8.2 and go straight for 3.9.1. > > On Dec 21, 2016 08:19, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: > >> Hi, >> >> Looking at the 3.8/master branch I see, that openssl 1.1 support is >> not included. I'd need that for Fedora 26 support. Could you add it? >> >> Thanks, >> Peter Czanik (CzP) <peter.czanik(a)balabit.com> >> Balabit / syslog-ng upstream >> https://www.balabit.com/blog/author/peterczanik/ >> https://twitter.com/PCzanik >> >> >> On Tue, Dec 20, 2016 at 8:54 AM, Laszlo Budai <laszlo.budai(a)outlook.com> >> wrote: >> > hi, >> > >> > well... searchguard could be part of 3.8.2, >> > so we can continue with this state. >> > >> > >> > >> > >> > _____________________________ >> > From: Scheidler, Balázs <balazs.scheidler(a)balabit.com> >> > Sent: Tuesday, December 20, 2016 8:48 AM >> > Subject: Re: [syslog-ng] When is 3.8.2 scheduled? >> > To: Syslog-ng users' and developers' mailing list >> > <syslog-ng(a)lists.balabit.hu> >> > >> > >> > I have just pushed a preliminary NEWS.md update on the 3.8/master >> branch. >> > Note that as it currently stands SearchGuard is part of 3.8/master. >> > >> > @lbudai: what do you think, shall we continue with this state, or reset >> the >> > branch to an earlier point and push these changes to 3.9 instead? >> > >> > -- >> > Bazsi >> > >> > On Mon, Dec 19, 2016 at 5:39 PM, Laszlo Budai <laszlo.budai(a)outlook.com >> > >> > wrote: >> >> >> >> Hi, >> >> >> >> I think it won't be part of 3.8.2 , but 3.9. >> >> >> >> Our plan is to release both versions in this year. >> >> >> >> regards, >> >> Laszlo Budai >> >> >> >> >> >> >> >> >> >> On Mon, Dec 19, 2016 at 5:36 PM +0100, "Fabien Wernli" < >> wernli(a)in2p3.fr> >> >> wrote: >> >> >> >>> Dear list,Are there plans for a 3.8.2 release?The reason I'm asking is >> >>> that I'm writing a blog post on Searchguardintegration, and this >> relies on >> >>> unreleased >> >>> features.Thanks!____________________________________________ >> __________________________________Member >> >>> info: https://lists.balabit.hu/mailman/listinfo/syslog-ngDocumenta >> tion: >> >>> http://www.balabit.com/support/documentation/?product=syslog-ngFAQ: >> >>> http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> >> >> >> ____________________________________________________________ >> __________________ >> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> >> Documentation: >> >> http://www.balabit.com/support/documentation/?product=syslog-ng >> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> > >> > >> > >> > >> > ____________________________________________________________ >> __________________ >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > Documentation: >> > http://www.balabit.com/support/documentation/?product=syslog-ng >> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > >> > >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support >> /documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product= > syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 3

When is 3.8.2 scheduled?
by Fabien Wernli 23 Dec '16

23 Dec '16

Dear list, Are there plans for a 3.8.2 release? The reason I'm asking is that I'm writing a blog post on Searchguard integration, and this relies on unreleased features. Thanks!

5 6

Unofficial syslog-ng-3.9.1 Debian/Ubuntu packages
by Laszlo Budai 21 Dec '16

21 Dec '16

Hi, syslog-ng 3.9.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 * Ubuntu 16.10 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng-3.9.1 has been released
by Laszlo Budai 21 Dec '16

21 Dec '16

3.9.1 Features * Improve parsing performance in case of keep-timestamp(no), earlier the timestamp was parsed and then dropped, now we don't parse it, which is a 2x performance improvement in reception speed. * TLS based transports will publish the peer's certificate in a set of name-value pairs, as follows: * .tls.x509_cn - X.509 common name * .tls.x509_o - X.509 organization string * .tls.x509_ou - X.509 organizational unit * Improve performance of the tcp() source, due to a bug, syslog-ng attempted to apply position tracking to messages coming over a TCP transport, which is used for file position tracking and causing performance degradation. This bug is fixed, causing performance to be increased. (#1195<https://github.com/balabit/syslog-ng/pull/1195>) * Make it possible to configure the listen-backlog() for any stream based transports (unix-stream and tcp). Earlier this was hard-wired at 256 connections, now can be tuned using an option. For example: tcp(port(6514) listen-backlog(2048)); * Add a groupunset() rewrite rule that pairs up with groupset() but instead of setting values it unsets them. (#1235<https://github.com/balabit/syslog-ng/pull/1235>) * Add support for Elastic Shield (#1228<https://github.com/balabit/syslog-ng/pull/1228>) and SearchGuard (#1223<https://github.com/balabit/syslog-ng/pull/1223>) * kv-parser() is now able to cope with unquoted values with an embedded space in them, it also trims whitespace from keys/values and is in general more reliable in extracting key-value pairs from arbitrary log messages. * Improve performance for java based destinations. (#1243<https://github.com/balabit/syslog-ng/pull/1243>) * Add prefix() option to add-contextual-data() Bugfixes * Fix a potential crash in the file destination, in case it is a template based filename and time-reap() is elapsed. (#1183<https://github.com/balabit/syslog-ng/pull/1183>) * Fix a potential ACK problem within syslog-ng that can cause input windows to overflow queue sizes over time, effectively causing message drops that shouldn't occur. (#1230<https://github.com/balabit/syslog-ng/pull/1230>) * Fix a heap corruption bug in the DNS cache, in case the maximum number of DNS cache entries is reached. (#1218<https://github.com/balabit/syslog-ng/pull/1218>) * Fix timestamp for suppression messages. (#1233<https://github.com/balabit/syslog-ng/pull/1233>) * Fix add-contextual-data() to support CRLF line endings in its CSV input files. * Fixed key() option parsing in riemann() destinations. * Find libsystemd-journal related functions in both libsystemd-journal.so and libsystemd.so, as recent systemd versions bundled all systemd related libs into the same library. * Fixed the build-time detection of system-wide installed librabbitmq, libmongoc and libcap. * Fix the file source to repeatedly check for unexisting files, as a bug caused syslog-ng to stop after two attempts previously. (#841<https://github.com/balabit/syslog-ng/issues/841>) * The performance testing tool "loggen" crashed if it was used to generate messages on multiple threads over TLS. This was now fixed. (#1182<https://github.com/balabit/syslog-ng/issues/1182>) * Fix an issue in the syslog-parser() parser, so that timestamps parsed earlier in the log path are properly overwritten. Earlier a time-zone setting may have remained in the timestamp in case the first timestamp did contain a timezone and then the one parsed by syslog-parser() didn't. (#1206<https://github.com/balabit/syslog-ng/pull/1206>) * Due to a compilation issue, tcp-keepalive-time(), tcp-keepalive-intvl() and tcp-keepalive-probes() were not working, now they are again. (#1214<https://github.com/balabit/syslog-ng/pull/1214>) * The --disable-shm-counters option is now passed to mongo-c-driver to work around a minor security issue (#1219<https://github.com/balabit/syslog-ng/pull/1219>). https://jira.mongodb.org/plugins/servlet/mobile#issue/CDRIVER-1691/comment/… * Fix compilation issues on FreeBSD. (#1252<https://github.com/balabit/syslog-ng/pull/1252>) * Add support to month names in all caps in syslog timestamps. At least one device seems to generate these. (#1263<https://github.com/balabit/syslog-ng/pull/1263>) * The options() option to java destination can now accept numbers and not just strings. * Fix a memory leak in the java destination driver, that may affect java based destinations like ElasticSearch, Kafka & HDFS. Other changes * HDFS was updated to 2.7.3 * Elasticsearch was updated to 2.4.0 * Support was added for OpenSSL 1.1.x (#1281<https://github.com/balabit/syslog-ng/pull/1281>) #Notes to the Developers * We started to standardize our tests on the criterion unit testing framework, please submit all new tests using this framework. Patches to convert existing ones are also welcome. https://github.com/Snaipe/Criterion * We also added a configuration file for astyle and accompanying make targets to check/reformat the source code to meet syslog-ng's style. * debian/ directory has been removed from the "master" branch and is now maintained in a separate "release" branch. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Lászlo Várady, 0xddaa, Balázs Scheidler, Tamás Nagy, László Budai, Fabien Wernli, Viktor Juhász, Kyle Manna, Michael Wimpy, Noémi Ványi, Attila Szalay, Tibor Bodnár, Zoltán Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [https://avatars3.githubusercontent.com/u/986981?v=3&s=400]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> github.com syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. regards, Laszlo Budai

1 0

how to install the latest stable syslog-ng on Debian / Ubuntu
by Czanik, Péter 21 Dec '16

21 Dec '16

Hi, As a follow up to my RPM blog last week, here comes an updated version of my blog tailored to Debian and Ubuntu users: how to install the latest stable syslog-ng packages. https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Parsing Messages for Elasticsearch
by Jentz, Tim 21 Dec '16

21 Dec '16

Hi, I get a lot of name="value" formated messages which I want to receive in syslog-ng and pass them into elasticsearch via the elasticsearch2 module. The message passing works fine, however I'm not able to parse the messages for elasticsearch yet. What I do get in elasticsearch is the whole syslog as a string in the message, what I actually want is all the keys as fields with the value. My elasticsearch destination is configured as followed: destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elastic-5-lib/lib") client-mode("http") cluster("ng") index("ng-${YEAR}.${MONTH}.${DAY}") type("syslog") cluster-url("http://172.18.1.5:9200/") template("$(format-json --scope nv-pairs --exclude R_DATE --key ISODATE)\n") flush-limit("100") concurrent-requests("10") disk-buffer( disk-buf-size(500000000) dir("/opt/disk-buffer") reliable(yes) ) ); }; I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all? Thanks Tim

3 2

Insider 2016-12: latest RPMs; heat maps; Elasticsearch 5;
by Czanik, Péter 16 Dec '16

16 Dec '16

Dear syslog-ng users, This is the 53rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why and how to install the latest syslog-ng RPMs ------------------------------------------------ The syslog-ng application is included in all major Linux distributions. Learn why it is still worth using unofficial repositories providing the latest version of syslog-ng. This blog provides instructions for RPM distributions, like RHEL or SUSE: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Creating heat maps using syslog-ng and Kibana --------------------------------------------- Using the key=value parser of syslog-ng you can easily extract IP addresses from different firewall log messages. The GeoIP parser can add the geolocation belonging to the IP address and Kibana can display the results on a map. Learn how to do this from https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Getting started with Elasticsearch 5 ------------------------------------ For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. As Elastic Stack 5.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0 on RHEL/CentOS 7: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… Fedora and openSUSE now feature syslog-ng 3.8.1 ----------------------------------------------- The second half of November brought us two exciting new Linux distribution releases: openSUSE Leap 42.2 and Fedora 25. Both of them are based on the RPM packaging format and cover everything from embedded through desktops to servers. While there are considerable differences: both feature the latest syslog-ng release, version 3.8.1: https://www.balabit.com/blog/fedora-opensuse-what-is-common-in-the-latest-r… NEW RELEASES - syslog-ng PE 7.0: https://www.balabit.com/blog/syslog-ng-premium-edition-7-released/ - SSB 4.6: https://www.balabit.com/blog/syslog-ng-store-box-goes-to-the-cloud/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Syslog-ng Crashes on CentOS Linux release 7.2.1511 (Core) - Segfaults
by Traiano Welcome 13 Dec '16

13 Dec '16

Hi List syslog-ng 3.5.6 fails to start up on a newly installed CentOS 7 server: Environment details: OS: 3.10.0-327.10.1.el7.x86_64 Distro: CentOS Linux release 7.2.1511 (Core) Syslog-ng: syslog-ng-3.5.6-3.el7.x86_64 I'm doing a fresh install on CentOS linux and installing syslog ng from epel-release repo. syslog-ng fails to start up via systemd as follows: systemctl restart syslog-ng Job for syslog-ng.service failed because a fatal signal was delivered to the control process. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. Looking at journalctl -xe output: - Unit syslog-ng.service has failed. -- -- The result is failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Unit syslog-ng.service entered failed state. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Dec 12 09:40:40 ip-172-31-23-167 kernel: syslog-ng[9920]: segfault at 0 ip (null) sp 00007fff5bbba5b8 error 14 in syslog-ng[400000+2000] Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service: main process exited, code=killed, status=11/SEGV Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. Would anyone know how to fix this, or if this is a bug? Thanks in advance, Traiano

2 1