- syslog-ng - lists.balabit.hu

udp tuning - where to look.
by Jim Hendrick 11 Jan '17

11 Jan '17

Hi, I am working an issue with UDP drops in a high EPS rate environment (several thousand steady) and struggling with making any changes in the loss rate. (according to netstat -su | grep error). This is happening on two nodes, (one running an older syslog-ng 3.2 OSE, the other running a splunk forwarder) but I don't actually think it is making it out of the kernel to either application. On the syslog-ng side syslog-ng-ctl stats shows *no* drops at all. Increasing net.core.rmem_max and so_rcvbuf together all the way to 64 MB did not seem to make any significant difference. This is a RHEL 6 box with 16 GB and 4 cores (virtual - running in an ESX environment) Are there other parameters, things I should be looking at? Thanks, Jim

4 4

CentOS 7.2 // Syslog-ng (syslog-ng_3.6.4)
by vijay amruth 10 Jan '17

10 Jan '17

Hello guys, Happy new year!! Hope you are doing great. I've installed (compiled ) syslog-ng on CentOS Linux 7 (Core) but after I got through the install which is all good, I cant get it to start, all it says is : systemctl status syslog-ng ● syslog-ng.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) [vch@slc ~]$ sudo systemctl enable syslog-ng Failed to execute operation: No such file or directory [vch@slc ~]$ sudo systemctl start syslog-ng Failed to start syslog-ng.service: Unit syslog-ng.service failed to load: No such file or directory. Rsyslog is not running, I can't think of any, I am I missing some simple ?????? Any help is appreciated. -- Thanks, Vijay Amrut.

5 6

Enable TLS encryption
by Sathish Sundaravel 09 Jan '17

09 Jan '17

Hi Team I have a Syslog Ng server running on Solaris and receiving events from native Solaris syslog clients. How do I enable UDP traffic pass through via TLS and how to configure client certificate for native Solaris syslog agent. We don't have rsyslog.

6 10

Adding specific information from Syslog-ng to Redis
by johnsc301＠gmail.com 07 Jan '17

07 Jan '17

I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); }; I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis. I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters. Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example). I did find this list of parameters: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this? Sent from Mail for Windows 10

4 3

v3.9: Core dump on python module
by Clayton Dukes 04 Jan '17

04 Jan '17

Hey guys, just want you to know Ubuntu16 is dumping core on loading the python module. Ubuntu14 does not do this. [2017-01-01T19:54:52.576749] Module loaded and initialized successfully; module='mod-python' Segmentation fault (core dumped) -Clayton Dukes

4 16

Re: [syslog-ng] Host variable substitution in syslog-ng.conf include statement
by Broome, John P - OASAM OCIO CTR 03 Jan '17

03 Jan '17

Robert, Thanks for your reply, but we are constrained to version 3.2 for now (has to be in the RHEL EPEL). I did not see this feature discussed in the 3.2 admin guide. Is it an undocumented feature of version 3.2, or did support for system environment variables come in a later version? Thank you, JB

2 1

Host variable substitution in syslog-ng.conf include statement
by Broome, John P - OASAM OCIO CTR 03 Jan '17

03 Jan '17

I am supporting a large number of syslog-ng hosts and we manage the configuration files with gitlab. I have been trying to figure out how to reference a variable based on hostname in the syslog-ng.conf file on each machine. For example, I would like to be able to have an include statement based on hostname - this would allow our common repository to have an exception set of filters/rules for each host but to use the exact same syslog-ng.conf file everywhere: ################################## syslog-ng.conf ########################################### @version: 3.2 Include "`hostname`.conf" ... Is there a way to accomplish this in version 3.2 (if not, does it appear in a later version)? We are restricted to EPEL6 included versions of syslog-ng, so that's why we're at version 3.2 - JB

2 1

Re: [syslog-ng] When is 3.8.2 scheduled?
by Balazs Scheidler 23 Dec '16

23 Dec '16

Well, you would want to backport 3.8.2 into existing distro releases? Because openssl 1.1 is I thought relevant to the bleeding edge. On Dec 23, 2016 08:06, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: Hi, Which is problematic in Linux distributions, where I have 3.8.1 (openSUSE, Fedora). There are tons of bugfixes, but while I can easily push 3.8.2 as an update, 3.9.1 is not an option as it also requires a config file modification. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Fri, Dec 23, 2016 at 8:01 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: > We decided to skip 3.8.2 and go straight for 3.9.1. > > On Dec 21, 2016 08:19, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: > >> Hi, >> >> Looking at the 3.8/master branch I see, that openssl 1.1 support is >> not included. I'd need that for Fedora 26 support. Could you add it? >> >> Thanks, >> Peter Czanik (CzP) <peter.czanik(a)balabit.com> >> Balabit / syslog-ng upstream >> https://www.balabit.com/blog/author/peterczanik/ >> https://twitter.com/PCzanik >> >> >> On Tue, Dec 20, 2016 at 8:54 AM, Laszlo Budai <laszlo.budai(a)outlook.com> >> wrote: >> > hi, >> > >> > well... searchguard could be part of 3.8.2, >> > so we can continue with this state. >> > >> > >> > >> > >> > _____________________________ >> > From: Scheidler, Balázs <balazs.scheidler(a)balabit.com> >> > Sent: Tuesday, December 20, 2016 8:48 AM >> > Subject: Re: [syslog-ng] When is 3.8.2 scheduled? >> > To: Syslog-ng users' and developers' mailing list >> > <syslog-ng(a)lists.balabit.hu> >> > >> > >> > I have just pushed a preliminary NEWS.md update on the 3.8/master >> branch. >> > Note that as it currently stands SearchGuard is part of 3.8/master. >> > >> > @lbudai: what do you think, shall we continue with this state, or reset >> the >> > branch to an earlier point and push these changes to 3.9 instead? >> > >> > -- >> > Bazsi >> > >> > On Mon, Dec 19, 2016 at 5:39 PM, Laszlo Budai <laszlo.budai(a)outlook.com >> > >> > wrote: >> >> >> >> Hi, >> >> >> >> I think it won't be part of 3.8.2 , but 3.9. >> >> >> >> Our plan is to release both versions in this year. >> >> >> >> regards, >> >> Laszlo Budai >> >> >> >> >> >> >> >> >> >> On Mon, Dec 19, 2016 at 5:36 PM +0100, "Fabien Wernli" < >> wernli(a)in2p3.fr> >> >> wrote: >> >> >> >>> Dear list,Are there plans for a 3.8.2 release?The reason I'm asking is >> >>> that I'm writing a blog post on Searchguardintegration, and this >> relies on >> >>> unreleased >> >>> features.Thanks!____________________________________________ >> __________________________________Member >> >>> info: https://lists.balabit.hu/mailman/listinfo/syslog-ngDocumenta >> tion: >> >>> http://www.balabit.com/support/documentation/?product=syslog-ngFAQ: >> >>> http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> >> >> >> ____________________________________________________________ >> __________________ >> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> >> Documentation: >> >> http://www.balabit.com/support/documentation/?product=syslog-ng >> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> > >> > >> > >> > >> > ____________________________________________________________ >> __________________ >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > Documentation: >> > http://www.balabit.com/support/documentation/?product=syslog-ng >> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > >> > >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support >> /documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product= > syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 3

When is 3.8.2 scheduled?
by Fabien Wernli 23 Dec '16

23 Dec '16

Dear list, Are there plans for a 3.8.2 release? The reason I'm asking is that I'm writing a blog post on Searchguard integration, and this relies on unreleased features. Thanks!

5 6

Unofficial syslog-ng-3.9.1 Debian/Ubuntu packages
by Laszlo Budai 21 Dec '16

21 Dec '16

Hi, syslog-ng 3.9.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 * Ubuntu 16.10 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng-3.9.1 has been released
by Laszlo Budai 21 Dec '16

21 Dec '16

3.9.1 Features * Improve parsing performance in case of keep-timestamp(no), earlier the timestamp was parsed and then dropped, now we don't parse it, which is a 2x performance improvement in reception speed. * TLS based transports will publish the peer's certificate in a set of name-value pairs, as follows: * .tls.x509_cn - X.509 common name * .tls.x509_o - X.509 organization string * .tls.x509_ou - X.509 organizational unit * Improve performance of the tcp() source, due to a bug, syslog-ng attempted to apply position tracking to messages coming over a TCP transport, which is used for file position tracking and causing performance degradation. This bug is fixed, causing performance to be increased. (#1195<https://github.com/balabit/syslog-ng/pull/1195>) * Make it possible to configure the listen-backlog() for any stream based transports (unix-stream and tcp). Earlier this was hard-wired at 256 connections, now can be tuned using an option. For example: tcp(port(6514) listen-backlog(2048)); * Add a groupunset() rewrite rule that pairs up with groupset() but instead of setting values it unsets them. (#1235<https://github.com/balabit/syslog-ng/pull/1235>) * Add support for Elastic Shield (#1228<https://github.com/balabit/syslog-ng/pull/1228>) and SearchGuard (#1223<https://github.com/balabit/syslog-ng/pull/1223>) * kv-parser() is now able to cope with unquoted values with an embedded space in them, it also trims whitespace from keys/values and is in general more reliable in extracting key-value pairs from arbitrary log messages. * Improve performance for java based destinations. (#1243<https://github.com/balabit/syslog-ng/pull/1243>) * Add prefix() option to add-contextual-data() Bugfixes * Fix a potential crash in the file destination, in case it is a template based filename and time-reap() is elapsed. (#1183<https://github.com/balabit/syslog-ng/pull/1183>) * Fix a potential ACK problem within syslog-ng that can cause input windows to overflow queue sizes over time, effectively causing message drops that shouldn't occur. (#1230<https://github.com/balabit/syslog-ng/pull/1230>) * Fix a heap corruption bug in the DNS cache, in case the maximum number of DNS cache entries is reached. (#1218<https://github.com/balabit/syslog-ng/pull/1218>) * Fix timestamp for suppression messages. (#1233<https://github.com/balabit/syslog-ng/pull/1233>) * Fix add-contextual-data() to support CRLF line endings in its CSV input files. * Fixed key() option parsing in riemann() destinations. * Find libsystemd-journal related functions in both libsystemd-journal.so and libsystemd.so, as recent systemd versions bundled all systemd related libs into the same library. * Fixed the build-time detection of system-wide installed librabbitmq, libmongoc and libcap. * Fix the file source to repeatedly check for unexisting files, as a bug caused syslog-ng to stop after two attempts previously. (#841<https://github.com/balabit/syslog-ng/issues/841>) * The performance testing tool "loggen" crashed if it was used to generate messages on multiple threads over TLS. This was now fixed. (#1182<https://github.com/balabit/syslog-ng/issues/1182>) * Fix an issue in the syslog-parser() parser, so that timestamps parsed earlier in the log path are properly overwritten. Earlier a time-zone setting may have remained in the timestamp in case the first timestamp did contain a timezone and then the one parsed by syslog-parser() didn't. (#1206<https://github.com/balabit/syslog-ng/pull/1206>) * Due to a compilation issue, tcp-keepalive-time(), tcp-keepalive-intvl() and tcp-keepalive-probes() were not working, now they are again. (#1214<https://github.com/balabit/syslog-ng/pull/1214>) * The --disable-shm-counters option is now passed to mongo-c-driver to work around a minor security issue (#1219<https://github.com/balabit/syslog-ng/pull/1219>). https://jira.mongodb.org/plugins/servlet/mobile#issue/CDRIVER-1691/comment/… * Fix compilation issues on FreeBSD. (#1252<https://github.com/balabit/syslog-ng/pull/1252>) * Add support to month names in all caps in syslog timestamps. At least one device seems to generate these. (#1263<https://github.com/balabit/syslog-ng/pull/1263>) * The options() option to java destination can now accept numbers and not just strings. * Fix a memory leak in the java destination driver, that may affect java based destinations like ElasticSearch, Kafka & HDFS. Other changes * HDFS was updated to 2.7.3 * Elasticsearch was updated to 2.4.0 * Support was added for OpenSSL 1.1.x (#1281<https://github.com/balabit/syslog-ng/pull/1281>) #Notes to the Developers * We started to standardize our tests on the criterion unit testing framework, please submit all new tests using this framework. Patches to convert existing ones are also welcome. https://github.com/Snaipe/Criterion * We also added a configuration file for astyle and accompanying make targets to check/reformat the source code to meet syslog-ng's style. * debian/ directory has been removed from the "master" branch and is now maintained in a separate "release" branch. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Lászlo Várady, 0xddaa, Balázs Scheidler, Tamás Nagy, László Budai, Fabien Wernli, Viktor Juhász, Kyle Manna, Michael Wimpy, Noémi Ványi, Attila Szalay, Tibor Bodnár, Zoltán Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 [https://avatars3.githubusercontent.com/u/986981?v=3&s=400]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1> github.com syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. regards, Laszlo Budai

1 0

how to install the latest stable syslog-ng on Debian / Ubuntu
by Czanik, Péter 21 Dec '16

21 Dec '16

Hi, As a follow up to my RPM blog last week, here comes an updated version of my blog tailored to Debian and Ubuntu users: how to install the latest stable syslog-ng packages. https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Parsing Messages for Elasticsearch
by Jentz, Tim 21 Dec '16

21 Dec '16

Hi, I get a lot of name="value" formated messages which I want to receive in syslog-ng and pass them into elasticsearch via the elasticsearch2 module. The message passing works fine, however I'm not able to parse the messages for elasticsearch yet. What I do get in elasticsearch is the whole syslog as a string in the message, what I actually want is all the keys as fields with the value. My elasticsearch destination is configured as followed: destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elastic-5-lib/lib") client-mode("http") cluster("ng") index("ng-${YEAR}.${MONTH}.${DAY}") type("syslog") cluster-url("http://172.18.1.5:9200/") template("$(format-json --scope nv-pairs --exclude R_DATE --key ISODATE)\n") flush-limit("100") concurrent-requests("10") disk-buffer( disk-buf-size(500000000) dir("/opt/disk-buffer") reliable(yes) ) ); }; I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all? Thanks Tim

3 2

Insider 2016-12: latest RPMs; heat maps; Elasticsearch 5;
by Czanik, Péter 16 Dec '16

16 Dec '16

Dear syslog-ng users, This is the 53rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why and how to install the latest syslog-ng RPMs ------------------------------------------------ The syslog-ng application is included in all major Linux distributions. Learn why it is still worth using unofficial repositories providing the latest version of syslog-ng. This blog provides instructions for RPM distributions, like RHEL or SUSE: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Creating heat maps using syslog-ng and Kibana --------------------------------------------- Using the key=value parser of syslog-ng you can easily extract IP addresses from different firewall log messages. The GeoIP parser can add the geolocation belonging to the IP address and Kibana can display the results on a map. Learn how to do this from https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Getting started with Elasticsearch 5 ------------------------------------ For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. As Elastic Stack 5.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0 on RHEL/CentOS 7: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… Fedora and openSUSE now feature syslog-ng 3.8.1 ----------------------------------------------- The second half of November brought us two exciting new Linux distribution releases: openSUSE Leap 42.2 and Fedora 25. Both of them are based on the RPM packaging format and cover everything from embedded through desktops to servers. While there are considerable differences: both feature the latest syslog-ng release, version 3.8.1: https://www.balabit.com/blog/fedora-opensuse-what-is-common-in-the-latest-r… NEW RELEASES - syslog-ng PE 7.0: https://www.balabit.com/blog/syslog-ng-premium-edition-7-released/ - SSB 4.6: https://www.balabit.com/blog/syslog-ng-store-box-goes-to-the-cloud/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Syslog-ng Crashes on CentOS Linux release 7.2.1511 (Core) - Segfaults
by Traiano Welcome 13 Dec '16

13 Dec '16

Hi List syslog-ng 3.5.6 fails to start up on a newly installed CentOS 7 server: Environment details: OS: 3.10.0-327.10.1.el7.x86_64 Distro: CentOS Linux release 7.2.1511 (Core) Syslog-ng: syslog-ng-3.5.6-3.el7.x86_64 I'm doing a fresh install on CentOS linux and installing syslog ng from epel-release repo. syslog-ng fails to start up via systemd as follows: systemctl restart syslog-ng Job for syslog-ng.service failed because a fatal signal was delivered to the control process. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. Looking at journalctl -xe output: - Unit syslog-ng.service has failed. -- -- The result is failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Unit syslog-ng.service entered failed state. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service failed. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Dec 12 09:40:40 ip-172-31-23-167 kernel: syslog-ng[9920]: segfault at 0 ip (null) sp 00007fff5bbba5b8 error 14 in syslog-ng[400000+2000] Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: syslog-ng.service: main process exited, code=killed, status=11/SEGV Dec 12 09:40:40 ip-172-31-23-167 systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. Would anyone know how to fix this, or if this is a bug? Thanks in advance, Traiano

2 1

why and how to install the latest syslog-ng RPMs
by Czanik, Péter 13 Dec '16

13 Dec '16

Hi, There are two returning questions about syslog-ng RPMs: - where to find the latest packages? - why to use unofficial packages if syslog-ng is available in distros? I combined the answers into a single blog: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

character handling in kv-parser
by David Boles (dboles) 13 Dec '16

13 Dec '16

I am trying to process structured data coming from the Linux kernel's printk_emit() function which shows up on /dev/kmesg. Since printk_emit() forces you to flatten hierarchical data into key/value pairs I use a character delimiter between pieces of the key. So, with the '!' character as a separator something that looks like { "cat": { "family" : "mammal" } } would become "cat!family=mammal". When I use a kv-parser, and I use characters like '.', '$', '!', '+' or '^' as a separator the parser seems to just discard everything in the key except the last part. Alphabetic characters and '_' appear to work - but they are the most likely to conflict with other strings. Is there some way to tell the kv-parser to only consider '=' as special (or whatever the separator was defined to be) and keep my keys intact? Thanks, - db

4 8

TimeStamps
by David Campeau 08 Dec '16

08 Dec '16

Hello, Using a syslog-ng sever to filter syslogs before forwarding. I'm being asked to not change the timestamp in the syslog message. I've tried the "keep-timestamp(yes);" option in the syslog-ng.conf, but there's no change in the timestamp. There must be an option I'm missing? Example of the change: Dec 07 15:08:57 <<<< Not filtered by syslog-ng 1 2016-12-07T15:07:32-06:00 <<< Filtering currently with syslog-ng Thank you for looking Best Regards, David

2 2

elasticsearch 5 on rhe 6.4
by Russell Fulton 08 Dec '16

08 Dec '16

Hi I had syslog-ng 3.8.1 (built from tar ball) working nicely with ES 2.x running on the local machine as a pilot. Last week I got two new machines to run ES on with lots of memory and SSD. I installed ES 5.x on them and upgraded ES to 5.x on the original pilot machine. So I now have a cluster of 3. Back to the original machine that has the feed from syslog-ng I was puzzled by the stream of error messages regarding 2.x client that was trying to connect. It took me a while to figure out that this must be the syslog-ng plug in. Stopping syslog-ng stopped the errors. A web search quickly found: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… So I set about reinstalling syslog-ng from Peter’s repos — there were some issues but I got there without too much problem. (see notes at the end). My config file matched the blog example pretty closely - I need to add the http transport for ES 5.x but that was the only change, When I tried to start the new version of syslog-ng I get an error saying that it can not find the elasticsearch2 plugin. I can’t find any reference to insgtalling the plugin in the post. Did I miss something. My old config also had @module mod-java line but this does not appear to make any difference. I have gone back to the 3.8 manual but can not find anything about having to install the ES plugin so I am thoroughly puzzled. Any ideas what is wrong? Russell Problems with the syslog-ng epel 6 repos: since I was on RHE 6 I figured I needed the epel6 rather than the epel7 repo so I wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-6/czan… but when I tried to install Yum said https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-6-x…: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found” After a little poking I figured that the url should be /syslog-ng38eple6/ and that worked.

2 1

how to stop the processing if the filter return true?
by Jorge Pereira 07 Dec '16

07 Dec '16

Hi folks, Basically, in my setup, I received a jSON and the all usual (default) syslog packets in my server. But, I have a simple filter "f_blacklist_network_by_clientip" based in a jSON field. everything works well, the problem is because when my filter "f_blacklist_network_by_clientip" return OK, the junction() goes to the next statement processing the d_default_handler() destination. My doubt is: how to stop the processing when my filter f_blacklist_network_by_clientip() returns OK? *<SNIP>* log { source(s_internet); junction { channel { filter(f_wb_access_log); parser(p_msg2json); filter(f_blacklist_network_by_clientip); destination(d_wb_access_log); flags(final); }; # Default destination channel { * # DOUBT: how to don't processing if the f_blacklist_network_by_clientip return OK?* destination(d_default_handler); }; }; }; filter f_wb_access_log { program("wb_access"); }; parser p_msg2json { json-parser( marker("") prefix("j.")); }; filter f_blacklist_network_by_clientip { not match("^127\.0\.", value("j.clientip")); not match("^172\.16\.", value("j.clientip")); not match("^172\.26\.", value("j.clientip")); not match("^172\.31\.", value("j.clientip")); * # if match, stop the processing and don't jump to "channel { destination(d_default_handler); };* }; destination d_wb_access_log { file("/var/log/syslog-ng/wb/${j.webapp_domain:-invalid_gw_access}_access.log" create_dirs(yes) template("${MSG}\n") ); }; destination d_default_handler { file("/var/log/syslog-ng/servers/${HOST}/${FACILITY:-invalid_facility}.log" create_dirs(yes) ); }; *</SNIP>*

2 1

heat map from firewall logs: syslog-ng GeoIP + Kibana
by Czanik, Péter 28 Nov '16

28 Nov '16

Hi, This was part of my talk last week at a conference, but I also made a blog from it: https://www.balabit.com/blog/how-to-create-heat-maps-to-show-whos-trying-to… You can parse iptables logs (or other firewalls using key=value format) using in syslog-ng using the key=value and GeoIP parsers, store it to Elasticsearch and display the results in Kibana. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Elasticsearch 5 and syslog-ng
by Czanik, Péter 28 Nov '16

28 Nov '16

Hi, For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch <https://www.balabit.com/blog/logging-to-elasticsearch-made-simple-with-sysl…> . As Elastic Stack 5.0.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0.0 on RHEL/CentOS 7: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

5 8

Syslog destination time/folder issues
by Marco Mignone 24 Nov '16

24 Nov '16

Hi All, I am experiencing a weird problem with Syslog-NG 3.8.1 on Ubuntu 14.04 When syslog receives syslog messages from couple of specific nodes it saves it on a destination folder as per the config below: source s_rohnet { network( transport("udp") ); }; destination d_rohnet_switches { file("/var/log/ROHNetwork/${YEAR}.${WEEK}/${HOST}.log" create-dirs(yes) dir-owner("rohadmin")); }; The devices are NTP synchronised and the date output is correct on the Ubuntu server: >date Tue Nov 22 11:21:14 GMT 2016 Beside these the log folders created where the files gets stored are: /2015.51/192.168.33.8.log (it should be /2016.47/). This is happening only for two nodes while all the rest seems to work fine. I have captured some network traffic and the message received by syslog-ng on the network card seems also correct as per Wireshark output: Syslog message: LOCAL6.NOTICE: NOV 22 10:31:23 192.168.33.8-1 CMDLOGGER[165319912]: cmd_logger_api.c(83) 13518 %% CLI:192.168.32.100:root:User logged in This is a Dell switch and I am opening a case with them but I would like to know where else I should check for configuration errors. Syslog config is exactly the one reported above. Any idea of what I could check for further troubleshooting on the Syslog side? Thanks, Marco

3 3

Re: [syslog-ng] Syslog destination time/folder issues
by Scheidler, Balázs 23 Nov '16

23 Nov '16

Yes, an incorrectly parsed month is considered -1, thus it underflows back into the last year. Wrt testing, if you can just deploy syslog-ng, that'd be great. On Nov 23, 2016 08:54, "Marco Mignone" <info(a)marcomignone.com> wrote: Hi Bazsi, Thanks for the info. Do you think that can affect also the ‘year’ behaviour? I would love to provide some help testing this... but I am not good at it :( Thanks for the explanation and for pointing at the mod. Regards, Marco On 22 Nov 2016, at 23:41, Balazs Scheidler <bazsi77(a)gmail.com> wrote: The issue is that syslog-ng only processes mixed case month names, e.g. "Nov" instead of "NOV" This pull request contains the as-of-now unmerged fix: https://github.com/balabit/syslog-ng/pull/1263 Any testing is absolutely welcome. On Tue, Nov 22, 2016 at 12:30 PM, Marco Mignone <info(a)marcomignone.com> wrote: > Hi All, > I am experiencing a weird problem with Syslog-NG 3.8.1 on Ubuntu 14.04 > > When syslog receives syslog messages from couple of specific nodes it > saves it on a destination folder as per the config below: > > source s_rohnet { > network( > transport("udp") > ); > }; > > destination d_rohnet_switches { > file("/var/log/ROHNetwork/${YEAR}.${WEEK}/${HOST}.log" create-dirs(yes) > dir-owner("rohadmin")); > }; > > > The devices are NTP synchronised and the date output is correct on the > Ubuntu server: > > >date > > Tue Nov 22 11:21:14 GMT 2016 > > Beside these the log folders created where the files gets stored are: > */2015.51*/192.168.33.8.log (it should be /*2016.47*/). > > This is happening only for two nodes while all the rest seems to work fine. > > I have captured some network traffic and the message received by syslog-ng > on the network card seems also correct as per Wireshark output: > > Syslog message: LOCAL6.NOTICE: NOV 22 10:31:23 192.168.33.8-1 > CMDLOGGER[165319912]: cmd_logger_api.c(83) 13518 %% CLI:192.168.32.100:root:User > logged in > > This is a Dell switch and I am opening a case with them but I would like > to know where else I should check for configuration errors. > > Syslog config is exactly the one reported above. > > Any idea of what I could check for further troubleshooting on the Syslog > side? > > Thanks, > Marco > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product= > syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > -- Bazsi ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Debian Wheezy, Ubuntu Precise support
by Laszlo Budai 23 Nov '16

23 Nov '16

Hi, I've built syslog-ng 3.8.1 on Debian Wheezy and Ubuntu Precise in my OBS repo (some of you requested for Ubuntu Precise support, so here it is). so from now my syslog-ng packages are available on the following distribs: * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 * Ubuntu 16.10 https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

3 2