- syslog-ng - lists.balabit.hu

RHEL 6 / CentOS 6 packages
by Czanik, Péter 18 Jan '18

18 Jan '18

Hi, There were many requests for RHEL 6 / CentOS 6 support. The last version compiling on these ancient platforms was syslog-ng 3.9. Bazsi provided a fix: https://github.com/balabit/syslog-ng/pull/1837 which I applied to the syslog-ng 3.13 branch. Packages are available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313epel6/ Compared to the 3.9 package other than making it to compile I enabled GeoIP2 (libmaxminddb) support and fixed any problems in packaging I ran into. I consider these packages ready to use, but if you run into any problems let me know! Please RT https://twitter.com/PCzanik/status/953931953669050368 to reach more people$ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 3

match filter scale issue
by Kavita Salve 14 Jan '18

14 Jan '18

Hello, I have an issue related to the syslog-ng “match” filter on which I need some inputs here. I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog. E.g. · match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header · match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter) The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros. We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. Following are the scale test results we are seeing for various combinations of message and match filters: 1. When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around *6667 eps.* 2. When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around *1150-1200.* 3. When I used message filter, it resulted in eps of processing syslogs as *4730 eps* 4. When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of *~1080eps* *My analysis and questions:* 1. I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. *How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?* 2. I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this*. Is there any other way/filter to use here?* *3. *I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit*. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?* *PS:* I am filtering syslogs generated by Cisco IOS devices that are in the format: *%FACILITY-SEVERITY-MNEMONIC: Message-text* E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up For these kind of syslogs, we are seeing that syslog-ng treats *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “*Message-text*” as the *Message Body*. Thanks Kavita

2 2

Anyone sourcing from beats
by Evan Rempel 11 Jan '18

11 Jan '18

Looking for a clean way to get beats products to send data to syslog-ng. Does anyone have a working example? -- Evan

4 4

SYSLOG-NG issue with ES 6.X
by hari ram 10 Jan '18

10 Jan '18

Hi I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs. root@ES6:/etc/syslog-ng# syslog-ng -V syslog-ng 3 (3.13.2) Config version: 3.13 Installer-Version: 3.13.2 Revision: 3.13.2-1 Compile-Date: Dec 5 2017 13:24:07 Module-Directory: /usr/lib/syslog-ng/3.13 Module-Path: /usr/lib/syslog-ng/3.13 Available-Modules: afuser,mod-python,afstomp,http,afsql,disk-buffer,mod-java,cef,pseudofile,sdjournal,kvformat,xml,csvparser,snmptrapd-parser,appmodel,confgen,pacctformat,linux-kmsg-format,dbparser,system-source,map-value-pairs,add-contextual-data,date,syslogformat,afamqp,geoip2-plugin,tfgetent,graphite,afmongodb,cryptofuncs,geoip-plugin,afsmtp,afsocket,redis,affile,stardate,basicfuncs,riemann,json-plugin,tags-parser,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on === root@ES6:/etc/syslog-ng# more syslog-ng.conf @version:3.13 @module mod-java @include "scl.conf" options { flush_lines(0); keep_hostname(yes); normalize_hostnames(yes); threaded(yes); }; source s_local { system(); internal(); }; source s_network { syslog(transport(tcp)); }; destination d_all { file ("/var/log/all.log"); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") client_mode("http") cluster_url("http://192.168.1.75:9200") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") cluster("test") flush-limit("1000") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; log { source(s_network); destination(d_elastic); }; log { source(s_local); destination(d_all); }; === root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/ elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar hppc-0.7.1.jar java-version-checker-6.0.1.jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.jar spatial4j-0.6.jar jackson-dataformat-cbor-2.8.6.jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar ==== root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-modules/ elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar == root@ES6:/etc/syslog-ng# syslog-ng -Fevd [2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12] [2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:35] [2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.conf:27:24] [2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:20] [2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:12:21] [2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat' [2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' [2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/; [2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-common.jar; [2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/hdfs.jar; [2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/http.jar; [2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/kafka.jar; [2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/log4j-1.2.16.jar; [2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic-v2.jar; [2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic.jar; [2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Any suggestions ? R!

3 6

3.13 maintenance branch
by Scheidler, Balázs 09 Jan '18

09 Jan '18

Hi, I have just pushed a branch containing a number of bugfixes for the 3.13 branch, that I felt would be useful to be published in various syslog-ng binary packages over the existing 3.13.2 release that we produced a few weeks ago. https://github.com/balabit/syslog-ng/tree/3.13/master The most important change is this: https://github.com/balabit/syslog-ng/commit/7e96705ec0b3570857ed6cd66913999… Which is bug that causes the elasticsearch2 destination SCL wrapper to fails to start up if any __VARARGS__ is used. (see https://github.com/balabit/syslog-ng/issues/1804) The other patches are just my selection of fixes that were deemed low risk to me. We expect the next official syslog-ng release to be 3.14 at the end of February, and I've felt this set of fixes would be useful even in the meantime. At this point, I take this as an experiment: if publishing fixes this way "works" (e.g. fixes get published in the OS packages that we don't maintain ourselves), then we can get some form of maintenance to older open source releases as time permits, with collaboration from our fellow OS maintainers. Also, power users, who are able to compile source code for themselves get help too. Let me know if this is useful. Cheers, -- Bazsi

2 2

Does anyone use kafka as a source
by Evan Rempel 08 Jan '18

08 Jan '18

I would like to source log messages from kafka with syslog-ng so that I can leverage some message parsing components. I may place the post parsed messages back into a kafka pipe but I am having problems determining a good way to get messages from kafka to syslog-ng. Any recommendations? -- Evan

3 2

syslog-ng inside LXC guest receives kernel messages from host
by webman＠manfbraun.de 03 Jan '18

03 Jan '18

Hello! It's the first time, that I use syslog-ng (although the plan ist old - due to the ability to use rabbitmq ...). The host (which is a VM too - do not know exactly which type) has the normal rsyslog installed (was "shipped" with it and not directly of my interest - so I kept it). What I am getting from the host are kernel messages generated from iptables logging - I know the log prefix. The guest has just now no iptables rules at all, but a running ulog2, which (no iptables rules at the moment) just runs, but has nothing to log and messages continue to arrive, after I've stopped it. I had a reboot in between, just to be sure, iptables has not something in its memory. There is a bridge to the host and the outside. While the iptables rules were active, I blocked port 514, but this does not change anything. As told, the messages now continue, even iptables has no active rules. A tcpdump inside the lxc guest does not show packages on port 514. BTW, the messages are logged with the hostname of the guest. syslog-ng uses the standards for it input (system, internal). Probably someone could shed some light on it. It is nothing more worrying, then messages from unknown source! Thanks anyway and best regards, Manfred

2 1

Syslog-ng versions available with RHEL 6
by Carter, Jed 02 Jan '18

02 Jan '18

Good morning, I am currently running syslog-ng OSE version 3.7 and RHEL 6. I'm trying to update my syslog-ng OSE through EPEL. It looks like the only way to get syslog-ng v3.12 is from RHEL 7. What is the latest version of syslog-ng OSE that is supported by RHEL 6? Thank you, Jed -- Jed Carter GCIH, GSEC, CEH Information Security Engineer Partners Health Care The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

5 5

Problem using Python Parser
by Ronald Fenner 30 Dec '17

30 Dec '17

When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20' I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported. Ronald Fenner Programmer Game Circus LLC. rfenner(a)gamecircus.com

3 4

Host name issue while logging
by hari ram 16 Dec '17

16 Dec '17

Hi It was running ok till yesterday all the logs are stored with Hostname and date format after upgrade the hostname not able to log in the log format, what is the fix ? Looks like after upgrade to new 3.13 my logs no longer able to use hostname it is storing like this :-2017-12-11.log before INET-2017-12-11.log ( INET Missing why ) any changes ? in the new code 3.13 ? R!

2 2