January 2018 - syslog-ng - lists.balabit.hu

format-json incorrectly parsing some events
by Alicia Smith 19 Jan '18

19 Jan '18

I'm using syslog-ng rpm version 3.12.1-2 on CentOS 7 When we receive events remotely from another CentOS 7 host it uses the RFC5424 format and parses the messages correctly. However we have some hosts that are older and still using rsyslog which is using the RFC3164 format - those events do not parse correctly. My question is what is the best way to get syslog-ng to parse them? This is how they come out: {"TAGS":".source.test","SOURCEIP":"127.0.0.1","SOURCE":"test","SEQNUM":"26","PROGRAM":"info","PRIORITY":"notice","MESSAGE":" mig-agent 10430 - - - [info] refreshing agent environment","LEGACY_MSGHDR":"info ","HOST_FROM":" syslog-dev1.private.mdc1.mozilla.com","HOST":"sanvmadm1.ops.mdc1.mozilla.com","FILE_NAME":"/var/log/test.log","FACILITY":"user","DATE":"Jan 17 23:57:52","CATEGORY":"syslog"] Notice the Program says "info" and the mig-agent and pid are in the message key's value. This is a correctly parsed event that has those fields parsed properly: {"TAGS":".source.moz_net","SOURCEIP":"127.0.0.1","SOURCE":"moz_net","SEQNUM":"35","PROGRAM":"mig-agent","PRIORITY":"info","PID":"2698","MESSAGE":"- - - [info] Public IP retrieval failed through proxy http://proxy.dmz.scl3.mozilla.com:3128 - Get https://api.mig.mozilla.org/api/v1//ip: proxyconnect tcp: dial tcp 10.22.74.78:3128: i/o timeout","LEGACY_MSGHDR":"mig-agent[2698]: ","HOST_FROM":"localhost6.localdomain","HOST":" syslog-dev1.private.mdc1.mozilla.com","FACILITY":"daemon","DATE":"Jan 18 00:02:25","CATEGORY":"syslog"} destination d_amqp { amqp( vhost("/") host("localhost") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope selected_macros --scope nv_pairs)") persistent(no) username("rabbituser") password("*****") ); }; -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

2 1

Balabit acquired by One Identity
by Scheidler, Balázs 18 Jan '18

18 Jan '18

Dear syslog-ng users, You might have already heard that Balabit, the company behind syslog-ng has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. I'd like to share a bit of personal details: I've founded the syslog-ng project back in 1998, during my university years, and a bit later on, I was also one of the founders of Balabit. I've worked on syslog-ng as a personal hobby until about 2007, at which point Balabit has officially started sponsoring the project, building products on top and also providing professional services. Although I was CEO and later CTO of Balabit, I always remained an active member of the syslog-ng community. Whenever I've found some leisure time (e.g. on a plane :), I was working on syslog-ng, and I've kept doing that parallel to Balabit establishing a professional dev team that worked on syslog-ng and related products. All of this work went straight to the open source project: https://github.com/balabit/syslog-ng/graphs/contributors With that story, I’d like to assure you that this acquisition will not negatively affect syslog-ng. One Identity considers the log management products of Balabit an important part of the acquisition and will continue to develop both the commercial and the open source branches further. Of course, I also have a role in the combined entity, and will continue to keep working on syslog-ng personally. I seem to be expected to be sitting on planes much more often :) One Identity as a business will also keep working on syslog-ng, and are committed to keeping it open source, adding new features, and so on, just like we did in the past. As a first step of this process, the syslog-ng product line will receive its own homepage in the coming weeks. Stay tuned, and keep using syslog-ng -- Bazsi

1 0

RHEL 6 / CentOS 6 packages
by Czanik, Péter 18 Jan '18

18 Jan '18

Hi, There were many requests for RHEL 6 / CentOS 6 support. The last version compiling on these ancient platforms was syslog-ng 3.9. Bazsi provided a fix: https://github.com/balabit/syslog-ng/pull/1837 which I applied to the syslog-ng 3.13 branch. Packages are available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313epel6/ Compared to the 3.9 package other than making it to compile I enabled GeoIP2 (libmaxminddb) support and fixed any problems in packaging I ran into. I consider these packages ready to use, but if you run into any problems let me know! Please RT https://twitter.com/PCzanik/status/953931953669050368 to reach more people$ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 3

match filter scale issue
by Kavita Salve 14 Jan '18

14 Jan '18

Hello, I have an issue related to the syslog-ng “match” filter on which I need some inputs here. I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog. E.g. · match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header · match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter) The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros. We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. Following are the scale test results we are seeing for various combinations of message and match filters: 1. When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around *6667 eps.* 2. When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around *1150-1200.* 3. When I used message filter, it resulted in eps of processing syslogs as *4730 eps* 4. When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of *~1080eps* *My analysis and questions:* 1. I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. *How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?* 2. I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this*. Is there any other way/filter to use here?* *3. *I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit*. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?* *PS:* I am filtering syslogs generated by Cisco IOS devices that are in the format: *%FACILITY-SEVERITY-MNEMONIC: Message-text* E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up For these kind of syslogs, we are seeing that syslog-ng treats *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “*Message-text*” as the *Message Body*. Thanks Kavita

2 2

Anyone sourcing from beats
by Evan Rempel 11 Jan '18

11 Jan '18

Looking for a clean way to get beats products to send data to syslog-ng. Does anyone have a working example? -- Evan

4 4

SYSLOG-NG issue with ES 6.X
by hari ram 10 Jan '18

10 Jan '18

Hi I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs. root@ES6:/etc/syslog-ng# syslog-ng -V syslog-ng 3 (3.13.2) Config version: 3.13 Installer-Version: 3.13.2 Revision: 3.13.2-1 Compile-Date: Dec 5 2017 13:24:07 Module-Directory: /usr/lib/syslog-ng/3.13 Module-Path: /usr/lib/syslog-ng/3.13 Available-Modules: afuser,mod-python,afstomp,http,afsql,disk-buffer,mod-java,cef,pseudofile,sdjournal,kvformat,xml,csvparser,snmptrapd-parser,appmodel,confgen,pacctformat,linux-kmsg-format,dbparser,system-source,map-value-pairs,add-contextual-data,date,syslogformat,afamqp,geoip2-plugin,tfgetent,graphite,afmongodb,cryptofuncs,geoip-plugin,afsmtp,afsocket,redis,affile,stardate,basicfuncs,riemann,json-plugin,tags-parser,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on === root@ES6:/etc/syslog-ng# more syslog-ng.conf @version:3.13 @module mod-java @include "scl.conf" options { flush_lines(0); keep_hostname(yes); normalize_hostnames(yes); threaded(yes); }; source s_local { system(); internal(); }; source s_network { syslog(transport(tcp)); }; destination d_all { file ("/var/log/all.log"); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") client_mode("http") cluster_url("http://192.168.1.75:9200") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") cluster("test") flush-limit("1000") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; log { source(s_network); destination(d_elastic); }; log { source(s_local); destination(d_all); }; === root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/ elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar hppc-0.7.1.jar java-version-checker-6.0.1.jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.jar spatial4j-0.6.jar jackson-dataformat-cbor-2.8.6.jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar ==== root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-modules/ elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar == root@ES6:/etc/syslog-ng# syslog-ng -Fevd [2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12] [2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:35] [2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.conf:27:24] [2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:20] [2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:12:21] [2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat' [2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' [2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/; [2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-common.jar; [2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/hdfs.jar; [2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/http.jar; [2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/kafka.jar; [2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/log4j-1.2.16.jar; [2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic-v2.jar; [2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic.jar; [2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Any suggestions ? R!

3 6

3.13 maintenance branch
by Scheidler, Balázs 09 Jan '18

09 Jan '18

Hi, I have just pushed a branch containing a number of bugfixes for the 3.13 branch, that I felt would be useful to be published in various syslog-ng binary packages over the existing 3.13.2 release that we produced a few weeks ago. https://github.com/balabit/syslog-ng/tree/3.13/master The most important change is this: https://github.com/balabit/syslog-ng/commit/7e96705ec0b3570857ed6cd66913999… Which is bug that causes the elasticsearch2 destination SCL wrapper to fails to start up if any __VARARGS__ is used. (see https://github.com/balabit/syslog-ng/issues/1804) The other patches are just my selection of fixes that were deemed low risk to me. We expect the next official syslog-ng release to be 3.14 at the end of February, and I've felt this set of fixes would be useful even in the meantime. At this point, I take this as an experiment: if publishing fixes this way "works" (e.g. fixes get published in the OS packages that we don't maintain ourselves), then we can get some form of maintenance to older open source releases as time permits, with collaboration from our fellow OS maintainers. Also, power users, who are able to compile source code for themselves get help too. Let me know if this is useful. Cheers, -- Bazsi

2 2

Does anyone use kafka as a source
by Evan Rempel 08 Jan '18

08 Jan '18

I would like to source log messages from kafka with syslog-ng so that I can leverage some message parsing components. I may place the post parsed messages back into a kafka pipe but I am having problems determining a good way to get messages from kafka to syslog-ng. Any recommendations? -- Evan

3 2

syslog-ng inside LXC guest receives kernel messages from host
by webman＠manfbraun.de 03 Jan '18

03 Jan '18

Hello! It's the first time, that I use syslog-ng (although the plan ist old - due to the ability to use rabbitmq ...). The host (which is a VM too - do not know exactly which type) has the normal rsyslog installed (was "shipped" with it and not directly of my interest - so I kept it). What I am getting from the host are kernel messages generated from iptables logging - I know the log prefix. The guest has just now no iptables rules at all, but a running ulog2, which (no iptables rules at the moment) just runs, but has nothing to log and messages continue to arrive, after I've stopped it. I had a reboot in between, just to be sure, iptables has not something in its memory. There is a bridge to the host and the outside. While the iptables rules were active, I blocked port 514, but this does not change anything. As told, the messages now continue, even iptables has no active rules. A tcpdump inside the lxc guest does not show packages on port 514. BTW, the messages are logged with the hostname of the guest. syslog-ng uses the standards for it input (system, internal). Probably someone could shed some light on it. It is nothing more worrying, then messages from unknown source! Thanks anyway and best regards, Manfred

2 1

Syslog-ng versions available with RHEL 6
by Carter, Jed 02 Jan '18

02 Jan '18

Good morning, I am currently running syslog-ng OSE version 3.7 and RHEL 6. I'm trying to update my syslog-ng OSE through EPEL. It looks like the only way to get syslog-ng v3.12 is from RHEL 7. What is the latest version of syslog-ng OSE that is supported by RHEL 6? Thank you, Jed -- Jed Carter GCIH, GSEC, CEH Information Security Engineer Partners Health Care The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

5 5