- syslog-ng - lists.balabit.hu

Excessive memory usage
by Patrick Hemmer 26 Jan '18

26 Jan '18

We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts. I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers? Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/) Config: @version:3.13 @include "scl.conf" # This configuration is meant to utilize systemd journald options { flush_lines(0); time_reopen(10); log-msg-size(65536); log_fifo_size(10000); chain_hostnames(off); use_dns(no); use_fqdn(yes); keep_hostname(yes); create_dirs(no); stats_freq(3600); }; source s_system { system(); }; source s_internal { internal(); }; destination d_journald { unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n")); }; log { source(s_internal); destination(d_journald); }; destination d_loghost { syslog("syslog.ecom.chewy.com" frac-digits(3) log-fifo-size(10000) ); }; log { source(s_system); destination(d_loghost); }; Stats: # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216 destination;d_journald;;a;processed;1997 src.internal;s_internal#0;;a;processed;1997 src.internal;s_internal#0;;a;stamp;1515709115 center;;received;a;processed;1997 destination;d_loghost;;a;processed;489030242 source;s_internal;;a;processed;1997 center;;queued;a;processed;489032239 global;scratch_buffers_count;;a;queued;25769803782 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845 global;payload_reallocs;;a;processed;2932894697 src.journald;;journal;a;processed;489030242 src.journald;;journal;a;stamp;1515710081 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;135 global;internal_queue_length;;a;processed;0 Thanks -Patrick

2 3

SUMO Logic Integration
by Scot 25 Jan '18

25 Jan '18

Hi all, Thought I had asked already but I don't see it, so sorry if it's a repost. I now have the need to switch all my syslog-ng -> SPLUNK feeds over to SUMO. Including json/beats going through syslog-ng :( Sumo supports a syslog tcp/udp collection, I setup an "Installed Collector" local to the environment. Seems to work fine until you you need to match categories in sumo. The sumo config seems to be pretty limited per instance so looking at spinning up multiple instances of sumo just to support Category mapping. Not happy with the product. Anyone working with syslog-ng and SUMO who would like to collaborate? Scot

1 0

pulling in "sar" data
by Jim Hendrick 24 Jan '18

24 Jan '18

Hi list, Has anyone added sar data to syslog-ng? I know of a couple "brute-force" ways like running a cron to run the sar command and dump it to a file. But before I do that I wanted to check and see if there was a more elegant solution. The overall problem would be to run this across 10,000+ servers with minimal need to change configurations. (for example - if there was a need to change the parameters to sar, or change the frequency of the pull). I guess I was hoping for some add-on that either directly pulled the performance data from the kernel or could include a config that would specify what data elements would be included. FYI - we do use syslog-ng PE but this seems fairly generic so I thought I'd ask the list. Given the scale of the problem, pulling "all" data even relatively frequently (say every 1-5 minutes) would result in a huge volume increase in our logging solution (where we pay by the ingested GB...) Thoughts? Advice? Thanks in advance! Jim

2 1

new build of syslog-ng 3.13.2 has new dependencies
by Evan Rempel 23 Jan '18

23 Jan '18

The copr repo at https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng313/epel-7-… has a new build of syslog-ng. The actual version of syslog-ng has not changed, just the build. The package has gone from version 3.13.2-1 to 3.13.2-2 and new dependencies have been added. Specifically the 4 below --> Processing Dependency: libmongoc-1.0.so.0(LIBMONGOC_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0(LIBBSON_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libmongoc-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 Since the version of syslog-ng has not changed, how come the dependencies have changed? Is this a mistake and these dependencies are not actually needed? I ask because I am running the package 3.13.2-1 and not having any difficulties (I am not using the mondo module). If the dependencies are not required, can they be removed and a new build created? -- Evan Rempel

2 3

format-json incorrectly parsing some events
by Alicia Smith 19 Jan '18

19 Jan '18

I'm using syslog-ng rpm version 3.12.1-2 on CentOS 7 When we receive events remotely from another CentOS 7 host it uses the RFC5424 format and parses the messages correctly. However we have some hosts that are older and still using rsyslog which is using the RFC3164 format - those events do not parse correctly. My question is what is the best way to get syslog-ng to parse them? This is how they come out: {"TAGS":".source.test","SOURCEIP":"127.0.0.1","SOURCE":"test","SEQNUM":"26","PROGRAM":"info","PRIORITY":"notice","MESSAGE":" mig-agent 10430 - - - [info] refreshing agent environment","LEGACY_MSGHDR":"info ","HOST_FROM":" syslog-dev1.private.mdc1.mozilla.com","HOST":"sanvmadm1.ops.mdc1.mozilla.com","FILE_NAME":"/var/log/test.log","FACILITY":"user","DATE":"Jan 17 23:57:52","CATEGORY":"syslog"] Notice the Program says "info" and the mig-agent and pid are in the message key's value. This is a correctly parsed event that has those fields parsed properly: {"TAGS":".source.moz_net","SOURCEIP":"127.0.0.1","SOURCE":"moz_net","SEQNUM":"35","PROGRAM":"mig-agent","PRIORITY":"info","PID":"2698","MESSAGE":"- - - [info] Public IP retrieval failed through proxy http://proxy.dmz.scl3.mozilla.com:3128 - Get https://api.mig.mozilla.org/api/v1//ip: proxyconnect tcp: dial tcp 10.22.74.78:3128: i/o timeout","LEGACY_MSGHDR":"mig-agent[2698]: ","HOST_FROM":"localhost6.localdomain","HOST":" syslog-dev1.private.mdc1.mozilla.com","FACILITY":"daemon","DATE":"Jan 18 00:02:25","CATEGORY":"syslog"} destination d_amqp { amqp( vhost("/") host("localhost") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope selected_macros --scope nv_pairs)") persistent(no) username("rabbituser") password("*****") ); }; -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

2 1

Balabit acquired by One Identity
by Scheidler, Balázs 18 Jan '18

18 Jan '18

Dear syslog-ng users, You might have already heard that Balabit, the company behind syslog-ng has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. I'd like to share a bit of personal details: I've founded the syslog-ng project back in 1998, during my university years, and a bit later on, I was also one of the founders of Balabit. I've worked on syslog-ng as a personal hobby until about 2007, at which point Balabit has officially started sponsoring the project, building products on top and also providing professional services. Although I was CEO and later CTO of Balabit, I always remained an active member of the syslog-ng community. Whenever I've found some leisure time (e.g. on a plane :), I was working on syslog-ng, and I've kept doing that parallel to Balabit establishing a professional dev team that worked on syslog-ng and related products. All of this work went straight to the open source project: https://github.com/balabit/syslog-ng/graphs/contributors With that story, I’d like to assure you that this acquisition will not negatively affect syslog-ng. One Identity considers the log management products of Balabit an important part of the acquisition and will continue to develop both the commercial and the open source branches further. Of course, I also have a role in the combined entity, and will continue to keep working on syslog-ng personally. I seem to be expected to be sitting on planes much more often :) One Identity as a business will also keep working on syslog-ng, and are committed to keeping it open source, adding new features, and so on, just like we did in the past. As a first step of this process, the syslog-ng product line will receive its own homepage in the coming weeks. Stay tuned, and keep using syslog-ng -- Bazsi

1 0

RHEL 6 / CentOS 6 packages
by Czanik, Péter 18 Jan '18

18 Jan '18

Hi, There were many requests for RHEL 6 / CentOS 6 support. The last version compiling on these ancient platforms was syslog-ng 3.9. Bazsi provided a fix: https://github.com/balabit/syslog-ng/pull/1837 which I applied to the syslog-ng 3.13 branch. Packages are available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313epel6/ Compared to the 3.9 package other than making it to compile I enabled GeoIP2 (libmaxminddb) support and fixed any problems in packaging I ran into. I consider these packages ready to use, but if you run into any problems let me know! Please RT https://twitter.com/PCzanik/status/953931953669050368 to reach more people$ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 3

match filter scale issue
by Kavita Salve 14 Jan '18

14 Jan '18

Hello, I have an issue related to the syslog-ng “match” filter on which I need some inputs here. I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog. E.g. · match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header · match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter) The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros. We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. Following are the scale test results we are seeing for various combinations of message and match filters: 1. When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around *6667 eps.* 2. When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around *1150-1200.* 3. When I used message filter, it resulted in eps of processing syslogs as *4730 eps* 4. When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of *~1080eps* *My analysis and questions:* 1. I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. *How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?* 2. I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this*. Is there any other way/filter to use here?* *3. *I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit*. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?* *PS:* I am filtering syslogs generated by Cisco IOS devices that are in the format: *%FACILITY-SEVERITY-MNEMONIC: Message-text* E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up For these kind of syslogs, we are seeing that syslog-ng treats *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “*Message-text*” as the *Message Body*. Thanks Kavita

2 2

Anyone sourcing from beats
by Evan Rempel 11 Jan '18

11 Jan '18

Looking for a clean way to get beats products to send data to syslog-ng. Does anyone have a working example? -- Evan

4 4

SYSLOG-NG issue with ES 6.X
by hari ram 10 Jan '18

10 Jan '18

Hi I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs. root@ES6:/etc/syslog-ng# syslog-ng -V syslog-ng 3 (3.13.2) Config version: 3.13 Installer-Version: 3.13.2 Revision: 3.13.2-1 Compile-Date: Dec 5 2017 13:24:07 Module-Directory: /usr/lib/syslog-ng/3.13 Module-Path: /usr/lib/syslog-ng/3.13 Available-Modules: afuser,mod-python,afstomp,http,afsql,disk-buffer,mod-java,cef,pseudofile,sdjournal,kvformat,xml,csvparser,snmptrapd-parser,appmodel,confgen,pacctformat,linux-kmsg-format,dbparser,system-source,map-value-pairs,add-contextual-data,date,syslogformat,afamqp,geoip2-plugin,tfgetent,graphite,afmongodb,cryptofuncs,geoip-plugin,afsmtp,afsocket,redis,affile,stardate,basicfuncs,riemann,json-plugin,tags-parser,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on === root@ES6:/etc/syslog-ng# more syslog-ng.conf @version:3.13 @module mod-java @include "scl.conf" options { flush_lines(0); keep_hostname(yes); normalize_hostnames(yes); threaded(yes); }; source s_local { system(); internal(); }; source s_network { syslog(transport(tcp)); }; destination d_all { file ("/var/log/all.log"); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") client_mode("http") cluster_url("http://192.168.1.75:9200") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") cluster("test") flush-limit("1000") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; log { source(s_network); destination(d_elastic); }; log { source(s_local); destination(d_all); }; === root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/ elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar hppc-0.7.1.jar java-version-checker-6.0.1.jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.jar spatial4j-0.6.jar jackson-dataformat-cbor-2.8.6.jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar ==== root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-modules/ elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar == root@ES6:/etc/syslog-ng# syslog-ng -Fevd [2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12] [2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:35] [2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.conf:27:24] [2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:20] [2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:12:21] [2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat' [2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' [2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/; [2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-common.jar; [2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/hdfs.jar; [2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/http.jar; [2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/kafka.jar; [2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/log4j-1.2.16.jar; [2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic-v2.jar; [2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic.jar; [2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Any suggestions ? R!

3 6