December 2017 - syslog-ng - lists.balabit.hu

pulling in "sar" data
by Jim Hendrick 24 Jan '18

24 Jan '18

Hi list, Has anyone added sar data to syslog-ng? I know of a couple "brute-force" ways like running a cron to run the sar command and dump it to a file. But before I do that I wanted to check and see if there was a more elegant solution. The overall problem would be to run this across 10,000+ servers with minimal need to change configurations. (for example - if there was a need to change the parameters to sar, or change the frequency of the pull). I guess I was hoping for some add-on that either directly pulled the performance data from the kernel or could include a config that would specify what data elements would be included. FYI - we do use syslog-ng PE but this seems fairly generic so I thought I'd ask the list. Given the scale of the problem, pulling "all" data even relatively frequently (say every 1-5 minutes) would result in a huge volume increase in our logging solution (where we pay by the ingested GB...) Thoughts? Advice? Thanks in advance! Jim

2 1

SYSLOG-NG issue with ES 6.X
by hari ram 10 Jan '18

10 Jan '18

Hi I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs. root@ES6:/etc/syslog-ng# syslog-ng -V syslog-ng 3 (3.13.2) Config version: 3.13 Installer-Version: 3.13.2 Revision: 3.13.2-1 Compile-Date: Dec 5 2017 13:24:07 Module-Directory: /usr/lib/syslog-ng/3.13 Module-Path: /usr/lib/syslog-ng/3.13 Available-Modules: afuser,mod-python,afstomp,http,afsql,disk-buffer,mod-java,cef,pseudofile,sdjournal,kvformat,xml,csvparser,snmptrapd-parser,appmodel,confgen,pacctformat,linux-kmsg-format,dbparser,system-source,map-value-pairs,add-contextual-data,date,syslogformat,afamqp,geoip2-plugin,tfgetent,graphite,afmongodb,cryptofuncs,geoip-plugin,afsmtp,afsocket,redis,affile,stardate,basicfuncs,riemann,json-plugin,tags-parser,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on === root@ES6:/etc/syslog-ng# more syslog-ng.conf @version:3.13 @module mod-java @include "scl.conf" options { flush_lines(0); keep_hostname(yes); normalize_hostnames(yes); threaded(yes); }; source s_local { system(); internal(); }; source s_network { syslog(transport(tcp)); }; destination d_all { file ("/var/log/all.log"); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") client_mode("http") cluster_url("http://192.168.1.75:9200") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") cluster("test") flush-limit("1000") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; log { source(s_network); destination(d_elastic); }; log { source(s_local); destination(d_all); }; === root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/ elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar hppc-0.7.1.jar java-version-checker-6.0.1.jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.jar spatial4j-0.6.jar jackson-dataformat-cbor-2.8.6.jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar ==== root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-modules/ elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar == root@ES6:/etc/syslog-ng# syslog-ng -Fevd [2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12] [2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:35] [2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.conf:27:24] [2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:20] [2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:12:21] [2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat' [2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' [2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/; [2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-common.jar; [2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/hdfs.jar; [2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/http.jar; [2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/kafka.jar; [2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/log4j-1.2.16.jar; [2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic-v2.jar; [2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic.jar; [2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Any suggestions ? R!

3 6

Syslog-ng versions available with RHEL 6
by Carter, Jed 02 Jan '18

02 Jan '18

Good morning, I am currently running syslog-ng OSE version 3.7 and RHEL 6. I'm trying to update my syslog-ng OSE through EPEL. It looks like the only way to get syslog-ng v3.12 is from RHEL 7. What is the latest version of syslog-ng OSE that is supported by RHEL 6? Thank you, Jed -- Jed Carter GCIH, GSEC, CEH Information Security Engineer Partners Health Care The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

5 5

Problem using Python Parser
by Ronald Fenner 30 Dec '17

30 Dec '17

When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20' I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported. Ronald Fenner Programmer Game Circus LLC. rfenner(a)gamecircus.com

3 4

Host name issue while logging
by hari ram 16 Dec '17

16 Dec '17

Hi It was running ok till yesterday all the logs are stored with Hostname and date format after upgrade the hostname not able to log in the log format, what is the fix ? Looks like after upgrade to new 3.13 my logs no longer able to use hostname it is storing like this :-2017-12-11.log before INET-2017-12-11.log ( INET Missing why ) any changes ? in the new code 3.13 ? R!

2 2

Insider 2017-12: 3.13; Splunk HEC; Application Adapters; Graylog;
by Czanik, Péter 14 Dec '17

14 Dec '17

Dear syslog-ng users, This is the 64th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.13 released -------------------------- The latest version of syslog-ng, 3.13 is now available. It now parses collected messages automatically using application adapters and can easily forward name-value pairs using the enterprise-wide message model. Support for Graylog and the GELF message format was also added. There are many more smaller features and bug fixes. For a complete list check the release announcements: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.13.1 and https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.13.2 Sending logs to Splunk through HTTP ----------------------------------- For quite some time, Splunk has recommended to collect syslog messages using syslog-ng, save them to files, and send them to Splunk using forwarders. Unless you have a very high message rate, the HTTP destination of syslog-ng can greatly simplify this logging architecture. Instead of writing messages to files and reading them by a forwarder, syslog-ng can forward messages to Spunk HTTP Event Collector (HEC) directly, using HTTP or HTTPS connections. And if you parse messages using syslog-ng, you can send the resulting name-value pairs to Splunk in JSON format and be able to search them instantly. https://www.balabit.com/blog/sending-logs-splunk-http/ Application Adapters & Enterprise-wide Message Model ---------------------------------------------------- Do you want to simplify parsing your log messages? Try the new “application adapter” and “enterprise-wide message model” frameworks in syslog-ng: you can automatically parse log messages and forward the results to another syslog-ng instance. Optionally, you can also include the original, raw message that you can forward unmodified to a SIEM system for further analysis. Learn how to use these new features from https://www.balabit.com/blog/application-adapters-enterprise-wide-message-m… Graylog as destination in syslog-ng ----------------------------------- Version 3.13 of syslog-ng introduced a graylog2() destination and a GELF (Graylog Extended Log Format) template to make sending syslog messages to Graylog easier. You can also use them to forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw tcp port in Graylog, which can then extract fields from nested JSON. https://www.balabit.com/blog/graylog-destination-syslog-ng/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

"Error initializing message pipeline; plugin name='java', location='#buffer:2:3'" after update to syslog-ng 3.12
by Roegels Ruben 14 Dec '17

14 Dec '17

Hi there, I'm using the OSE of syslog-ng for some weeks. Today I updated syslog-ng to 3.13 and afterwards I started seeing -- Unit syslog-ng.service has begun starting up. Dez 13 11:53:59 syslog syslog-ng[23803]: [2017-12-13T11:53:59.572246] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Dez 13 11:53:59 syslog systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Dez 13 11:53:59 syslog systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd-13T11:53:59.572246] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' In "journalctl -xe" If I disable the elasticsearch destination syslog-ng starts up fine. I don't know what to look at as the error message is not very meaningful to me. Did I miss some configuration format changes regarding the "elasticsearch2" destination? The header of my "/etc/syslog-ng/syslog-ng.conf": === Snip === @version: 3.13 @module mod-java @include "scl.conf" === Snip === My elasticsearch config for syslog-ng: https://pastebin.com/g6UB3gBi "ls -l {/usr/share/elasticsearch/lib/,/usr/lib/syslog-ng/3.13/java-modules/}" https://pastebin.com/nqYAVhHL "cat /etc/systemd/system/multi-user.target.wants/syslog-ng.service" https://pastebin.com/UzfQXvZL I really appreciate any help. Thank you! Best regards, Ruben ____________________________________________________________________________ netsys IT GmbH Ruben Roegels Nelly-Sachs-Str. 4 79111 Freiburg Tel. +49 761 888 691 0 Fax +49 761 888 691 10 <http://www.netsys-it.de> Ruben.Roegels(a)netsys-it.de ____________________________________________________________________________ Amtsgericht Freiburg HRB 5752 Geschaeftsfuehrer: Juergen Dufner ____________________________________________________________________________

3 2

SYSLOG-NG config issue and suggestion
by hari ram 08 Dec '17

08 Dec '17

Hi Now my syslog server running. Just try to use in my network and export the logs Logstash or elastic Search Take example I have SWITCH1, SWITCH2 ....100 How can i send logs to elasticsearch, as a ID of switch name and so i can search in Elastic search and make Kibana dashboards R!

2 1

Elastic 5 Support For More Than HTTP?
by JONATHAN PRICE 08 Dec '17

08 Dec '17

All, A project I'm working on is wanting to leverage syslog-ng as our log shipper, but we're ideally wanting it to communicate with Elasticsearch 5 over HTTPS or searchguard. My understanding is current support is only for HTTP for Elastic 5. Are there plans to add elastic 5 support for other client modes than HTTP anytime soon? I'm fine just doing HTTP for now and switching to secured communications later. If not, I'm okay spending a bit of time trying to get that working myself; I haven't delved too deep into the source, but the changes involving updates to the elastic module do not seem to be too too intensive and only include Java changes. Is this a correct interpretation? It doesn't look like it would be too difficult to do, but since it hasn't been done I was wondering if there was something else involved I'm missing. -- Thanks, Jonathan Price Associate Scientist, Cyber Security Raytheon BBN Technologies

2 1

Installation issue
by hari ram 07 Dec '17

07 Dec '17

Hi I have ubuntu Server install 16.X with all dependency packages #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # more /etc/debian_version stretch/sid when i try to install syslog-ng with debian package as mentioned below steps. ~# more /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ # apt-get update Hit:1 http://gb.archive.ubuntu.com/ubuntu xenial InRelease Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease Get:3 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Ign:4 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ InRelease Hit:5 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ Release Get:6 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Fetched 204 kB in 0s (350 kB/s) Reading package lists... Done # apt-get install syslog-ng-core Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: syslog-ng-core : Depends: libjson-c3 (>= 0.10) but it is not installable E: Unable to correct problems, you have held broken packages. But i have already install libjason package # apt-get install libjson-c-dev Reading package lists... Done Building dependency tree Reading state information... Done libjson-c-dev is already the newest version (0.11-4ubuntu2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. what is the issue, how can i install latest version of syslog-ng default syslog-ng only come with 3.5.6 version, but iam looking 3.11 and above version to install any suggestions ? R!

2 2