- syslog-ng - lists.balabit.hu

Global flush-lines option VS elasticsearch2 destination flush-limit param?
by Jose Angel Santiago 11 Nov '18

11 Nov '18

Hi, It seems both parameters are made for the same purpose, but I guess they don't. ¿Could you explain to me what is the difference between these two parameters? Thanks in advance. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

2 2

sending log messages to graylog
by Rodney Bizzell 06 Nov '18

06 Nov '18

I got syslog to send and echo test message from syslog server to my graylog box. How do I ensure that my syslog box will send other servers logs to my graylog box through my syslog server. I am going to setup ipvsadm as load-balancer to point my legacy application to my syslog server and then they should get shipped through to graylog. Any information is greatly appreciated

4 16

log drop-and-delete directory
by Nik Ambrosch 06 Nov '18

06 Nov '18

Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this: 1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy. Thanks.

6 10

Incremental update of syslog-ng config
by Monitto Pitchaimani sebastin (mopitcha) 05 Nov '18

05 Nov '18

Hi All, Is there a way to update the syslog-ng config without restarting the syslog-ng? Regards, Monitto

4 5

Unable to make syslog-ng listen to port 514
by Fosiul Alam 05 Nov '18

05 Nov '18

Hello i have installed syslog-ng in our ubuntu server with version : 3.13 its running fine, I am using the default syslog-ng file and i have change this line as per documentation source s_net { syslog(ip(192.168.1.19) transport("tcp")); }; but its not listing on port 514 What Do i have to do make it listen ? Thanks for the help ● syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-11-02 18:22:46 UTC; 4min 5s ago Docs: man:syslog-ng(8) Main PID: 16470 (syslog-ng) Tasks: 1 (limit: 2361) CGroup: /system.slice/syslog-ng.service └─16470 /usr/sbin/syslog-ng -F Nov 02 18:22:46 syslog systemd[1]: Starting System Logger Daemon... Nov 02 18:22:46 syslog systemd[1]: Started System Logger Daemon. -- Regards Fosiul Alam

7 12

unofficial syslog-ng 3.18 rpm packages
by Czanik, Péter 25 Oct '18

25 Oct '18

Hi, All my unofficial syslog-ng 3.18 rpm packages are now on-line. You can learn about what is new or what changed in this release in the release announcement: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1 Note, that not all features are available in all packages. Changes in packaging: - openSUSE & SLES: no changes Available at: https://build.opensuse.org/project/show/home:czanik:syslog-ng318 - Fedora & RHEL/CentOS 7: Python support changed from 2.7 to 3.X, where X is the Python 3 version available in the base distribution. For EPEL 7 packages it means 3.4, even if EPEL has 3.6. Available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng318/ - RHEL/CentOS 6: no spoof source support (does not compile), no MonogDB support (source no more bundled with syslog-ng and no ready to use external package is available). Let me know if you miss either of these features. Available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng318epel6/ Installation instructions are available at https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… but note, that you need to change the version number from 3.14 to 3.18 in the examples. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Unofficial syslog-ng-3.18 packages for Debian/Ubuntu
by Laszlo Budai 24 Oct '18

24 Oct '18

Hi, syslog-ng 3.18.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [armv7l is still building] * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 * and Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

parsing JSON logs just won't work
by Michael Niemand 18 Oct '18

18 Oct '18

Hi, I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck. I have an app, that puts out simple json log messages like: {"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"} All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as application name: {"level" message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"} it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now... This is my config (on the application host; it should send the logs directly to the logging cluster) @version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf" options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; source s_src { file( “/var/log/worker/error.log" flags(no-parse) ); }; template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); }; destination ovhPaaSLogs { tcp("gra2.logs.ovh.com" port(6514), template(unitManagerTemplate), ts_format("iso"), tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")), keep-alive(yes), so_keepalive(yes), ); }; parser p_json { json-parser(prefix(".json.")); }; log { source(s_src); parser(p_json); destination(ovhPaaSLogs); }; @include "/etc/syslog-ng/conf.d/" I also tried a different a template variant like this: template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”); I also tried parsing the messages as text: template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n"); What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though. I’d appreciate any help! Best regards, Michael

3 3

Insider 2018-10: 3.18 release; Splunk .conf18; Python destination; Python parser;
by Czanik, Péter 18 Oct '18

18 Oct '18

Dear syslog-ng users, This is the 70th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.18 of syslog-ng released ------------------------------------- Version 3.18 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to batching and multi-threading. The new Python source allows you to create source drivers for syslog-ng in Python. Learn about the small, but important changes at https://www.syslog-ng.com/community/b/blog/posts/small-is-beautiful-what-s-… or read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1. syslog-ng at Splunk .conf18 --------------------------- Balázs Scheidler, original author of syslog-ng, gave a talk at the annual Splunk conference. His talk about simplifying complex log data ingestion with syslog-ng is now available on-line. Learn about the new features coming up in syslog-ng to make it an even better choice to feed Splunk with log data: https://conf.splunk.com/files/2018/recordings/simplifying-complex-log-data-… Python destination ------------------ You can send your log messages to many different destinations using syslog-ng, but of course not everywhere. This is where the Python destination of syslog-ng can come handy. You can extend syslog-ng easily with your own code written in Python and store your log messages to the destination of your choice. Learn how to develop your own Python code for syslog-ng from our blog series, starting with the very basics and showing a complete working MQTT destination implemented in Python at the end: https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting… https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting… https://www.syslog-ng.com/community/b/blog/posts/writing-python-destination… Python parser ------------- The Python parser of syslog-ng not only enables you to parse any type of log message, but you can also use it to enrich messages. From this blog you will learn how to extract information from a specially formatted log message, and how to create new name-value pairs by consulting external databases about data contained in your log messages. I will show these using two simple parsers: one resolves host names from IP addresses, the other one uses regular expressions to parse synthetic log messages from the loggen utility. https://www.syslog-ng.com/community/b/blog/posts/parsing-log-messages-with-… WEBINARS You can watch our past webinars: * Supercharge your syslog-ng parsing: https://www.brighttalk.com/webcast/16207/330198 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

OSE 3.18.1
by Nemes, Antal 17 Oct '18

17 Oct '18

3.18.1 Features - HTTP batching (#2135 <https://github.com/balabit/syslog-ng/pull/2135>) - HTTP multi threading (#2291 <https://github.com/balabit/syslog-ng/pull/2291>) - HTTP framing support for batches (#2190 <https://github.com/balabit/syslog-ng/pull/2190>) - Python source (#2308 <https://github.com/balabit/syslog-ng/pull/2308>) - Template support for Python destination (#2196 <https://github.com/balabit/syslog-ng/pull/2196>) - HDFS time-reap (#2257 <https://github.com/balabit/syslog-ng/pull/2257>) - Riemann batching (#2098 <https://github.com/balabit/syslog-ng/pull/2098>) - flush-timeout() for HTTP (#2251 <https://github.com/balabit/syslog-ng/pull/2251>) - Base64 encoding template function (#2223 <https://github.com/balabit/syslog-ng/pull/2223>) - url-decode() template function (#2278 <https://github.com/balabit/syslog-ng/pull/2278>) - Allow IPv4 mapped addresses in IPv6 mode (#2312 <https://github.com/balabit/syslog-ng/pull/2312>) - app-parser: set ${.app.name} to the application identified (#2290 <https://github.com/balabit/syslog-ng/pull/2290>) - Value pairs improvements (#2287 <https://github.com/balabit/syslog-ng/pull/2287>) - syslog-ng-ctl config to print pre-processed configuration (#2280 <https://github.com/balabit/syslog-ng/pull/2280>) - Add support for whitespace in template functions (#2271 <https://github.com/balabit/syslog-ng/pull/2271>) - Verify the configuration is the same as on the disk (#2345 <https://github.com/balabit/syslog-ng/pull/2345>) Bugfixes - Fix queue counters race condition (#2316 <https://github.com/balabit/syslog-ng/pull/2316>) - AMQP configurable max connection and frame size (#2343 <https://github.com/balabit/syslog-ng/pull/2343>) - Fix wakeup in threaded sources (#2339 <https://github.com/balabit/syslog-ng/pull/2339>) - Fix libnet memory leak in network destinations (#2331 <https://github.com/balabit/syslog-ng/pull/2331>) - Fix unexpected flag check-hostname in syslog-parser (#2314 <https://github.com/balabit/syslog-ng/pull/2314>) - Fix memory leak in dbparser (#2311 <https://github.com/balabit/syslog-ng/pull/2311>) - Inline Python code comment generates syntax error (#2319 <https://github.com/balabit/syslog-ng/pull/2319>) - Password protected SSL keys portability (MADV_DONTDUMP) (#2341 <https://github.com/balabit/syslog-ng/pull/2341>) - Fixing compiler warnings from armv7l (#2301 <https://github.com/balabit/syslog-ng/pull/2301>) - Riemann crashes in flush (#2296 <https://github.com/balabit/syslog-ng/pull/2296>) - Revert sticky hook option (#2295 <https://github.com/balabit/syslog-ng/pull/2295>) - Small stats fixes (#2294 <https://github.com/balabit/syslog-ng/pull/2294>) - Detect filter loop (#2283 <https://github.com/balabit/syslog-ng/pull/2283>, #2288 <https://github.com/balabit/syslog-ng/pull/2288>) - Fix infinite loop in threaded destinations with ivykis prior 0.39 ( #2275 <https://github.com/balabit/syslog-ng/pull/2275>) - Fix log expr node use after free (#2269 <https://github.com/balabit/syslog-ng/pull/2269>) - Remove gradle from the list of "BuildRequires" (RPM packaging) (#2266 <https://github.com/balabit/syslog-ng/pull/2266>) - Fix wildcard-source memleak when directory removed (#2261 <https://github.com/balabit/syslog-ng/pull/2261>, #2267 <https://github.com/balabit/syslog-ng/pull/2267>) - Missing macros: C_AMPM, C_USEC, C_MSEC, C_HOUR12 (#2259 <https://github.com/balabit/syslog-ng/pull/2259>) - Fix cisco timestamp parsing (#2272 <https://github.com/balabit/syslog-ng/pull/2272>) - Undefined filter reference (#2273 <https://github.com/balabit/syslog-ng/pull/2273>) - Fix the literal() type hint (#2286 <https://github.com/balabit/syslog-ng/pull/2286>) - logwriter, affile, afsocket: fixing "internal overflow". (#2250 <https://github.com/balabit/syslog-ng/pull/2250>) - lib/gsockaddr.c: modify the unix salen calculation (#2285 <https://github.com/balabit/syslog-ng/pull/2285>) - dbld: fix dbld/rules deb failure (#2282 <https://github.com/balabit/syslog-ng/pull/2282>) - Update data type to avoid conversion. (#2281 <https://github.com/balabit/syslog-ng/pull/2281>) - compat/getent: add support for platforms that lack the r versions ( #2244 <https://github.com/balabit/syslog-ng/pull/2244>) - Fix memory leak caused by saving stats counter to persist config (#2279 <https://github.com/balabit/syslog-ng/pull/2279>) - Hdfs: disable archive when append-enabled is configured (#2235 <https://github.com/balabit/syslog-ng/pull/2235>) - scl: add linux-audit() SCL to make files (#2230 <https://github.com/balabit/syslog-ng/pull/2230>) - DebianPackaging: Add linux-audit SCL to included dirs (#2254 <https://github.com/balabit/syslog-ng/pull/2254>) - cap_syslog capability detection (#2227 <https://github.com/balabit/syslog-ng/pull/2227>) - LogProto partial write (#2194 <https://github.com/balabit/syslog-ng/pull/2194>) Other changes - telegram, urlencode: api changes (#2299 <https://github.com/balabit/syslog-ng/pull/2299>) - python: include python2/3 in plugin description (#2337 <https://github.com/balabit/syslog-ng/pull/2337>) - Stats prepare for multiple queues per destdrv (#2302 <https://github.com/balabit/syslog-ng/pull/2302>) - Update deprecated use of tcp()/udp() to network(). This addresses #2322 <https://github.com/balabit/syslog-ng/issues/2322> (#2326 <https://github.com/balabit/syslog-ng/pull/2326>) - Message about not supported cap_syslog only at debug level (#2333 <https://github.com/balabit/syslog-ng/pull/2333>) - Few test leak fix (#2323 <https://github.com/balabit/syslog-ng/pull/2323>) - warning elimination: pointer arithmetics (#2305 <https://github.com/balabit/syslog-ng/pull/2305>) - templates: get rid off the args_lock (#2289 <https://github.com/balabit/syslog-ng/pull/2289>) - Ack tracker small refact (#2277 <https://github.com/balabit/syslog-ng/pull/2277>) - ElasticSearchDestination: Display deprecated warning message about us… (#2274 <https://github.com/balabit/syslog-ng/pull/2274>) - Improve Readme.md header structure (#2258 <https://github.com/balabit/syslog-ng/pull/2258>) - Rewrite json tests based on criterion (#2255 <https://github.com/balabit/syslog-ng/pull/2255>) - Rewrite dbparser tests based on criterion (#2252 <https://github.com/balabit/syslog-ng/pull/2252>) - Processed timestamp (#2243 <https://github.com/balabit/syslog-ng/pull/2243>) - msg parameters: remove last NULL parameter from msg macros (#2242 <https://github.com/balabit/syslog-ng/pull/2242>) - Fix threaded destination test cases (#2236 <https://github.com/balabit/syslog-ng/pull/2236>) - dbld: missing packages, deps changes (#2232 <https://github.com/balabit/syslog-ng/pull/2232>, #2332 <https://github.com/balabit/syslog-ng/pull/2332>, #2327 <https://github.com/balabit/syslog-ng/pull/2327>, #2260 <https://github.com/balabit/syslog-ng/pull/2260>, #2256 <https://github.com/balabit/syslog-ng/pull/2256>) - Use msg_trace() when emitting trace information (#2226 <https://github.com/balabit/syslog-ng/pull/2226>) Notes to the developers - Threaded source and fetcher (#2247 <https://github.com/balabit/syslog-ng/pull/2247>) - "Examples" module (#2248 <https://github.com/balabit/syslog-ng/pull/2248>) - Do not ship mongo-c-driver with syslog-ng (remove as submodule) (#2245 <https://github.com/balabit/syslog-ng/pull/2245>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, Gabor Nagy, Gergely Tonté, Laszlo Budai, Laszlo Szemere, László Várady, Maurice T. Meyer, Mahmoud Salama, Norbert Takacs, Peter Czanik, Peter Gyorko, Peter Kokai, Robert Fekete, Terez Nemes, Tibor Bodnar, Zoltan Pallagi, y-l-i. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1 <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> balabit/syslog-ng <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. github.com Regards, Antal Nemes

1 0

syslog-ng 3.5
by Rodney Bizzell 15 Oct '18

15 Oct '18

I have install syslog-ng 3.5 and I am configuring it to send all logs to Graylog but once changes are made syslog-ng is failing. I change the address of graylog server to 0.0.0.0 @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); internal(); udp(ip(0.0.0.0) port(514)); }; source s_net { udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(514) max-connections(256)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_graylog { tcp("0.0.0.0" port (12201) spoof_sources(yes) ); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_net); destination(d_graylog); }; log { source(s_sys); filter(f_default); destination(d_graylog);}; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:

4 14

CentOS syslog-ng using tls won't start via systemd but runs fine manually (sort of)
by craig bowser 12 Oct '18

12 Oct '18

I'm been trying to figure this out for several days now. I have Centos 7 installed on two servers and I want to use syslog-ng to send logs between them securely using TLS. On both I installed syslog-ng 3.5 using this method: https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… I have been following these methods as a starting point to set up: https://www.nineproductions.com/syslog-ng-with-tls-setup-for-centralized-lo… https://www.linux.com/blog/tls-encryption-and-mutual-authentication-using-s… https://www.flyn.org/notes/logging/index.html On the receiving server, I have syslog-ng with the following config file: ---------------------------------------------------------- @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); # log_fifo_size (1000); # chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (1); keep_hostname (yes); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; source s_net { tcp(ip(10.1.0.2) port(5441) tls( ca_dir("/etc/syslog-ng/ca.d") key_file("/etc/syslog-ng/newCA/serverkey.pem") cert_file("/etc/syslog-ng/newCA/servercert.pem") peer_verify(optional-untrusted)) ); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_log { file("/opt/syslog/$YEAR$MONTH$DAY-logs.log" create_dirs(yes) dir_group(logstash) dir_owner(logstash) owner(logstash) group(logstash));}; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_net); destination(d_log); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: ------------------------------------------------------ Here is my java verison: # java -version openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) This is my OS version: # cat /etc/*release CentOS Linux release 7.2.1511 (Core) NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" CentOS Linux release 7.2.1511 (Core) CentOS Linux release 7.2.1511 (Core) If I try to start syslog-ng via systemctl, this is what happens: # systemctl start syslog-ng Job for syslog-ng.service failed because the control process exited with error code. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. # systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Fri 2018-10-12 11:57:09 EDT; 13s ago Docs: man:syslog-ng(8) Process: 5706 ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid (code=exited, status=2) Main PID: 5706 (code=exited, status=2) Status: "Starting up... (Fri Oct 12 11:57:09 2018" Oct 12 11:57:09 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 11:57:09 logger.com systemd[1]: start request repeated too quickly for syslog-ng.service Oct 12 11:57:09 logger.com systemd[1]: Failed to start System Logger Daemon. Oct 12 11:57:09 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:09 logger.com systemd[1]: start request repeated too quickly for syslog-ng.service Oct 12 11:57:09 logger.com systemd[1]: Failed to start System Logger Daemon. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. #journalctl -xe Oct 12 11:57:08 logger.com polkitd[818]: Registered Authentication Agent for unix-process:5693:6083335 (system bus name :1.97 [/usr/bin/pkttyagent --notify-fd 5 - Oct 12 11:57:08 logger.com systemd[1]: Listening on Syslog Socket. -- Subject: Unit syslog.socket has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog.socket has finished starting up. -- -- The start-up result is done. Oct 12 11:57:08 logger.com systemd[1]: Starting Syslog Socket. -- Subject: Unit syslog.socket has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog.socket has begun starting up. Oct 12 11:57:08 logger.com systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT Oct 12 11:57:08 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:08 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:08 logger.com polkitd[818]: Unregistered Authentication Agent for unix-process:5693:6083335 (system bus name :1.97, object path /org/freedesktop/Poli Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 11:57:08 logger.com systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT Oct 12 11:57:08 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:08 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:08 logger.com systemd-journal[494]: Forwarding to syslog missed 3 messages. -- Subject: One or more messages could not be forwarded to syslog -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- One or more messages could not be forwarded to the syslog service -- running side-by-side with journald. This usually indicates that the -- syslog implementation has not been able to keep up with the speed of -- messages queued. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 11:57:08 logger.com systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT Oct 12 11:57:08 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:08 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:08 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 11:57:09 logger.com systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT Oct 12 11:57:09 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:09 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 11:57:09 logger.com systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT Oct 12 11:57:09 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:09 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 11:57:09 logger.com systemd[1]: start request repeated too quickly for syslog-ng.service Oct 12 11:57:09 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:09 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. Oct 12 11:57:09 logger.com systemd[1]: start request repeated too quickly for syslog-ng.service Oct 12 11:57:09 logger.com systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. Oct 12 11:57:09 logger.com systemd[1]: Unit syslog.socket entered failed state. Oct 12 11:57:09 logger.com systemd[1]: syslog-ng.service failed. But if I start with the command line: # /usr/sbin/syslog-ng -Fedv -p /var/run/syslogd.pid Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1' Global value changed; define='scl-root', value='/usr/share/syslog-ng/include/scl' Global value changed; define='include-path', value='/etc/syslog-ng:/usr/share/syslog-ng/include' Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' Module loaded and initialized successfully; module='system-source' Finishing include; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' Starting to read include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' Reading path for candidate modules; path='//usr/lib64/syslog-ng' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='affile.so', module='affile' Registering candidate plugin; module='affile', context='source', name='file', preference='0' Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' Registering candidate plugin; module='affile', context='destination', name='file', preference='0' Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afprog.so', module='afprog' Registering candidate plugin; module='afprog', context='source', name='program', preference='0' Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afsocket-notls.so', module='afsocket-notls' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-stream', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-dgram', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='udp6', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='syslog', preference='0' Registering candidate plugin; module='afsocket-notls', context='source', name='network', preference='0' Registering candidate plugin; module='afsocket-notls', context='destination', name='network', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afsocket-tls.so', module='afsocket-tls' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket-tls', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket-tls', context='destination', name='network', preference='100' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afsocket.so', module='afsocket' Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afstomp.so', module='afstomp' Registering candidate plugin; module='afstomp', context='destination', name='stomp', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afuser.so', module='afuser' Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='basicfuncs.so', module='basicfuncs' Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' Registering candidate plugin; module='basicfuncs', context='template-func', name='env', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='confgen.so', module='confgen' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='csvparser.so', module='csvparser' Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='dbparser.so', module='dbparser' Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='linux-kmsg-format.so', module='linux-kmsg-format' Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='syslogformat.so', module='syslogformat' Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='system-source.so', module='system-source' Finishing include; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' Starting to read include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' Module loaded and initialized successfully; module='confgen' Finishing include; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' Finishing include; filename='/etc/syslog-ng/scl.conf', depth='1' Module loaded and initialized successfully; module='afsocket-tls' Module loaded and initialized successfully; module='affile' Finishing include; content='source confgen system', depth='1' Module loaded and initialized successfully; module='afuser' Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling s_sys sequence [source] at [/etc/syslog-ng/syslog-ng.conf:2] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:2] Compiling #unnamed single [log] at [#buffer:1:1] Compiling #unnamed single [log] at [#buffer:2:1] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:2] Compiling f_kernel reference [filter] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling f_kernel sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling d_kern reference [destination] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling d_kern sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_default reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_default sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling d_mesg reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_mesg sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_auth reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_auth sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling d_auth reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_auth sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_mail reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_mail sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling d_mail reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_mail sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_emergency reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_emergency sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling d_mlal reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_mlal sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_news reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_news sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling d_spol reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_spol sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_boot reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_boot sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling d_boot reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_boot sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_sys reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_cron reference [filter] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling f_cron sequence [filter] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:5] Compiling d_cron reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_cron sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_net reference [source] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling s_net sequence [source] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:3] Compiling d_log reference [destination] at [/etc/syslog-ng/syslog-ng.conf:6] Compiling d_log sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:4] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:4] Module loaded and initialized successfully; module='syslogformat' Module loaded and initialized successfully; module='linux-kmsg-format' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.5.6' If I comment out this line from syslog-ng.conf: # log { source(s_net); destination(d_log); }; Then restart syslog-ng (I know, I should use restart...) # systemctl start syslog-ng Starts no problem. # systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-10-12 12:00:47 EDT; 38s ago Docs: man:syslog-ng(8) Main PID: 5727 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─5727 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid Oct 12 12:00:47 logger.com systemd[1]: Starting System Logger Daemon... Oct 12 12:00:47 logger.com systemd[1]: Started System Logger Daemon. It just won't stop (I guess by design) # systemctl stop syslog-ng Warning: Stopping syslog-ng.service, but it can still be activated by: syslog.socket # systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-10-12 12:01:55 EDT; 1s ago Docs: man:syslog-ng(8) Main PID: 5755 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─5755 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid Oct 12 12:01:55 logger.com systemd[1]: Starting System Logger Daemon... Oct 12 12:01:55 logger.com systemd[1]: Started System Logger Daemon. But if I uncomment that line and restart: # systemctl restart syslog-ng Job for syslog-ng.service failed because the control process exited with error code. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. # service syslog-ng status Redirecting to /bin/systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Fri 2018-10-12 12:04:18 EDT; 2s ago Docs: man:syslog-ng(8) Process: 5811 ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid (code=exited, status=2) Main PID: 5811 (code=exited, status=2) Status: "Starting up... (Fri Oct 12 12:04:18 2018" Oct 12 12:04:18 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 12:04:18 logger.com systemd[1]: syslog-ng.service failed. Oct 12 12:04:18 logger.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Oct 12 12:04:18 logger.com systemd[1]: start request repeated too quickly for syslog-ng.service Oct 12 12:04:18 logger.com systemd[1]: Failed to start System Logger Daemon. Oct 12 12:04:18 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 12:04:18 logger.com systemd[1]: syslog-ng.service failed. Oct 12 12:04:18 logger.com systemd[1]: start request repeated too quickly for syslog-ng.service Oct 12 12:04:18 logger.com systemd[1]: Failed to start System Logger Daemon. Oct 12 12:04:18 logger.com systemd[1]: syslog-ng.service failed. It won't work. If I change syslog-ng.conf to a specific IP in the source I get nearly the same results. One note: Occasionally, in the journalctl -xe, I get the following: Oct 12 12:08:55 logger.com systemd[1]: Unit syslog-ng.service entered failed state. Oct 12 12:08:55 logger.com systemd[1]: syslog-ng.service failed. Oct 12 12:08:55 logger.com systemd-journal[494]: Forwarding to syslog missed 3 messages. -- Subject: One or more messages could not be forwarded to syslog -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- One or more messages could not be forwarded to the syslog service -- running side-by-side with journald. This usually indicates that the -- syslog implementation has not been able to keep up with the speed of -- messages queued. But that doesn't show up all the time. As far as options, it doesn't look like https://www.centos.org/forums/viewtopic.php?t=58296 https://www.syslog-ng.com/community/b/blog/posts/troubleshooting-java-suppo… rsyslog is disabled # systemctl status rsyslog ● rsyslog.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; disabled; vendor preset: enabled) Active: inactive (dead) Oct 11 19:03:17 logger.com systemd[1]: Starting System Logging Service... Oct 11 19:03:17 logger.com systemd[1]: Started System Logging Service. Oct 11 20:52:56 logger.com systemd[1]: Stopping System Logging Service... Oct 11 20:52:56 logger.com systemd[1]: Stopped System Logging Service. https://www.centos.org/forums/viewtopic.php?t=57379 Nothing seems to be blocking access to the log directory: # lsof /dev/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd 1 root 42u unix 0xffff8800368952c0 0t0 9277 /dev/log systemd-j 494 root 5u unix 0xffff8800368952c0 0t0 9277 /dev/log https://www.reddit.com/r/linuxadmin/comments/5ezj5c/syslogng_service_doesnt… I've also looked at https://unix.stackexchange.com/questions/202044/syslog-ng-service-not-start… and https://lists.balabit.hu/pipermail/syslog-ng/2017-September/024024.html

1 0

RFC-5424 configuration with IPV6 is not working
by venkateswarlu vinjamuri 09 Oct '18

09 Oct '18

Hi, *issue*: syslog-ng is not starting if we configure IPV6 IP along with RFC-5424 format Using below command to run syslog-ng: /sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -R /var/syslog-ng.persist -p /var/syslog-ng.pid Getting the below error: ------------------------------ *syslog-ng: Error changing to Unknown syslog transport specified, please use one of udp, tcp, or tls; transport='udp6', id='df_remote_0#0'* Could anyone please let me know what should be the configuration in syslog-ng.conf for IPV6 syslog-ng server IP with RFC-5424 format. we are using syslog-ng-3.3.7 version. *If the below configuration is correct, will it work if we upgrade to newer version with the below configuration?* I am using the below configuration in syslog-ng.conf: ======================================== options { stats_freq (0); flush_lines (0); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); create_dirs (no); keep_hostname (no); perm(0640); group("root"); }; # sources source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; filter f_remote { facility(local7); }; destination df_remote_0 {syslog("xxxx:xxxx:xxxx:xxxx:xxxx" transport("udp6"));}; log { source(s_all); filter(f_remote); destination(df_remote_0 Please help if there is any issue in the above configuration? Regards, V/

2 8

syslong_ng and bind9
by Ruben Safir 09 Oct '18

09 Oct '18

Hello I am trying to log named to syslog_ng and it is not set up by default on arch or artex. I set up named for logging, I think logging { category default { log_syslog; }; channel log_syslog { syslog; }; }; in the config file for syslog_ng I had added destination d_mail { file("/var/log/mail.log"); }; destination d_named { file("/var/log/named.log"); }; destination d_news { file("/var/log/news.log"); }; I am looking at filters and have no idea what to do filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { program(syslog-ng); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kernel { facility(kern) and not filter(f_iptables); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_news { facility(news); }; filter f_user { facility(user); }; filter f_uucp { facility(uucp); }; filter f_ppp { facility(local2); }; filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news, cron) and not program(syslog-ng) and not filter(f_iptables); }; filter f_everything { level(debug..emerg) and not facility(auth, authpriv); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; filter f_iptables { match("IN=" value("MESSAGE")) and match("OUT=" value("MESSAGE")); }; filter f_acpid { program("acpid"); }; I don't know if there is a built in facilty for named. The docs for the server and very detailed and rich, and overwhelming. Ruben -- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www.brooklyn-living.com Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013

2 2

'network' Destination With Hostname Resolution (IPv4 vs IPv6)
by David Hauck 06 Oct '18

06 Oct '18

Hi, I have a question regarding how to specify a network destination when using a hostname when the hostname can be resolved to either IPv4 or IPv6. In particular what should be specified by the ip-protocol() parameter? There are some configuration scenarios and/or target installations that don't know a priori whether the DNS configuration will resolve to an IPv4 or IPv6 address. E.g., destination d_tcp6 { network( "myhost" port(514) transport(udp) ip-protocol(6 or 4 or ??) ); }; It seems like it would simple enough to have syslog-ng simply validate the resulting IP address string to determine which of ip-protocol(4) or ip-protocol(6) is actually needed. In fact, I would argue that specifying an IP address string (as the "<destination-address>" value) could result in the same determination (the address string necessarily unambiguously determines whether the reference is an IPv4 or an IPv6 address and I would think there is a 1-1 relationship between this determination and whether ip-protocol(4) or ip-protocol(6) is used - (in other words it would never make sense to have these mixed: "::1" and ip-protocol(4) would be invalid). Thanks, -David

3 12

pattern usage and optimization
by Joel Carnat 01 Oct '18

01 Oct '18

Hi, I want to parse my sshd logs to store information in a remote database. I already did it using logstash. But I just discovered syslog-ng can do such things using patterndb. I could manage to setup a few <pattern> but I have difficulties building generic rules. I end up with 4 or 5 rules where I can only deal with one or two using logstash. So I expect to be missing something with patterns :) Here's a log example: Disconnected from user joe 192.168.0.5 port 50121 Disconnected from invalid user www 192.168.0.7 port 6794 [preauth] Disconnected from authenticating user root 192.168.0.3 port 52591 [preauth] So I wrote those three patterns:  <pattern>@ESTRING:EVENT: from @user @ESTRING:USERNAME: @@ESTRING:IP: @port @NUMBER:PORT:@</pattern>  <pattern>@ESTRING:EVENT: from @invalid user @ESTRING:USERNAME: @@ESTRING:IP: @port @NUMBER:PORT:@@ANYSTRING:EXTRA:@</pattern>  <pattern>@ESTRING:EVENT: from @authenticating user @ESTRING:USERNAME: @@ESTRING:IP: @port @NUMBER:PORT:@@ANYSTRING:EXTRA:@</pattern> To me, those 3 lines can be described using a single expression this way : ("Disconnected from") ("user"|"invalid user"|"authenticating user") (username) (ip_host) port (ip_port)(empty|extra_stuff) Basically, the features I couldn't find are : - "match a defined string and affect to variable" - "match a string or another and affect to variable" - "match a string or EOL and affect to variable if not empty". Is it possible to have a single pattern that would lead to have: - EVENT = "Disconnected from" - METHOD = "user" | "invalid user" | "authenticating user" - USERNAME = <parsed username> - IP = <parsed ip address> - PORT = <parsed port number> - EXTRA = <empty> | <parsed extra information> Thanks for you help.

2 3

Undesirable behavior from Cisco parser?
by Nik Ambrosch 27 Sep '18

27 Sep '18

Recently I upgraded my centralized loghost from 3.9 -> 3.15 and I noticed that some of my cisco devices started being logged in an undesirable format... I don't want to enable the cisco parser because more than just cisco messages get delivered to this interface. Here are the relevant fields that have changed before/after the upgrade: syslog-ng 3.9, before upgrade --- ${FULLHOST}: "mydevice.com" ${PROGRAM}: "" message: "%CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for..." syslog-ng 3.15, before upgrade --- ${FULLHOST}: ":" ${PROGRAM}: "%CRYPTO-4-RECVD_PKT_INV_SPI" ${MSG}: "decaps: rec'd IPSEC packet has invalid spi for..." Is this unintended behavior or a bug? This particular device is a Cisco 3845 running ios 12.4(22)T4. Thanks in advance.

5 17

disk-buffer in elasticsearch2 destination loses messages if docker container is killed
by Jose Angel Santiago 25 Sep '18

25 Sep '18

Hi, I'm running syslog-ng (with an elasticsearch2 destination configured) within a docker container, and I'm trying to avoid loss of messages if I kill the docker container and I start it again. This is my scenary: - A service which produces 20 lines of log per second - A sislog-ng instance reading from a wildcard-file source (but actually it only reads logs from the above service, let's call it syslog-agent), which sends all logs to another syslog-ng instance (the one running in a docker container, let's call it syslog-relay) though a network destination. - The syslog-relay sends messages to an elasticsearch instance, with following configuration: options { chain-hostnames(no); use-dns(no); keep-hostname(yes); owner("syslog-ng"); group("stratio"); perm(0640); time-reap(30); mark-freq(10); stats-freq(0); bad-hostname("^gconfd$"); flush-lines(100); log-fifo-size(1000); }; *destination d_elastic_default_0 { elasticsearch2( cluster("myelastic") cluster-url("https://myelastic.logs:9200 <https://myelastic.logs:9200>") client_mode("https") index("default") type("log") flush-limit(1) disk-buffer( mem-buf-size(16M) disk-buf-size(16M) reliable(yes) dir("/syslog-ng/log") ) http-auth-type("clientcert") java-keystore-filepath("/etc/syslog-ng/certificates/syslog-relay.jks") java-keystore-password("XXXXXX") java-truststore-filepath("/etc/syslog-ng/certificates/ca-bundle.jks") java-truststore-password("XXXXXXXXXX") );};* - The dir "/syslog-ng/log" is mapped to a path "/tmp/buffer" from the host where the docker container is running, so when I kill the docker container, the buffer file is not lost. - I've set flush-limit to 1 because I thought that I may lost 1 message only as much. This architecture is working fine (flush-limit=1 makes very slow, but for this test is ok), but if I kill the syslog-relay docker container, wait 5 to 10 seconds and start it again from scratch, I can see that several hundreds of logs are missing in elasticsearch. I check it by stopping the logger service and letting syslog-ng agent & relay to finish the process enqueued messages. I can see in the syslog-agent stats that all logs messages have been processed, so it seems the problem is on the syslog-relay. Is this behaviour expected? If so, how can I protect against loss of messages in case of a syslog-relay docker container unexpected kill? Thanks in advance. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

3 16

Syslog-ng missing few logs to write to local file after showing in tcpdump
by Harish Shetty 24 Sep '18

24 Sep '18

Hi All I have syslog-ng (syslog-ng-3.9.1-1.el6.x86_64) which is running on Red Hat 6.6. I have rule defined as mentioned below destination d_networkx { file("/mnt/u001/syslog/$HOST/$YEAR$MONTH$DAY/network.log"); }; But syslog-ng is unable to write some logs to local file even though we see those logs in the tcpdump? What could be causing this. And one more thing need to know , is ther any way to check per filter (d_networkx) how many we received and how many we Processed(wrote) to a file Regards Harish Shetty

3 2

Insider 2018-09: SCL; Splunk .conf18; failover; human readable numbers; webinars;
by Czanik, Péter 20 Sep '18

20 Sep '18

Dear syslog-ng users, This is the 69th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Creating your first block for the syslog-ng configuration library (SCL) --------------------------------------------------------- The syslog-ng configuration library (SCL) is a collection of ready-to-use configuration snippets that hide away the complexity of the specifics of your log processing pipeline. If you already reuse parts of your configuration on different machines, it is not rocket science to turn these parts into blocks, making your configuration files a lot simpler. And if they are not site-specific, you can even help other syslog-ng users by submitting them to be included in SCL, and distributed in the official syslog-ng releases. https://www.syslog-ng.com/community/b/blog/posts/creating-your-first-block-… syslog-ng at Splunk .conf18 --------------------------- Going to Splunk .conf18? See a demo, meet with us (Booth #M49) and talk to syslog-ng creator, Balázs Scheidler. He will also give a talk on 02 October, 2:15 - 3:00 p.m. about simplifying complex log data ingestion with syslog-ng: https://conf.splunk.com/learn/session-catalog.html?search=ITS2133#/ Client-side failover and failback using syslog-ng ------------------------------------------------- When you have multiple syslog servers collecting logs, syslog-ng, on the client side, can fail over to secondary servers if the primary one becomes unavailable. It can also fail back to the primary server soon after it is back online – if configured so. Using client-side failover lowers the risk of message loss and makes the maintenance of servers easier. https://www.syslog-ng.com/community/b/blog/posts/client-side-failover-and-f… Human readable numbers in syslog-ng ----------------------------------- Writing an extra zero at the wrong place in a configuration file can result in a filled-up partition and data loss. This is where human readable numbers in syslog-ng can help: using kilo / mega / giga instead of writing many-digit numbers can save you lots of debugging and headache. https://www.syslog-ng.com/community/b/blog/posts/human-readable-numbers-in-… WEBINARS You can watch our past webinars: * Logging in docker with syslog-ng: https://www.brighttalk.com/webcast/16207/333091 * Supercharge your syslog-ng parsing: https://www.brighttalk.com/webcast/16207/330198 Or register for our upcoming syslog-ng webinars: * High performance log streaming to HDFS with syslog-ng: https://www.syslog-ng.com/event/high-performance-log-streaming-to-hdfs-with… Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> One Identity / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Mutual TLS issues
by Matthew Pounsett 19 Sep '18

19 Sep '18

I'm trying to get mutual TLS validation working and seem to be having issues with the config. The situation I've got is that I'm trying to accept a stream from another organization. And so my preferred configuration involves two CA's.. mine and theirs. My syslog-ng server needs to validate their server certificate using their CA certificate, and their syslog-ng client needs to validate my server certificate using my CA certificate. I've set up a test using the server that will be production, and a handy VM I control. I assumed that it would be as simple as using the example configs from the manual [0], and anywhere a CA cert is involved just pick the appropriate one: store their CA on my server, and my CA on "their" server (the test VM). That didn't work. I've backed down to trying to get this working using all certificates signed by my CA, but I can't seem to make that work either. The error timing I'm getting doesn't quite match similar errors I can see reported online. If I don't have the correct CA cert on the client, then I see a connect, immediate disconnect, and a "SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca" error on the server. If I do have the CA cert on the client, I see a connect ... and then nothing until I try to send a message. Then I get the same error on the server and a logged "Certificate validation failed" error on the client referencing the server's cert, and a disconnect. Server logs: Sep 17 23:17:49 adm syslog-ng[31128]: Syslog connection accepted; fd='24', client='AF_INET(216.235.10.40:54063)', local='AF_INET(64.191.0.17:6514)' Sep 17 23:18:01 adm syslog-ng[31128]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' Sep 17 23:18:01 adm syslog-ng[31128]: I/O error occurred while reading; fd='24', error='Connection reset by peer (104)' Sep 17 23:18:01 adm syslog-ng[31128]: Syslog connection closed; fd='24', client='AF_INET(216.235.10.40:54063)', local='AF_INET(64.191.0.17:6514)' Client logs: Sep 17 23:17:48 ubuntu syslog-ng[4603]: syslog-ng starting up; version='3.13.2' Sep 17 23:17:48 ubuntu syslog-ng[4603]: Syslog connection established; fd='13', server='AF_INET(64.191.0.17:6514)', local='AF_INET(0.0.0.0:0)' Sep 17 23:18:00 ubuntu syslog-ng[4603]: Certificate validation failed; subject='emailAddress=xxx(a)dns-oarc.net, CN=64.191.0.17, O=OARC\, Inc., L=Indianapolis, ST=Indiana, C=US', issuer='emailAddress=xxx(a)dns-oarc.net, CN=OARC Inc. Certificate Authority, O=OARC\, Inc., ST=Indiana, C=US', error='unable to get local issuer certificate', depth='0' Sep 17 23:18:00 ubuntu syslog-ng[4603]: SSL error while writing stream; tls_error='SSL routines:tls_process_server_certificate:certificate verify failed' Sep 17 23:18:00 ubuntu syslog-ng[4603]: I/O error occurred while writing; fd='13', error='Broken pipe (32)' Sep 17 23:18:00 ubuntu syslog-ng[4603]: Syslog connection broken; fd='13', server='AF_INET(64.191.0.17:6514)', time_reopen='60' Note the 12 second gap between startup and cert validation failure. The failure coincides with logging a message that should be sent from the client to the server. Server config: source tls_source { network(ip(64.191.0.17) port(6514) transport("tls") tls( key-file("/etc/ssl/private/64.191.0.17.key") cert-file("/etc/ssl/certs/64.191.0.17.crt") ca-dir("/etc/ssl/certs") ) ); }; Client config: destination d_tls { network("64.191.0.17" port(6514) transport("tls") tls( ca-dir("/etc/ssl/certs") key-file("/etc/ssl/private/216.235.10.40.key") cert-file("/etc/ssl/certs/216.235.10.40.crt") ) ); }; I have the CA cert stored as /etc/ssl/certs/server_ca.pem on both systems. Any idea what's wrong here? And, should I be able to get this to work using different CAs for the client and server certs? Thanks [0]: <https://www.syslog-ng.com/technical-documents/doc/ syslog-ng-open-source-edition/3.16/mutual-authentication- using-tls#TOPIC-956368>

2 3

@ as delimiter in patterndb parser
by joao reis 13 Sep '18

13 Sep '18

How can I specify @ as a delimiter in patterndb? I want to parse user and domain separated in email adresses. Thanks in advance.

2 1

Packet loss.
by vijay amruth 13 Sep '18

13 Sep '18

Hello everyone, we’ve been experiencing packet loss on syslog-ng collectors (these are open stack VMs), lately we have increased ingestion and that refelcted packet loss ( especially UDP). Tuning parameters didn’t help. If anyone here on the list has experienced a similar issue please share your resolution or work around. We are at 3.9.1, we are planning to update to 3.17.x for better data handling/other bug fixes. As an alternate approach we have increased the VM capacity ( no of VMs). If there is any that I missed, please point. Any thoughts? Thank you, Vijay Amrut. -- Thanks, Vijay Amrut.

3 4

Crash on syslog-ng 3.8.1
by xen rkar 12 Sep '18

12 Sep '18

Hi, Looking to see if this is a know issue, cause i have been unsucessful in find an exact match of the signature anywhere in the forums. Regards, rkar #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x76c5775c in __GI_abort () at abort.c:79 #2 0x76f537a8 in iv_fatal (fmt=0x76f750dc "iv_timer_unregister: timer index %d > %d") at ../../../../syslog-ng-3.8.1/lib/ivykis/src/iv_fatal.c:57 #3 0x76f53f4c in iv_timer_unregister (_t=0x17e5284) at ../../../../syslog-ng-3.8.1/lib/ivykis/src/iv_timer.c:305 #4 0x76f53fb8 in iv_run_timers (st=st@entry=0x17a50c0) at ../../../../syslog-ng-3.8.1/lib/ivykis/src/iv_timer.c:148 #5 0x76f55ad8 in iv_main () at ../../../../syslog-ng-3.8.1/lib/ivykis/src/iv_main_posix.c:107 #6 0x76f24524 in main_loop_run () at ../syslog-ng-3.8.1/lib/mainloop.c:497 #7 0x000113e0 in main (argc=70624, argv=0x7ea96054) at ../syslog-ng-3.8.1/syslog-ng/main.c:289

2 1

syslog-ng never frees RAM
by Jose Angel Santiago 10 Sep '18

10 Sep '18

Hi, I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay. I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation. log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. I've performed the following test to reproduce the situation in small scale: - Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is. How do I get syslog-ng to free RAM? Or is it a memory leak? Thanks in advance. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

3 7