October 2018 - syslog-ng - lists.balabit.hu

IPV6 and IPV4 syslog-ng issue
by venkateswarlu vinjamuri 22 Nov '18

22 Nov '18

Hi, *Issue:* *when configured both IPV4 and IPV6 syslog server IP's, I could see syslog-ng messages going to only one syslog server, either IPV4 or IPV6 syslog-ng server and that too to the latest server IP configured only.* Is there any known issue ? Please help me on this. Thanks & Regards, Venkat.

2 3

Global flush-lines option VS elasticsearch2 destination flush-limit param?
by Jose Angel Santiago 11 Nov '18

11 Nov '18

Hi, It seems both parameters are made for the same purpose, but I guess they don't. ¿Could you explain to me what is the difference between these two parameters? Thanks in advance. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

2 2

log drop-and-delete directory
by Nik Ambrosch 06 Nov '18

06 Nov '18

Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this: 1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy. Thanks.

6 10

Incremental update of syslog-ng config
by Monitto Pitchaimani sebastin (mopitcha) 05 Nov '18

05 Nov '18

Hi All, Is there a way to update the syslog-ng config without restarting the syslog-ng? Regards, Monitto

4 5

unofficial syslog-ng 3.18 rpm packages
by Czanik, Péter 25 Oct '18

25 Oct '18

Hi, All my unofficial syslog-ng 3.18 rpm packages are now on-line. You can learn about what is new or what changed in this release in the release announcement: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1 Note, that not all features are available in all packages. Changes in packaging: - openSUSE & SLES: no changes Available at: https://build.opensuse.org/project/show/home:czanik:syslog-ng318 - Fedora & RHEL/CentOS 7: Python support changed from 2.7 to 3.X, where X is the Python 3 version available in the base distribution. For EPEL 7 packages it means 3.4, even if EPEL has 3.6. Available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng318/ - RHEL/CentOS 6: no spoof source support (does not compile), no MonogDB support (source no more bundled with syslog-ng and no ready to use external package is available). Let me know if you miss either of these features. Available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng318epel6/ Installation instructions are available at https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… but note, that you need to change the version number from 3.14 to 3.18 in the examples. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Unofficial syslog-ng-3.18 packages for Debian/Ubuntu
by Laszlo Budai 24 Oct '18

24 Oct '18

Hi, syslog-ng 3.18.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [armv7l is still building] * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 * and Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

parsing JSON logs just won't work
by Michael Niemand 18 Oct '18

18 Oct '18

Hi, I can’t get JSON parsing to work. I’ve consulted the documentation and Google but with no luck. I have an app, that puts out simple json log messages like: {"level":"error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"} All I want to do, is parse these 3 values and send them to a hosted Graylog cluster. Sending works, but the message gets inserted as application name: {"level" message: "error","message":"connection ended without disconnect receipt","timestamp":"2018-10-12T17:49:08.650Z"} it's almost like syslog-ng doesn't even interpret the file as json. I tried different variants but I am at my wits end now... This is my config (on the application host; it should send the logs directly to the logging cluster) @version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf" options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; source s_src { file( “/var/log/worker/error.log" flags(no-parse) ); }; template unitManagerTemplate { template("$(format-json --scope dot-nv-pairs) [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n"); }; destination ovhPaaSLogs { tcp("gra2.logs.ovh.com" port(6514), template(unitManagerTemplate), ts_format("iso"), tls(peer-verify("require-trusted") ca_dir("/etc/ssl/certs/")), keep-alive(yes), so_keepalive(yes), ); }; parser p_json { json-parser(prefix(".json.")); }; log { source(s_src); parser(p_json); destination(ovhPaaSLogs); }; @include "/etc/syslog-ng/conf.d/" I also tried a different a template variant like this: template("${.json.level} ${.json.message} ${.json.timestamp} [sdid@123456 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n”); I also tried parsing the messages as text: template("{\"level\":\"${PRIORITY}\",\"message\":\"${MSG}\",\"timestamp\":\"${ISODATE}\"} - [sdid@32473 X-OVH-TOKEN=\"XXXXXXXXXXXXXXXXXXXXXXXXXX\" pid=\"${PID}\" facility=\"${FACILITY}\" priority=\"${PRIORITY}\"] ${MSG}\n"); What shows up in Graylog is absolutely identical (like described in the beginning). In fact, every variant that I tried changed absolutely nothing. The conf.d folder is empty though. I’d appreciate any help! Best regards, Michael

3 3

Insider 2018-10: 3.18 release; Splunk .conf18; Python destination; Python parser;
by Czanik, Péter 18 Oct '18

18 Oct '18

Dear syslog-ng users, This is the 70th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.18 of syslog-ng released ------------------------------------- Version 3.18 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to batching and multi-threading. The new Python source allows you to create source drivers for syslog-ng in Python. Learn about the small, but important changes at https://www.syslog-ng.com/community/b/blog/posts/small-is-beautiful-what-s-… or read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1. syslog-ng at Splunk .conf18 --------------------------- Balázs Scheidler, original author of syslog-ng, gave a talk at the annual Splunk conference. His talk about simplifying complex log data ingestion with syslog-ng is now available on-line. Learn about the new features coming up in syslog-ng to make it an even better choice to feed Splunk with log data: https://conf.splunk.com/files/2018/recordings/simplifying-complex-log-data-… Python destination ------------------ You can send your log messages to many different destinations using syslog-ng, but of course not everywhere. This is where the Python destination of syslog-ng can come handy. You can extend syslog-ng easily with your own code written in Python and store your log messages to the destination of your choice. Learn how to develop your own Python code for syslog-ng from our blog series, starting with the very basics and showing a complete working MQTT destination implemented in Python at the end: https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting… https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting… https://www.syslog-ng.com/community/b/blog/posts/writing-python-destination… Python parser ------------- The Python parser of syslog-ng not only enables you to parse any type of log message, but you can also use it to enrich messages. From this blog you will learn how to extract information from a specially formatted log message, and how to create new name-value pairs by consulting external databases about data contained in your log messages. I will show these using two simple parsers: one resolves host names from IP addresses, the other one uses regular expressions to parse synthetic log messages from the loggen utility. https://www.syslog-ng.com/community/b/blog/posts/parsing-log-messages-with-… WEBINARS You can watch our past webinars: * Supercharge your syslog-ng parsing: https://www.brighttalk.com/webcast/16207/330198 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

OSE 3.18.1
by Nemes, Antal 17 Oct '18

17 Oct '18

3.18.1 Features - HTTP batching (#2135 <https://github.com/balabit/syslog-ng/pull/2135>) - HTTP multi threading (#2291 <https://github.com/balabit/syslog-ng/pull/2291>) - HTTP framing support for batches (#2190 <https://github.com/balabit/syslog-ng/pull/2190>) - Python source (#2308 <https://github.com/balabit/syslog-ng/pull/2308>) - Template support for Python destination (#2196 <https://github.com/balabit/syslog-ng/pull/2196>) - HDFS time-reap (#2257 <https://github.com/balabit/syslog-ng/pull/2257>) - Riemann batching (#2098 <https://github.com/balabit/syslog-ng/pull/2098>) - flush-timeout() for HTTP (#2251 <https://github.com/balabit/syslog-ng/pull/2251>) - Base64 encoding template function (#2223 <https://github.com/balabit/syslog-ng/pull/2223>) - url-decode() template function (#2278 <https://github.com/balabit/syslog-ng/pull/2278>) - Allow IPv4 mapped addresses in IPv6 mode (#2312 <https://github.com/balabit/syslog-ng/pull/2312>) - app-parser: set ${.app.name} to the application identified (#2290 <https://github.com/balabit/syslog-ng/pull/2290>) - Value pairs improvements (#2287 <https://github.com/balabit/syslog-ng/pull/2287>) - syslog-ng-ctl config to print pre-processed configuration (#2280 <https://github.com/balabit/syslog-ng/pull/2280>) - Add support for whitespace in template functions (#2271 <https://github.com/balabit/syslog-ng/pull/2271>) - Verify the configuration is the same as on the disk (#2345 <https://github.com/balabit/syslog-ng/pull/2345>) Bugfixes - Fix queue counters race condition (#2316 <https://github.com/balabit/syslog-ng/pull/2316>) - AMQP configurable max connection and frame size (#2343 <https://github.com/balabit/syslog-ng/pull/2343>) - Fix wakeup in threaded sources (#2339 <https://github.com/balabit/syslog-ng/pull/2339>) - Fix libnet memory leak in network destinations (#2331 <https://github.com/balabit/syslog-ng/pull/2331>) - Fix unexpected flag check-hostname in syslog-parser (#2314 <https://github.com/balabit/syslog-ng/pull/2314>) - Fix memory leak in dbparser (#2311 <https://github.com/balabit/syslog-ng/pull/2311>) - Inline Python code comment generates syntax error (#2319 <https://github.com/balabit/syslog-ng/pull/2319>) - Password protected SSL keys portability (MADV_DONTDUMP) (#2341 <https://github.com/balabit/syslog-ng/pull/2341>) - Fixing compiler warnings from armv7l (#2301 <https://github.com/balabit/syslog-ng/pull/2301>) - Riemann crashes in flush (#2296 <https://github.com/balabit/syslog-ng/pull/2296>) - Revert sticky hook option (#2295 <https://github.com/balabit/syslog-ng/pull/2295>) - Small stats fixes (#2294 <https://github.com/balabit/syslog-ng/pull/2294>) - Detect filter loop (#2283 <https://github.com/balabit/syslog-ng/pull/2283>, #2288 <https://github.com/balabit/syslog-ng/pull/2288>) - Fix infinite loop in threaded destinations with ivykis prior 0.39 ( #2275 <https://github.com/balabit/syslog-ng/pull/2275>) - Fix log expr node use after free (#2269 <https://github.com/balabit/syslog-ng/pull/2269>) - Remove gradle from the list of "BuildRequires" (RPM packaging) (#2266 <https://github.com/balabit/syslog-ng/pull/2266>) - Fix wildcard-source memleak when directory removed (#2261 <https://github.com/balabit/syslog-ng/pull/2261>, #2267 <https://github.com/balabit/syslog-ng/pull/2267>) - Missing macros: C_AMPM, C_USEC, C_MSEC, C_HOUR12 (#2259 <https://github.com/balabit/syslog-ng/pull/2259>) - Fix cisco timestamp parsing (#2272 <https://github.com/balabit/syslog-ng/pull/2272>) - Undefined filter reference (#2273 <https://github.com/balabit/syslog-ng/pull/2273>) - Fix the literal() type hint (#2286 <https://github.com/balabit/syslog-ng/pull/2286>) - logwriter, affile, afsocket: fixing "internal overflow". (#2250 <https://github.com/balabit/syslog-ng/pull/2250>) - lib/gsockaddr.c: modify the unix salen calculation (#2285 <https://github.com/balabit/syslog-ng/pull/2285>) - dbld: fix dbld/rules deb failure (#2282 <https://github.com/balabit/syslog-ng/pull/2282>) - Update data type to avoid conversion. (#2281 <https://github.com/balabit/syslog-ng/pull/2281>) - compat/getent: add support for platforms that lack the r versions ( #2244 <https://github.com/balabit/syslog-ng/pull/2244>) - Fix memory leak caused by saving stats counter to persist config (#2279 <https://github.com/balabit/syslog-ng/pull/2279>) - Hdfs: disable archive when append-enabled is configured (#2235 <https://github.com/balabit/syslog-ng/pull/2235>) - scl: add linux-audit() SCL to make files (#2230 <https://github.com/balabit/syslog-ng/pull/2230>) - DebianPackaging: Add linux-audit SCL to included dirs (#2254 <https://github.com/balabit/syslog-ng/pull/2254>) - cap_syslog capability detection (#2227 <https://github.com/balabit/syslog-ng/pull/2227>) - LogProto partial write (#2194 <https://github.com/balabit/syslog-ng/pull/2194>) Other changes - telegram, urlencode: api changes (#2299 <https://github.com/balabit/syslog-ng/pull/2299>) - python: include python2/3 in plugin description (#2337 <https://github.com/balabit/syslog-ng/pull/2337>) - Stats prepare for multiple queues per destdrv (#2302 <https://github.com/balabit/syslog-ng/pull/2302>) - Update deprecated use of tcp()/udp() to network(). This addresses #2322 <https://github.com/balabit/syslog-ng/issues/2322> (#2326 <https://github.com/balabit/syslog-ng/pull/2326>) - Message about not supported cap_syslog only at debug level (#2333 <https://github.com/balabit/syslog-ng/pull/2333>) - Few test leak fix (#2323 <https://github.com/balabit/syslog-ng/pull/2323>) - warning elimination: pointer arithmetics (#2305 <https://github.com/balabit/syslog-ng/pull/2305>) - templates: get rid off the args_lock (#2289 <https://github.com/balabit/syslog-ng/pull/2289>) - Ack tracker small refact (#2277 <https://github.com/balabit/syslog-ng/pull/2277>) - ElasticSearchDestination: Display deprecated warning message about us… (#2274 <https://github.com/balabit/syslog-ng/pull/2274>) - Improve Readme.md header structure (#2258 <https://github.com/balabit/syslog-ng/pull/2258>) - Rewrite json tests based on criterion (#2255 <https://github.com/balabit/syslog-ng/pull/2255>) - Rewrite dbparser tests based on criterion (#2252 <https://github.com/balabit/syslog-ng/pull/2252>) - Processed timestamp (#2243 <https://github.com/balabit/syslog-ng/pull/2243>) - msg parameters: remove last NULL parameter from msg macros (#2242 <https://github.com/balabit/syslog-ng/pull/2242>) - Fix threaded destination test cases (#2236 <https://github.com/balabit/syslog-ng/pull/2236>) - dbld: missing packages, deps changes (#2232 <https://github.com/balabit/syslog-ng/pull/2232>, #2332 <https://github.com/balabit/syslog-ng/pull/2332>, #2327 <https://github.com/balabit/syslog-ng/pull/2327>, #2260 <https://github.com/balabit/syslog-ng/pull/2260>, #2256 <https://github.com/balabit/syslog-ng/pull/2256>) - Use msg_trace() when emitting trace information (#2226 <https://github.com/balabit/syslog-ng/pull/2226>) Notes to the developers - Threaded source and fetcher (#2247 <https://github.com/balabit/syslog-ng/pull/2247>) - "Examples" module (#2248 <https://github.com/balabit/syslog-ng/pull/2248>) - Do not ship mongo-c-driver with syslog-ng (remove as submodule) (#2245 <https://github.com/balabit/syslog-ng/pull/2245>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, Gabor Nagy, Gergely Tonté, Laszlo Budai, Laszlo Szemere, László Várady, Maurice T. Meyer, Mahmoud Salama, Norbert Takacs, Peter Czanik, Peter Gyorko, Peter Kokai, Robert Fekete, Terez Nemes, Tibor Bodnar, Zoltan Pallagi, y-l-i. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.18.1 <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> balabit/syslog-ng <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. github.com Regards, Antal Nemes

1 0

syslog-ng 3.5
by Rodney Bizzell 15 Oct '18

15 Oct '18

I have install syslog-ng 3.5 and I am configuring it to send all logs to Graylog but once changes are made syslog-ng is failing. I change the address of graylog server to 0.0.0.0 @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); internal(); udp(ip(0.0.0.0) port(514)); }; source s_net { udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(514) max-connections(256)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_graylog { tcp("0.0.0.0" port (12201) spoof_sources(yes) ); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_net); destination(d_graylog); }; log { source(s_sys); filter(f_default); destination(d_graylog);}; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:

4 14