- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Elasticsearch destination and date macros
by Fabien Wernli 22 Aug '16

22 Aug '16

On Mon, Aug 22, 2016 at 06:00:29PM +0000, Adam Carter wrote: > I followed a couple of guides to set it up originally, now clearly from old versions. The (latest) online documentation lead me down the right path - thanks for the pointer great to hear that the documentation still does its job so well :-)

1 0

Elasticsearch destination and date macros
by Adam Carter 22 Aug '16

22 Aug '16

Hi All, I am having a problem getting the date macros (Year, Month, Day) to work in syslog-ng OSE when using the elasticsearch plugin. They work for file based outputs so I know the syntax is correct and being parsed correctly. I have now upgraded from syslog-ng 3.6 to syslog-ng 3.8.1, running on Ubuntu 16.04 but still the same symptoms. It is like the ${YEAR} is not being passed as a variable value For example I tried using the index name "syslog-ng-${YEAR}", starting syslog-ng with -Fevd: [2016-08-22T17:26:35.440602] Sending destination program a TERM signal; cmdline='/usr/share/syslog-ng/include/scl/elasticsearch/es-bridge localhost 9200 syslog-ng-${YEAR} syslog-ng', child_pid='12134' The index created in elasticsearch: yellow open syslog-ng- 5 1 239611 0 69.7mb 69.7mb Very simple syslog-ng.conf: destination d_elastic { elasticsearch(index("syslog-ng-${YEAR}") type("syslog-ng") ); }; log { source(s_netsyslog); destination(d_elastic); }; Is this a bug or have I missed something? Thanks

2 1

syslog-ng-3.8.1 has been released
by Laszlo Budai 20 Aug '16

20 Aug '16

3.8.1 Library updates * Kafka-client updated to version to 0.9.0.0 * Minimal required version of hiredis is set to 0.11.0 to avoid possible deadlocks * Minimal version of libdbi is set to 0.9.0 New dependencies * From now autoconf-archive package is a build-dependency. Improvements and features * Added the long-waited disk-buffer. * date-parser ported from incubator to upstream * New template functions: min, max, sum, average * Added Apache-accesslog-parser * Added loggly destination * Added logmatic destination * Added template function for supporting CEF. * cURL-based HTTP destination driver added (implemented in C programming language) * SELinux policy installer script now has support for Red Hat Enterprise Linux/CentOS/ Oracle Linux 5, 6 and 7. * Implemented add-contextual-data: With add-context-data syslog-ng can use an external database file to append custom name-value pairs on incoming logs (to enrich messages). The 'database' is actually a file that containing <selector, name, value> records. Currently only CSVformat is supported. It is like geoip parser where the selector is $HOST, but in this case, the user can define the selector, and also the database contents. Drivers * Program destination/source drivers * Added inherit-environment configuration option to program source and destination. When it is set to true then the process will inherit the entire environment of the parent process. * Added keep-alive option to program destination (afprog). This option will control whether the destination program should be terminated at reload, or should be left running. * Java drivers * HTTP destination * Added the ability to use templates in both url and message. * ElasticSearch Destination driver : * Support 2.2.x series of ElasticSearch (transport and node mode) . * Support Shield plugin for both ElasticSearch 1.x and ElasticSearch 2.x . * Implemented new mode (HTTP) that can work with ElasticSearch 1.x, ElasticSearch 2.x, and even with Elastic 5. HTTP mode is based on a Java HTTP Rest client (Jest :https://github.com/searchbox-io/Jest). Note: make install will copy Jest library to the syslog-ng install directory. * MongoDB destination driver * Replaced submodule limongo-client with mongo-c-driver. * Additional support for previous syntax used by libmongo-client before we started using mongo-c-driver and its URI syntax exclusively. Note that these are plainly translated to a connection URI without much sanity checking or preserving their former semantic meaning. So various aspects of the MongoDB connection like health checks, retries, error reporting and synchronicity will still follow the slightly altered semantics of mongo-c-driver. * Riemann destination driver * Use cert-file() and key-file() options to match afsocket keywords as the same way as afsocket drivers use these options. The old one still work though. Rewrite rules * Introduced template options in rewrite rules. * Added unset operation to make it possible to unset a specific name-value pair for a logmessage. Parsers * kvformat: make it possible to specify name-value separator * linux-audit-scanner: recognize a0-a9* as fields to be decoded Argument lists are encoded in a0, a1, ... fields that can potentially be hex-encoded. * csv-parser has been refactored, extended with new dialect and prefix options. Dialect is to convey CSV format information, instead of using flags Prefix option gets prefixed to all column names, just like with other parsers. PatternDB * added groupingby() parser that can perform simple correlation on log messages. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first always operates on a enumerable list of records, whereas groupingby() works on a stream of data. A few use-cases where this can be useful: * Linux audit logs * postfix logs * added create-context action Added a new possible action in the element, to create a new correlation context out of the current message and its associated context. This can be used to "split" a state. * Added NLSTRING parser that captures a string until the following end-of-line. It can be used in patterns as: @NLSTRING:value@ It doesn't expect any additional parameters. This makes it pretty easy to parse multi-line Windows logs. Miscellaneous features * syslog-debun (debug bundle script for syslog-ng) has been improved Bugfixes * geoip-parser: When default database if not specified, syslog-ng crashed. * Added support for multiple drivers with the same name in syslog-ng config. * Fixed aack counting logic for junctions that have branches that modify the LogMessage. * Fixed a potential crash for code that uses log_msg_clear() in production (e.g. syslog-parser()). * Fixed potential crash in reload logic * system(): use string comparison instead of numeric in PID rewrite The meaning of the != operator has been fixed to refer to numeric comparison in @version<https://github.com/version>: 3.8, so make sure we are using string comparison. * Support encoding on glib compiled with libiconv * pdbtool: Fix the ordering of the debug-info list in PatternDB * afprog: Don't kill our own process group If, for some reason, the program source/destination failed to set up its own process group, we need to make sure we do not run killpg() on that process group, as it would kill ourselves. * Handle option names with hyphen (-) characters in java scls * dnscache performance improved Instead of getting rid off the per-thread DNSCache when a worker thread exits, store them in a linked list and acquire them as a new thread starts. The set of cached hostnames are valuable as worker threads come and go (they exit after 10seconds of inactivity), but without this reusing of cache instances, our DNS cache is filled again and again. * Fixed IPv6 parser in patterndb. * Fixed journald program name flapping * Fixed create-dirs() inheritance in file destinations * Fixed pass-unix-credentials() global inheritance in afunix The global pass-unix-credentialsoption was not inherited in afunix-source if the options{}; block was positioned lower in the configuration file than the given module declaration. * Fixed create-dirs() global inheritance in afunix When the global create-dirs option was set toyes, the local one was ignored. * Fixed byteorder handling on bigendian systems in netmask6 filter * Fixed flow-control issue when overflow queue is full (suspending source by setting the window size to 0). * Log HTTP response error codes in HTTPDestination (Java). * Fixed potential leaks related $(sanitize) argument parsing in basicfuncs. * Fixed a memory leak in python debugger * Fixed a use-after-free bug in templates. * Fixed a memory leak around reload in netmask6 filter. * Fixed a memory leak in LogProtoBufferedServer in case the encoding() option is used. * configure: don't override $enable_python while executing pkg-config * Fixed BSD timestamp parsing in syslog-format. * Fixed a SIGPIPE bug in program destination. * Error handling has been improved in AMQP destination. * value-pairs performance improvements, memleak fixes * Various issues around UTF-8 support fixed. * Fixed integer overflow in numerical operations template function * Fixed an integer underflow in afsocket. * Fixed numerical comperisons issues around filters. There's a problem in straight fixing this issue though: anyone who used the numeric operators erroneously will have their behaviour changed, therefore this patch also adds a configuration update warning in case someone is using the wrong syntax. * Fixed kernel log message time drift on Linux. * Take CRLF sequences equivalent to an LF in patterndb. Windows logs contain embedded CRLFs which is difficult to match against from db-parser(), as we use a UNIX text file to store the patterns. Also, the fact that the input contains CRLF whereas our patterns only contain an LF makes it a very unintuitive non-match, which is difficult to debug. * When syslog-ng failed to insert data into Redis, it has crashed. * When device file is set as a file destination then syslog-ng will not try to change the permission of the device file. * Various fixes around config file parsing: * in some circumstances syslog-ng crashed when the config file contained non-readable characters * fixed a memory leak * fixed memory leak around backtick substitution Notes to the Developers * copyright cleanup in source tree * install tools and scl under a syslog-ng specific subdirectory These should never be installed in /usr/share directly, but rather under a subdirectory and as described in https://www.gnu.org/prep/standards/html_node/Directory-Variables.html we should do that right within the source and not rely on packaging tools to do it for us.This will trigger a required change in packaging scripts to avoid changing the --datadir, as the default of /usr/share should work out-of-the-box. * Support for native-lanugage (compiled languages, like Rust) bindings. These bindings just forward the calls to the native side. This whole module compiles into a static library (libsyslog-ng-native-connector.a) which is linked to all external native modules. A native module defines the required functions (like native_parser_proxy_new()) so those symbols will be resolved. Some symbols have the visibility(hidden) attribute applied to them. Those symbols are defined by the other half of the native bindings, we only need their signature here. They are hidden because their definition is contained in other source files but we would like to keep them as "library private". If they are exported, the dynamic linker will resolve them when a module is loaded, therefore every module would be mapped to the first loaded one. It is best to hide everything in this native connector. * Support system librabbitmq-c (AMQP destination) Currently only the internal librabbitmq-c is supported, if we want to use the preexisting library, the configuration will fail. This change is required if we want to get rid of the internal libraries. * Added check-valgrind target to Makefile * Remove dependency on Gradle for building Java language bindings (not modules, just the language binding) * Experimental CMake support added * Experimental OSX support added * Improved Travis build matrix * Added plugin skeleton creator. * Debian packaging improved: Debian packaging from unofficial OBS repository has been ported to upstream. From now, someone could easily build debian packages from upstream source. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Istvan Mozes, Andras Mitzki, Arnaud Vamorec, Balazs Scheidler, David Schweikert, Fabien Wernli, Flavio Medeiros, Hanno Böck, Henrik Grindal Bakken, Gergo Nagy, Gyorgy Pasztor, Laszlo Budai, Laszlo Varady, Marc Falzon, Noemi Vanyi, Peter Czanik, Robert Fekete, Tamas Nagy,Tibor Benke, Viktor Juhasz, Viktor Tusa, Vincent Bernat, Zdenek Styblik, Zoltan Fried, Zoltan Pallagi, Yilin Li https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1 https://github.com/balabit/syslog-ng/releases/download/syslog-ng-3.8.1/sysl… <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1> regards, Laszlo Budai

2 1

Disabling SSLv3 with Syslog-NG
by Lupo, Joseph 19 Aug '16

19 Aug '16

The syslog-ng documentation is very unclear. Where would we put the ssl-options(no-sslv2, no-sslv3, no-tlsv1) entry for disabling sslv2, sslv3 and tlsv1? The following works for enabling TLS on our host, but I can’t figure out where to insert the ssl-options. source s_net { network(ip('<our IP>') port(<our port>) transport("tls") tls( key-file("/usr/local/etc/hostcert.key") cert-file("/usr/local/etc/hostcert.pem") peer_verify(optional-untrusted)) ); }; Joe Lupo T-Mobile USA Principal Engineer, System Design & Strategy (973) 440-8768

4 8

syslog-ng Open Source Edition 3.8 Administrator Guide - initial release
by Fekete, Róbert 19 Aug '16

19 Aug '16

Hi, You can find the Administrator Guide for version 3.8 of syslog-ng Open Source Edition at https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… Please note that it is a draft version, as it does not cover every new feature of 3.8 yet. You can find a (possibly incomplete) list of what is already covered below. Any comment, correction, or feedback is most welcome either here on the list, or at https://github.com/balabit/syslog-ng-ose-guides (Better, real-life configuration examples would be especially useful.) I will release updates to the docs ~daily next week, and hope to cover most of the new features by the end of next week. * grouping-by parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * elastic2 destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Apache Access Log parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * date-parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * libcurl http-destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Loggly destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Logmatic destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… Kind Regards, Robert

1 0

3.8 RHEL 6 packaging
by Czanik, Péter 19 Aug '16

19 Aug '16

Hi, I think, that RHEL / CentOS 6 is ancient, but still I ask: is there anyone who needs syslog-ng 3.8 packaged for this old Linux release? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

3 5

Unofficial Debian/Ubuntu packages for syslog-ng-3.8.1
by Laszlo Budai 19 Aug '16

19 Aug '16

Hi, syslog-ng 3.8.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 8.0 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 (Debian 7 is not supported by OBS anymore. Precise: I haven't created packages yet... I'm thinking about dropping precise support... if someone would need it, just send a mail to me.) Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` You can replace Debian_8.0 to * xUbuntu_14.04 * xUbuntu_14.10 * xUbuntu_15.04 * xUbuntu_15.10 * xUbuntu 16.04 Available packages: syslog-ng syslog-ng-core syslog-ng-dbg syslog-ng-dev syslog-ng-mod-add-contextual-data syslog-ng-mod-amqp syslog-ng-mod-curl syslog-ng-mod-elastic syslog-ng-mod-geoip syslog-ng-mod-graphite syslog-ng-mod-hdfs syslog-ng-mod-http syslog-ng-mod-java-common-lib syslog-ng-mod-java syslog-ng-mod-journal syslog-ng-mod-json syslog-ng-mod-kafka syslog-ng-mod-kvformat syslog-ng-mod-mongodb syslog-ng-mod-python syslog-ng-mod-redis syslog-ng-mod-riemann syslog-ng-mod-smtp syslog-ng-mod-sql syslog-ng-mod-stomp Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

Syslog-ng client through a load balancer with SSL/TLS encryption
by Lupo, Joseph 18 Aug '16

18 Aug '16

I am trying to setup Syslog-ng to relay messages from one syslog server to another with a load balancer in between. I am also using TLS encryption. The issue I’m having right now is that when the client intiates the connection, it seems to lock on to a particular back end syslog server and send all of its messages there instead of switching off to another one. On its own this isn’t a big problem except that if that system goes down, the client doesn’t seem to be aware. I also haven’t found a good way to force syslog-ng to close and re-establish its connections without fully shutting down the relay system. We currently have no persistence setup on the load balancer. Is there a way to tell the relay server to periodically reconnect? Maybe send a certain amount of messages or data before reconnecting so that the data is balanced across the backend syslog-ng servers? Also, is there a better way to have the relay system learn about the remote server going offline so it can immediately reset its connection?

4 4

compression during tls
by thejaguar＠tutanota.de 17 Aug '16

17 Aug '16

Hi all, According to this https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… Allow-compression() should be a working option under tls. I did google on this a lot and found no config snippetwhich shows allow-compress() to be working and to my surprise when I tried toinclude it in my config I get syntax error ___________________________________________________ 2016-08-11T15:27:19.538347] Registering candidate plugin;module='afsocket', context='source', name='systemd-syslog', preference='100' [2016-08-11T15:27:19.539190] Using /dev/log Unix socketwith systemd is not possible. Changing to systemd-syslog source, which supportssocket activation.; Error parsing afsocket, syntax error, unexpectedLL_IDENTIFIER, expecting ')' in /etc/syslog-ng/syslog-ng.conf at line 27,column 124: destination d_net_tls { network("syslog1.xxxxxxxxx.com" port(6514) transport("tls") tls(ca-dir("/etc/syslog-ng/ca") allow-compress(yes)peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) ^^^^^^^^^^^^^^ ___________________________________________________ I am using 3.8.0 on ubuntu 15.10. Is this even really supported as claimed in here https://www.balabit.com/network-security/syslog-ng/comparing/detailed # /usr/sbin/syslog-ng -V syslog-ng 3.8.0beta2 Installer-Version: 3.8.0beta2 Revision: 3.8.0beta2-1 Module-Directory: /usr/lib/syslog-ng/3.8 Module-Path: /usr/lib/syslog-ng/3.8 Available-Modules:affile,basicfuncs,system-source,cryptofuncs,pseudofile,afuser,csvparser,linux-kmsg-format,confgen,sdjournal,syslogformat,afprog,dbparser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off

2 3

Re: [syslog-ng] sylog-ng filters not working
by Christian Turner 03 Aug '16

03 Aug '16

@version: 3.2 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # https://www.balabit.com/support/documentation # @include "scl.conf" options { flush_lines (0); time_reopen (10); log_fifo_size (2048); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (no); stats_freq(86400); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); }; ### MYAPP Dev Logs ### ## DEVENV ## source src_devenv { udp(ip(0.0.0.0) port(514)); }; filter f_devenv_01ui { netmask(10.22.206.0/24); }; filter f_devenv_02gw { netmask(10.22.207.0/24); }; filter f_devenv_03api { netmask(10.22.208.0/24); }; filter f_devenv_04net { netmask( "10.22.209.0/24" ); }; filter f_devenv_05bat { netmask(10.22.210.0/24); }; destination d_devenv_01ui { file("/mnt/syslogng/MYAPPlogs/DEVENV/01ui-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_02gw { file("/mnt/syslogng/MYAPPlogs/DEVENV/02gw-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_03api { file("/mnt/syslogng/MYAPPlogs/DEVENV/03api-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_04net { file("/mnt/syslogng/MYAPPlogs/DEVENV/04net-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_05bat { file("/mnt/syslogng/MYAPPlogs/DEVENV/05bat-$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_devenv); filter(f_devenv_01ui); destination(d_devenv_01ui); }; log { source(src_devenv); filter(f_devenv_02gw); destination(d_devenv_02gw); }; log { source(src_devenv); filter(f_devenv_03api); destination(d_devenv_03api); }; log { source(src_devenv); filter(f_devenv_04net); destination(d_devenv_04net); flags(final); }; log { source(src_devenv); filter(f_devenv_05bat); destination(d_devenv_05bat); }; ## MYAPP ALL ## source src_MYAPP { udp(ip(0.0.0.0) port(514)); }; destination d_MYAPP { file("/mnt/syslogng/MYAPPlogs/$HOST/$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_MYAPP); destination(d_MYAPP); }; #source external { tcp(); }; #source external { udp(); }; #destination d_hosts { file("/home/syslog/$HOST/application.log" owner("syslog") group("syslog") perm(0600)); }; destination d_mesg { file("/var/log/messages"); }; #destination d_cons { file("/dev/console"); }; #destination d_auth { file("/var/log/secure"); }; #destination d_mail { file("/var/log/maillog" flush_lines(10)); }; #destination d_spol { file("/var/log/spooler"); }; #destination d_boot { file("/var/log/boot.log"); }; #destination d_cron { file("/var/log/cron"); }; #destination d_kern { file("/var/log/kern"); }; #destination d_mlal { usertty("*"); }; #destination d_all { file("/var/log/splunk"); }; log { source(s_sys); destination(d_mesg); }; #log { source(external); destination(d_hosts); }; From: Christian Turner Sent: Wednesday, August 3, 2016 11:53 AM To: 'syslog-ng(a)lists.balabit.hu' <syslog-ng(a)lists.balabit.hu> Subject: RE: sylog-ng filters not working Hi, I have the following filter configured; source src_devenv01 { udp(ip(0.0.0.0) port(514)); }; filter f_devenv01_04net { netmask(10.22.209.0/24); }; destination d_devenv_04net { file("/mnt/syslogng/p2alogs/DEVENV/04net-$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_devenv01); filter(f_devenv_04net); destination(d_devenv_04net); flags(final); }; However, the filter does not work, and the logs from this source all go to the generic logging destination. I perform an strace and I can see that the IP appears as expected, so I'm figuring I have a syntax error somewhere; [pid 28481] recvfrom(11, "<182>1 2016-08-03T10:27:50.645062-04:00 ::1 [[REDACTED]]..., 8192, 0, {sa_family=AF_INET, sin_port=htons(58785), sin_addr=inet_addr("10.22.209.10")}, [16]) = 265 Christian Turner

2 1