- syslog-ng - lists.balabit.hu

Using patterndb in syslog-ng
by Scot Needy 31 Aug '16

31 Aug '16

Can someone point me in the right direction on how to use update-patterndb for syslog-ng ? Downloaded current git syslog-ng-patterndb to /opt/syslog-ng/etc/patterndb.d/, but not sure how to load and test it. Am I doing something wrong ? bin/pdbtool merge -r --glob \*.pdb -D /opt/syslog-ng/etc/patterndb.d -p /opt/syslog-ng/etc/patterndb.xml [@ROOT] sbin/syslog-ng -f /opt/syslog-ng/etc/syslog-ng.conf [2016-08-31T00:55:54.978717] Non-numeric correlation state ID found, assuming a literal '@' character. To avoid confusion when using a literal '@' after a macro or template function, write '@@' in the template.; Template='${temp.su_username}(a)${temp.xn--su_tty}-e36c blah… blah.. [2016-08-31T00:55:54.978978] Non-numeric correlation state ID found, assuming a literal '@' character. To avoid confusion when using a literal '@' after a macro or template function, write '@@' in the template.; Template='${temp.sudo_username}@unknown’

2 2

Syslog-ng and External variables
by Robert Blayzor 30 Aug '16

30 Aug '16

OS: FreeBSD 10.3 Syslog-ng: 3.7.3 I am trying to reference/use a OS level environment variable in my Syslog-Ng configuration. Everything I’ve read and tried, does not appear to work. I have confirmed simple environment variables like HOSTNAME and PATH are there, but when trying to reference them using backticks, ie: `HOSTNAME` does not appear to work. According to the documentation, environment variables are imported. ie: “Tip The environmental variables of the host are automatically imported and can be used as global variables.” But this does not appear to be the case. When I try referencing a known environment variable I get: Error parsing rewrite expression, syntax error, unexpected KW_VALUE, expecting LL_IDENTIFIER or LL_STRING in /usr/local/etc/syslog-ng.conf at line 56, column 24: rewrite r_host { set(`HOSTNAME`, value("HOST")); }; Or is there another easier way to override the HOST in all messages to be that of the actual servers hostname. (and not things like localhost, 127.0.0.1, etc) TIA -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu

4 8

Re: [syslog-ng] Syslog-ng and External variables
by Scheidler, Balázs 29 Aug '16

29 Aug '16

Only at config parse time. Regarding the back tick case, are you sure HOSTNAME is set for the syslog-ng process itself? Or is it only set for your interactive shell. On Linux you can check this in proc, /proc/<pid>/env, but I guess similar exists on freebsd. On Aug 29, 2016 6:17 PM, "Robert Blayzor" <rblayzor.bulk(a)inoc.net> wrote: On Aug 29, 2016, at 11:38 AM, Evan Rempel <erempel(a)uvic.ca> wrote: > > @module confgen context(rewrite) name(r_myself) exec("/bin/hostname") I guess the only remaining question on this is a performance one. Would the exec be done for each and every log message or is it done only once? -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Which version of syslog-ng support TLS1.2?
by Mark Mulligan 24 Aug '16

24 Aug '16

We are using using syslog-ng 3.2.5 and I and trying to figure out how to configure it to use TLS1.2 rather than the TLS1.1. It is working for TLS1.1 but not for connections using TLS1.2, this is a snippet from our config file. source s_waf { tcp(ip(xx.xx.xx.xx) port(xxxx) tls( key_file("/opt/syslog-ng/etc/certs/mux/privateKey.key") cert_file("/opt/syslog-ng/etc/certs/mux/splcollect-01.pem")) ); }; What versions of syslog-ng supports TLS1.2? The version we are using was the latest RH rpm I could find syslog-ng-3.2.5-4.el6.x86_64.rpm I would rather not compile it as it looks like most of the dependencies are also not rpms. Mark Mulligan Assoc Systems Engineer LL Bean Inc. Ext 25077 Cell 249-3823

3 2

Re: [syslog-ng] Elasticsearch destination and date macros
by Fabien Wernli 22 Aug '16

22 Aug '16

On Mon, Aug 22, 2016 at 06:00:29PM +0000, Adam Carter wrote: > I followed a couple of guides to set it up originally, now clearly from old versions. The (latest) online documentation lead me down the right path - thanks for the pointer great to hear that the documentation still does its job so well :-)

1 0

Elasticsearch destination and date macros
by Adam Carter 22 Aug '16

22 Aug '16

Hi All, I am having a problem getting the date macros (Year, Month, Day) to work in syslog-ng OSE when using the elasticsearch plugin. They work for file based outputs so I know the syntax is correct and being parsed correctly. I have now upgraded from syslog-ng 3.6 to syslog-ng 3.8.1, running on Ubuntu 16.04 but still the same symptoms. It is like the ${YEAR} is not being passed as a variable value For example I tried using the index name "syslog-ng-${YEAR}", starting syslog-ng with -Fevd: [2016-08-22T17:26:35.440602] Sending destination program a TERM signal; cmdline='/usr/share/syslog-ng/include/scl/elasticsearch/es-bridge localhost 9200 syslog-ng-${YEAR} syslog-ng', child_pid='12134' The index created in elasticsearch: yellow open syslog-ng- 5 1 239611 0 69.7mb 69.7mb Very simple syslog-ng.conf: destination d_elastic { elasticsearch(index("syslog-ng-${YEAR}") type("syslog-ng") ); }; log { source(s_netsyslog); destination(d_elastic); }; Is this a bug or have I missed something? Thanks

2 1

syslog-ng-3.8.1 has been released
by Laszlo Budai 20 Aug '16

20 Aug '16

3.8.1 Library updates * Kafka-client updated to version to 0.9.0.0 * Minimal required version of hiredis is set to 0.11.0 to avoid possible deadlocks * Minimal version of libdbi is set to 0.9.0 New dependencies * From now autoconf-archive package is a build-dependency. Improvements and features * Added the long-waited disk-buffer. * date-parser ported from incubator to upstream * New template functions: min, max, sum, average * Added Apache-accesslog-parser * Added loggly destination * Added logmatic destination * Added template function for supporting CEF. * cURL-based HTTP destination driver added (implemented in C programming language) * SELinux policy installer script now has support for Red Hat Enterprise Linux/CentOS/ Oracle Linux 5, 6 and 7. * Implemented add-contextual-data: With add-context-data syslog-ng can use an external database file to append custom name-value pairs on incoming logs (to enrich messages). The 'database' is actually a file that containing <selector, name, value> records. Currently only CSVformat is supported. It is like geoip parser where the selector is $HOST, but in this case, the user can define the selector, and also the database contents. Drivers * Program destination/source drivers * Added inherit-environment configuration option to program source and destination. When it is set to true then the process will inherit the entire environment of the parent process. * Added keep-alive option to program destination (afprog). This option will control whether the destination program should be terminated at reload, or should be left running. * Java drivers * HTTP destination * Added the ability to use templates in both url and message. * ElasticSearch Destination driver : * Support 2.2.x series of ElasticSearch (transport and node mode) . * Support Shield plugin for both ElasticSearch 1.x and ElasticSearch 2.x . * Implemented new mode (HTTP) that can work with ElasticSearch 1.x, ElasticSearch 2.x, and even with Elastic 5. HTTP mode is based on a Java HTTP Rest client (Jest :https://github.com/searchbox-io/Jest). Note: make install will copy Jest library to the syslog-ng install directory. * MongoDB destination driver * Replaced submodule limongo-client with mongo-c-driver. * Additional support for previous syntax used by libmongo-client before we started using mongo-c-driver and its URI syntax exclusively. Note that these are plainly translated to a connection URI without much sanity checking or preserving their former semantic meaning. So various aspects of the MongoDB connection like health checks, retries, error reporting and synchronicity will still follow the slightly altered semantics of mongo-c-driver. * Riemann destination driver * Use cert-file() and key-file() options to match afsocket keywords as the same way as afsocket drivers use these options. The old one still work though. Rewrite rules * Introduced template options in rewrite rules. * Added unset operation to make it possible to unset a specific name-value pair for a logmessage. Parsers * kvformat: make it possible to specify name-value separator * linux-audit-scanner: recognize a0-a9* as fields to be decoded Argument lists are encoded in a0, a1, ... fields that can potentially be hex-encoded. * csv-parser has been refactored, extended with new dialect and prefix options. Dialect is to convey CSV format information, instead of using flags Prefix option gets prefixed to all column names, just like with other parsers. PatternDB * added groupingby() parser that can perform simple correlation on log messages. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first always operates on a enumerable list of records, whereas groupingby() works on a stream of data. A few use-cases where this can be useful: * Linux audit logs * postfix logs * added create-context action Added a new possible action in the element, to create a new correlation context out of the current message and its associated context. This can be used to "split" a state. * Added NLSTRING parser that captures a string until the following end-of-line. It can be used in patterns as: @NLSTRING:value@ It doesn't expect any additional parameters. This makes it pretty easy to parse multi-line Windows logs. Miscellaneous features * syslog-debun (debug bundle script for syslog-ng) has been improved Bugfixes * geoip-parser: When default database if not specified, syslog-ng crashed. * Added support for multiple drivers with the same name in syslog-ng config. * Fixed aack counting logic for junctions that have branches that modify the LogMessage. * Fixed a potential crash for code that uses log_msg_clear() in production (e.g. syslog-parser()). * Fixed potential crash in reload logic * system(): use string comparison instead of numeric in PID rewrite The meaning of the != operator has been fixed to refer to numeric comparison in @version<https://github.com/version>: 3.8, so make sure we are using string comparison. * Support encoding on glib compiled with libiconv * pdbtool: Fix the ordering of the debug-info list in PatternDB * afprog: Don't kill our own process group If, for some reason, the program source/destination failed to set up its own process group, we need to make sure we do not run killpg() on that process group, as it would kill ourselves. * Handle option names with hyphen (-) characters in java scls * dnscache performance improved Instead of getting rid off the per-thread DNSCache when a worker thread exits, store them in a linked list and acquire them as a new thread starts. The set of cached hostnames are valuable as worker threads come and go (they exit after 10seconds of inactivity), but without this reusing of cache instances, our DNS cache is filled again and again. * Fixed IPv6 parser in patterndb. * Fixed journald program name flapping * Fixed create-dirs() inheritance in file destinations * Fixed pass-unix-credentials() global inheritance in afunix The global pass-unix-credentialsoption was not inherited in afunix-source if the options{}; block was positioned lower in the configuration file than the given module declaration. * Fixed create-dirs() global inheritance in afunix When the global create-dirs option was set toyes, the local one was ignored. * Fixed byteorder handling on bigendian systems in netmask6 filter * Fixed flow-control issue when overflow queue is full (suspending source by setting the window size to 0). * Log HTTP response error codes in HTTPDestination (Java). * Fixed potential leaks related $(sanitize) argument parsing in basicfuncs. * Fixed a memory leak in python debugger * Fixed a use-after-free bug in templates. * Fixed a memory leak around reload in netmask6 filter. * Fixed a memory leak in LogProtoBufferedServer in case the encoding() option is used. * configure: don't override $enable_python while executing pkg-config * Fixed BSD timestamp parsing in syslog-format. * Fixed a SIGPIPE bug in program destination. * Error handling has been improved in AMQP destination. * value-pairs performance improvements, memleak fixes * Various issues around UTF-8 support fixed. * Fixed integer overflow in numerical operations template function * Fixed an integer underflow in afsocket. * Fixed numerical comperisons issues around filters. There's a problem in straight fixing this issue though: anyone who used the numeric operators erroneously will have their behaviour changed, therefore this patch also adds a configuration update warning in case someone is using the wrong syntax. * Fixed kernel log message time drift on Linux. * Take CRLF sequences equivalent to an LF in patterndb. Windows logs contain embedded CRLFs which is difficult to match against from db-parser(), as we use a UNIX text file to store the patterns. Also, the fact that the input contains CRLF whereas our patterns only contain an LF makes it a very unintuitive non-match, which is difficult to debug. * When syslog-ng failed to insert data into Redis, it has crashed. * When device file is set as a file destination then syslog-ng will not try to change the permission of the device file. * Various fixes around config file parsing: * in some circumstances syslog-ng crashed when the config file contained non-readable characters * fixed a memory leak * fixed memory leak around backtick substitution Notes to the Developers * copyright cleanup in source tree * install tools and scl under a syslog-ng specific subdirectory These should never be installed in /usr/share directly, but rather under a subdirectory and as described in https://www.gnu.org/prep/standards/html_node/Directory-Variables.html we should do that right within the source and not rely on packaging tools to do it for us.This will trigger a required change in packaging scripts to avoid changing the --datadir, as the default of /usr/share should work out-of-the-box. * Support for native-lanugage (compiled languages, like Rust) bindings. These bindings just forward the calls to the native side. This whole module compiles into a static library (libsyslog-ng-native-connector.a) which is linked to all external native modules. A native module defines the required functions (like native_parser_proxy_new()) so those symbols will be resolved. Some symbols have the visibility(hidden) attribute applied to them. Those symbols are defined by the other half of the native bindings, we only need their signature here. They are hidden because their definition is contained in other source files but we would like to keep them as "library private". If they are exported, the dynamic linker will resolve them when a module is loaded, therefore every module would be mapped to the first loaded one. It is best to hide everything in this native connector. * Support system librabbitmq-c (AMQP destination) Currently only the internal librabbitmq-c is supported, if we want to use the preexisting library, the configuration will fail. This change is required if we want to get rid of the internal libraries. * Added check-valgrind target to Makefile * Remove dependency on Gradle for building Java language bindings (not modules, just the language binding) * Experimental CMake support added * Experimental OSX support added * Improved Travis build matrix * Added plugin skeleton creator. * Debian packaging improved: Debian packaging from unofficial OBS repository has been ported to upstream. From now, someone could easily build debian packages from upstream source. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Istvan Mozes, Andras Mitzki, Arnaud Vamorec, Balazs Scheidler, David Schweikert, Fabien Wernli, Flavio Medeiros, Hanno Böck, Henrik Grindal Bakken, Gergo Nagy, Gyorgy Pasztor, Laszlo Budai, Laszlo Varady, Marc Falzon, Noemi Vanyi, Peter Czanik, Robert Fekete, Tamas Nagy,Tibor Benke, Viktor Juhasz, Viktor Tusa, Vincent Bernat, Zdenek Styblik, Zoltan Fried, Zoltan Pallagi, Yilin Li https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1 https://github.com/balabit/syslog-ng/releases/download/syslog-ng-3.8.1/sysl… <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1> regards, Laszlo Budai

2 1

Disabling SSLv3 with Syslog-NG
by Lupo, Joseph 19 Aug '16

19 Aug '16

The syslog-ng documentation is very unclear. Where would we put the ssl-options(no-sslv2, no-sslv3, no-tlsv1) entry for disabling sslv2, sslv3 and tlsv1? The following works for enabling TLS on our host, but I can’t figure out where to insert the ssl-options. source s_net { network(ip('<our IP>') port(<our port>) transport("tls") tls( key-file("/usr/local/etc/hostcert.key") cert-file("/usr/local/etc/hostcert.pem") peer_verify(optional-untrusted)) ); }; Joe Lupo T-Mobile USA Principal Engineer, System Design & Strategy (973) 440-8768

4 8

syslog-ng Open Source Edition 3.8 Administrator Guide - initial release
by Fekete, Róbert 19 Aug '16

19 Aug '16

Hi, You can find the Administrator Guide for version 3.8 of syslog-ng Open Source Edition at https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… Please note that it is a draft version, as it does not cover every new feature of 3.8 yet. You can find a (possibly incomplete) list of what is already covered below. Any comment, correction, or feedback is most welcome either here on the list, or at https://github.com/balabit/syslog-ng-ose-guides (Better, real-life configuration examples would be especially useful.) I will release updates to the docs ~daily next week, and hope to cover most of the new features by the end of next week. * grouping-by parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * elastic2 destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Apache Access Log parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * date-parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * libcurl http-destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Loggly destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Logmatic destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… Kind Regards, Robert

1 0

3.8 RHEL 6 packaging
by Czanik, Péter 19 Aug '16

19 Aug '16

Hi, I think, that RHEL / CentOS 6 is ancient, but still I ask: is there anyone who needs syslog-ng 3.8 packaged for this old Linux release? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

3 5