- syslog-ng - lists.balabit.hu

Using patterndb in syslog-ng
by Scot Needy 31 Aug '16

31 Aug '16

Can someone point me in the right direction on how to use update-patterndb for syslog-ng ? Downloaded current git syslog-ng-patterndb to /opt/syslog-ng/etc/patterndb.d/, but not sure how to load and test it. Am I doing something wrong ? bin/pdbtool merge -r --glob \*.pdb -D /opt/syslog-ng/etc/patterndb.d -p /opt/syslog-ng/etc/patterndb.xml [@ROOT] sbin/syslog-ng -f /opt/syslog-ng/etc/syslog-ng.conf [2016-08-31T00:55:54.978717] Non-numeric correlation state ID found, assuming a literal '@' character. To avoid confusion when using a literal '@' after a macro or template function, write '@@' in the template.; Template='${temp.su_username}(a)${temp.xn--su_tty}-e36c blah… blah.. [2016-08-31T00:55:54.978978] Non-numeric correlation state ID found, assuming a literal '@' character. To avoid confusion when using a literal '@' after a macro or template function, write '@@' in the template.; Template='${temp.sudo_username}@unknown’

2 2

Syslog-ng and External variables
by Robert Blayzor 30 Aug '16

30 Aug '16

OS: FreeBSD 10.3 Syslog-ng: 3.7.3 I am trying to reference/use a OS level environment variable in my Syslog-Ng configuration. Everything I’ve read and tried, does not appear to work. I have confirmed simple environment variables like HOSTNAME and PATH are there, but when trying to reference them using backticks, ie: `HOSTNAME` does not appear to work. According to the documentation, environment variables are imported. ie: “Tip The environmental variables of the host are automatically imported and can be used as global variables.” But this does not appear to be the case. When I try referencing a known environment variable I get: Error parsing rewrite expression, syntax error, unexpected KW_VALUE, expecting LL_IDENTIFIER or LL_STRING in /usr/local/etc/syslog-ng.conf at line 56, column 24: rewrite r_host { set(`HOSTNAME`, value("HOST")); }; Or is there another easier way to override the HOST in all messages to be that of the actual servers hostname. (and not things like localhost, 127.0.0.1, etc) TIA -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu

4 8

Re: [syslog-ng] Syslog-ng and External variables
by Scheidler, Balázs 29 Aug '16

29 Aug '16

Only at config parse time. Regarding the back tick case, are you sure HOSTNAME is set for the syslog-ng process itself? Or is it only set for your interactive shell. On Linux you can check this in proc, /proc/<pid>/env, but I guess similar exists on freebsd. On Aug 29, 2016 6:17 PM, "Robert Blayzor" <rblayzor.bulk(a)inoc.net> wrote: On Aug 29, 2016, at 11:38 AM, Evan Rempel <erempel(a)uvic.ca> wrote: > > @module confgen context(rewrite) name(r_myself) exec("/bin/hostname") I guess the only remaining question on this is a performance one. Would the exec be done for each and every log message or is it done only once? -- Robert inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP Key: 78BEDCE1 @ pgp.mit.edu ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Which version of syslog-ng support TLS1.2?
by Mark Mulligan 24 Aug '16

24 Aug '16

We are using using syslog-ng 3.2.5 and I and trying to figure out how to configure it to use TLS1.2 rather than the TLS1.1. It is working for TLS1.1 but not for connections using TLS1.2, this is a snippet from our config file. source s_waf { tcp(ip(xx.xx.xx.xx) port(xxxx) tls( key_file("/opt/syslog-ng/etc/certs/mux/privateKey.key") cert_file("/opt/syslog-ng/etc/certs/mux/splcollect-01.pem")) ); }; What versions of syslog-ng supports TLS1.2? The version we are using was the latest RH rpm I could find syslog-ng-3.2.5-4.el6.x86_64.rpm I would rather not compile it as it looks like most of the dependencies are also not rpms. Mark Mulligan Assoc Systems Engineer LL Bean Inc. Ext 25077 Cell 249-3823

3 2

Re: [syslog-ng] Elasticsearch destination and date macros
by Fabien Wernli 22 Aug '16

22 Aug '16

On Mon, Aug 22, 2016 at 06:00:29PM +0000, Adam Carter wrote: > I followed a couple of guides to set it up originally, now clearly from old versions. The (latest) online documentation lead me down the right path - thanks for the pointer great to hear that the documentation still does its job so well :-)

1 0

Elasticsearch destination and date macros
by Adam Carter 22 Aug '16

22 Aug '16

Hi All, I am having a problem getting the date macros (Year, Month, Day) to work in syslog-ng OSE when using the elasticsearch plugin. They work for file based outputs so I know the syntax is correct and being parsed correctly. I have now upgraded from syslog-ng 3.6 to syslog-ng 3.8.1, running on Ubuntu 16.04 but still the same symptoms. It is like the ${YEAR} is not being passed as a variable value For example I tried using the index name "syslog-ng-${YEAR}", starting syslog-ng with -Fevd: [2016-08-22T17:26:35.440602] Sending destination program a TERM signal; cmdline='/usr/share/syslog-ng/include/scl/elasticsearch/es-bridge localhost 9200 syslog-ng-${YEAR} syslog-ng', child_pid='12134' The index created in elasticsearch: yellow open syslog-ng- 5 1 239611 0 69.7mb 69.7mb Very simple syslog-ng.conf: destination d_elastic { elasticsearch(index("syslog-ng-${YEAR}") type("syslog-ng") ); }; log { source(s_netsyslog); destination(d_elastic); }; Is this a bug or have I missed something? Thanks

2 1

syslog-ng-3.8.1 has been released
by Laszlo Budai 20 Aug '16

20 Aug '16

3.8.1 Library updates * Kafka-client updated to version to 0.9.0.0 * Minimal required version of hiredis is set to 0.11.0 to avoid possible deadlocks * Minimal version of libdbi is set to 0.9.0 New dependencies * From now autoconf-archive package is a build-dependency. Improvements and features * Added the long-waited disk-buffer. * date-parser ported from incubator to upstream * New template functions: min, max, sum, average * Added Apache-accesslog-parser * Added loggly destination * Added logmatic destination * Added template function for supporting CEF. * cURL-based HTTP destination driver added (implemented in C programming language) * SELinux policy installer script now has support for Red Hat Enterprise Linux/CentOS/ Oracle Linux 5, 6 and 7. * Implemented add-contextual-data: With add-context-data syslog-ng can use an external database file to append custom name-value pairs on incoming logs (to enrich messages). The 'database' is actually a file that containing <selector, name, value> records. Currently only CSVformat is supported. It is like geoip parser where the selector is $HOST, but in this case, the user can define the selector, and also the database contents. Drivers * Program destination/source drivers * Added inherit-environment configuration option to program source and destination. When it is set to true then the process will inherit the entire environment of the parent process. * Added keep-alive option to program destination (afprog). This option will control whether the destination program should be terminated at reload, or should be left running. * Java drivers * HTTP destination * Added the ability to use templates in both url and message. * ElasticSearch Destination driver : * Support 2.2.x series of ElasticSearch (transport and node mode) . * Support Shield plugin for both ElasticSearch 1.x and ElasticSearch 2.x . * Implemented new mode (HTTP) that can work with ElasticSearch 1.x, ElasticSearch 2.x, and even with Elastic 5. HTTP mode is based on a Java HTTP Rest client (Jest :https://github.com/searchbox-io/Jest). Note: make install will copy Jest library to the syslog-ng install directory. * MongoDB destination driver * Replaced submodule limongo-client with mongo-c-driver. * Additional support for previous syntax used by libmongo-client before we started using mongo-c-driver and its URI syntax exclusively. Note that these are plainly translated to a connection URI without much sanity checking or preserving their former semantic meaning. So various aspects of the MongoDB connection like health checks, retries, error reporting and synchronicity will still follow the slightly altered semantics of mongo-c-driver. * Riemann destination driver * Use cert-file() and key-file() options to match afsocket keywords as the same way as afsocket drivers use these options. The old one still work though. Rewrite rules * Introduced template options in rewrite rules. * Added unset operation to make it possible to unset a specific name-value pair for a logmessage. Parsers * kvformat: make it possible to specify name-value separator * linux-audit-scanner: recognize a0-a9* as fields to be decoded Argument lists are encoded in a0, a1, ... fields that can potentially be hex-encoded. * csv-parser has been refactored, extended with new dialect and prefix options. Dialect is to convey CSV format information, instead of using flags Prefix option gets prefixed to all column names, just like with other parsers. PatternDB * added groupingby() parser that can perform simple correlation on log messages. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first always operates on a enumerable list of records, whereas groupingby() works on a stream of data. A few use-cases where this can be useful: * Linux audit logs * postfix logs * added create-context action Added a new possible action in the element, to create a new correlation context out of the current message and its associated context. This can be used to "split" a state. * Added NLSTRING parser that captures a string until the following end-of-line. It can be used in patterns as: @NLSTRING:value@ It doesn't expect any additional parameters. This makes it pretty easy to parse multi-line Windows logs. Miscellaneous features * syslog-debun (debug bundle script for syslog-ng) has been improved Bugfixes * geoip-parser: When default database if not specified, syslog-ng crashed. * Added support for multiple drivers with the same name in syslog-ng config. * Fixed aack counting logic for junctions that have branches that modify the LogMessage. * Fixed a potential crash for code that uses log_msg_clear() in production (e.g. syslog-parser()). * Fixed potential crash in reload logic * system(): use string comparison instead of numeric in PID rewrite The meaning of the != operator has been fixed to refer to numeric comparison in @version<https://github.com/version>: 3.8, so make sure we are using string comparison. * Support encoding on glib compiled with libiconv * pdbtool: Fix the ordering of the debug-info list in PatternDB * afprog: Don't kill our own process group If, for some reason, the program source/destination failed to set up its own process group, we need to make sure we do not run killpg() on that process group, as it would kill ourselves. * Handle option names with hyphen (-) characters in java scls * dnscache performance improved Instead of getting rid off the per-thread DNSCache when a worker thread exits, store them in a linked list and acquire them as a new thread starts. The set of cached hostnames are valuable as worker threads come and go (they exit after 10seconds of inactivity), but without this reusing of cache instances, our DNS cache is filled again and again. * Fixed IPv6 parser in patterndb. * Fixed journald program name flapping * Fixed create-dirs() inheritance in file destinations * Fixed pass-unix-credentials() global inheritance in afunix The global pass-unix-credentialsoption was not inherited in afunix-source if the options{}; block was positioned lower in the configuration file than the given module declaration. * Fixed create-dirs() global inheritance in afunix When the global create-dirs option was set toyes, the local one was ignored. * Fixed byteorder handling on bigendian systems in netmask6 filter * Fixed flow-control issue when overflow queue is full (suspending source by setting the window size to 0). * Log HTTP response error codes in HTTPDestination (Java). * Fixed potential leaks related $(sanitize) argument parsing in basicfuncs. * Fixed a memory leak in python debugger * Fixed a use-after-free bug in templates. * Fixed a memory leak around reload in netmask6 filter. * Fixed a memory leak in LogProtoBufferedServer in case the encoding() option is used. * configure: don't override $enable_python while executing pkg-config * Fixed BSD timestamp parsing in syslog-format. * Fixed a SIGPIPE bug in program destination. * Error handling has been improved in AMQP destination. * value-pairs performance improvements, memleak fixes * Various issues around UTF-8 support fixed. * Fixed integer overflow in numerical operations template function * Fixed an integer underflow in afsocket. * Fixed numerical comperisons issues around filters. There's a problem in straight fixing this issue though: anyone who used the numeric operators erroneously will have their behaviour changed, therefore this patch also adds a configuration update warning in case someone is using the wrong syntax. * Fixed kernel log message time drift on Linux. * Take CRLF sequences equivalent to an LF in patterndb. Windows logs contain embedded CRLFs which is difficult to match against from db-parser(), as we use a UNIX text file to store the patterns. Also, the fact that the input contains CRLF whereas our patterns only contain an LF makes it a very unintuitive non-match, which is difficult to debug. * When syslog-ng failed to insert data into Redis, it has crashed. * When device file is set as a file destination then syslog-ng will not try to change the permission of the device file. * Various fixes around config file parsing: * in some circumstances syslog-ng crashed when the config file contained non-readable characters * fixed a memory leak * fixed memory leak around backtick substitution Notes to the Developers * copyright cleanup in source tree * install tools and scl under a syslog-ng specific subdirectory These should never be installed in /usr/share directly, but rather under a subdirectory and as described in https://www.gnu.org/prep/standards/html_node/Directory-Variables.html we should do that right within the source and not rely on packaging tools to do it for us.This will trigger a required change in packaging scripts to avoid changing the --datadir, as the default of /usr/share should work out-of-the-box. * Support for native-lanugage (compiled languages, like Rust) bindings. These bindings just forward the calls to the native side. This whole module compiles into a static library (libsyslog-ng-native-connector.a) which is linked to all external native modules. A native module defines the required functions (like native_parser_proxy_new()) so those symbols will be resolved. Some symbols have the visibility(hidden) attribute applied to them. Those symbols are defined by the other half of the native bindings, we only need their signature here. They are hidden because their definition is contained in other source files but we would like to keep them as "library private". If they are exported, the dynamic linker will resolve them when a module is loaded, therefore every module would be mapped to the first loaded one. It is best to hide everything in this native connector. * Support system librabbitmq-c (AMQP destination) Currently only the internal librabbitmq-c is supported, if we want to use the preexisting library, the configuration will fail. This change is required if we want to get rid of the internal libraries. * Added check-valgrind target to Makefile * Remove dependency on Gradle for building Java language bindings (not modules, just the language binding) * Experimental CMake support added * Experimental OSX support added * Improved Travis build matrix * Added plugin skeleton creator. * Debian packaging improved: Debian packaging from unofficial OBS repository has been ported to upstream. From now, someone could easily build debian packages from upstream source. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Istvan Mozes, Andras Mitzki, Arnaud Vamorec, Balazs Scheidler, David Schweikert, Fabien Wernli, Flavio Medeiros, Hanno Böck, Henrik Grindal Bakken, Gergo Nagy, Gyorgy Pasztor, Laszlo Budai, Laszlo Varady, Marc Falzon, Noemi Vanyi, Peter Czanik, Robert Fekete, Tamas Nagy,Tibor Benke, Viktor Juhasz, Viktor Tusa, Vincent Bernat, Zdenek Styblik, Zoltan Fried, Zoltan Pallagi, Yilin Li https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1 https://github.com/balabit/syslog-ng/releases/download/syslog-ng-3.8.1/sysl… <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1> regards, Laszlo Budai

2 1

Disabling SSLv3 with Syslog-NG
by Lupo, Joseph 19 Aug '16

19 Aug '16

The syslog-ng documentation is very unclear. Where would we put the ssl-options(no-sslv2, no-sslv3, no-tlsv1) entry for disabling sslv2, sslv3 and tlsv1? The following works for enabling TLS on our host, but I can’t figure out where to insert the ssl-options. source s_net { network(ip('<our IP>') port(<our port>) transport("tls") tls( key-file("/usr/local/etc/hostcert.key") cert-file("/usr/local/etc/hostcert.pem") peer_verify(optional-untrusted)) ); }; Joe Lupo T-Mobile USA Principal Engineer, System Design & Strategy (973) 440-8768

4 8

syslog-ng Open Source Edition 3.8 Administrator Guide - initial release
by Fekete, Róbert 19 Aug '16

19 Aug '16

Hi, You can find the Administrator Guide for version 3.8 of syslog-ng Open Source Edition at https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… Please note that it is a draft version, as it does not cover every new feature of 3.8 yet. You can find a (possibly incomplete) list of what is already covered below. Any comment, correction, or feedback is most welcome either here on the list, or at https://github.com/balabit/syslog-ng-ose-guides (Better, real-life configuration examples would be especially useful.) I will release updates to the docs ~daily next week, and hope to cover most of the new features by the end of next week. * grouping-by parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * elastic2 destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Apache Access Log parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * date-parser: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * libcurl http-destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Loggly destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… * Logmatic destination: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose… Kind Regards, Robert

1 0

3.8 RHEL 6 packaging
by Czanik, Péter 19 Aug '16

19 Aug '16

Hi, I think, that RHEL / CentOS 6 is ancient, but still I ask: is there anyone who needs syslog-ng 3.8 packaged for this old Linux release? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

3 5

Unofficial Debian/Ubuntu packages for syslog-ng-3.8.1
by Laszlo Budai 19 Aug '16

19 Aug '16

Hi, syslog-ng 3.8.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 8.0 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 (Debian 7 is not supported by OBS anymore. Precise: I haven't created packages yet... I'm thinking about dropping precise support... if someone would need it, just send a mail to me.) Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` You can replace Debian_8.0 to * xUbuntu_14.04 * xUbuntu_14.10 * xUbuntu_15.04 * xUbuntu_15.10 * xUbuntu 16.04 Available packages: syslog-ng syslog-ng-core syslog-ng-dbg syslog-ng-dev syslog-ng-mod-add-contextual-data syslog-ng-mod-amqp syslog-ng-mod-curl syslog-ng-mod-elastic syslog-ng-mod-geoip syslog-ng-mod-graphite syslog-ng-mod-hdfs syslog-ng-mod-http syslog-ng-mod-java-common-lib syslog-ng-mod-java syslog-ng-mod-journal syslog-ng-mod-json syslog-ng-mod-kafka syslog-ng-mod-kvformat syslog-ng-mod-mongodb syslog-ng-mod-python syslog-ng-mod-redis syslog-ng-mod-riemann syslog-ng-mod-smtp syslog-ng-mod-sql syslog-ng-mod-stomp Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

Syslog-ng client through a load balancer with SSL/TLS encryption
by Lupo, Joseph 18 Aug '16

18 Aug '16

I am trying to setup Syslog-ng to relay messages from one syslog server to another with a load balancer in between. I am also using TLS encryption. The issue I’m having right now is that when the client intiates the connection, it seems to lock on to a particular back end syslog server and send all of its messages there instead of switching off to another one. On its own this isn’t a big problem except that if that system goes down, the client doesn’t seem to be aware. I also haven’t found a good way to force syslog-ng to close and re-establish its connections without fully shutting down the relay system. We currently have no persistence setup on the load balancer. Is there a way to tell the relay server to periodically reconnect? Maybe send a certain amount of messages or data before reconnecting so that the data is balanced across the backend syslog-ng servers? Also, is there a better way to have the relay system learn about the remote server going offline so it can immediately reset its connection?

4 4

compression during tls
by thejaguar＠tutanota.de 17 Aug '16

17 Aug '16

Hi all, According to this https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… Allow-compression() should be a working option under tls. I did google on this a lot and found no config snippetwhich shows allow-compress() to be working and to my surprise when I tried toinclude it in my config I get syntax error ___________________________________________________ 2016-08-11T15:27:19.538347] Registering candidate plugin;module='afsocket', context='source', name='systemd-syslog', preference='100' [2016-08-11T15:27:19.539190] Using /dev/log Unix socketwith systemd is not possible. Changing to systemd-syslog source, which supportssocket activation.; Error parsing afsocket, syntax error, unexpectedLL_IDENTIFIER, expecting ')' in /etc/syslog-ng/syslog-ng.conf at line 27,column 124: destination d_net_tls { network("syslog1.xxxxxxxxx.com" port(6514) transport("tls") tls(ca-dir("/etc/syslog-ng/ca") allow-compress(yes)peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) ^^^^^^^^^^^^^^ ___________________________________________________ I am using 3.8.0 on ubuntu 15.10. Is this even really supported as claimed in here https://www.balabit.com/network-security/syslog-ng/comparing/detailed # /usr/sbin/syslog-ng -V syslog-ng 3.8.0beta2 Installer-Version: 3.8.0beta2 Revision: 3.8.0beta2-1 Module-Directory: /usr/lib/syslog-ng/3.8 Module-Path: /usr/lib/syslog-ng/3.8 Available-Modules:affile,basicfuncs,system-source,cryptofuncs,pseudofile,afuser,csvparser,linux-kmsg-format,confgen,sdjournal,syslogformat,afprog,dbparser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off

2 3

Re: [syslog-ng] sylog-ng filters not working
by Christian Turner 03 Aug '16

03 Aug '16

@version: 3.2 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # https://www.balabit.com/support/documentation # @include "scl.conf" options { flush_lines (0); time_reopen (10); log_fifo_size (2048); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (no); stats_freq(86400); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); }; ### MYAPP Dev Logs ### ## DEVENV ## source src_devenv { udp(ip(0.0.0.0) port(514)); }; filter f_devenv_01ui { netmask(10.22.206.0/24); }; filter f_devenv_02gw { netmask(10.22.207.0/24); }; filter f_devenv_03api { netmask(10.22.208.0/24); }; filter f_devenv_04net { netmask( "10.22.209.0/24" ); }; filter f_devenv_05bat { netmask(10.22.210.0/24); }; destination d_devenv_01ui { file("/mnt/syslogng/MYAPPlogs/DEVENV/01ui-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_02gw { file("/mnt/syslogng/MYAPPlogs/DEVENV/02gw-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_03api { file("/mnt/syslogng/MYAPPlogs/DEVENV/03api-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_04net { file("/mnt/syslogng/MYAPPlogs/DEVENV/04net-$HOST-$YEAR$MONTH$DAY.log"); }; destination d_devenv_05bat { file("/mnt/syslogng/MYAPPlogs/DEVENV/05bat-$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_devenv); filter(f_devenv_01ui); destination(d_devenv_01ui); }; log { source(src_devenv); filter(f_devenv_02gw); destination(d_devenv_02gw); }; log { source(src_devenv); filter(f_devenv_03api); destination(d_devenv_03api); }; log { source(src_devenv); filter(f_devenv_04net); destination(d_devenv_04net); flags(final); }; log { source(src_devenv); filter(f_devenv_05bat); destination(d_devenv_05bat); }; ## MYAPP ALL ## source src_MYAPP { udp(ip(0.0.0.0) port(514)); }; destination d_MYAPP { file("/mnt/syslogng/MYAPPlogs/$HOST/$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_MYAPP); destination(d_MYAPP); }; #source external { tcp(); }; #source external { udp(); }; #destination d_hosts { file("/home/syslog/$HOST/application.log" owner("syslog") group("syslog") perm(0600)); }; destination d_mesg { file("/var/log/messages"); }; #destination d_cons { file("/dev/console"); }; #destination d_auth { file("/var/log/secure"); }; #destination d_mail { file("/var/log/maillog" flush_lines(10)); }; #destination d_spol { file("/var/log/spooler"); }; #destination d_boot { file("/var/log/boot.log"); }; #destination d_cron { file("/var/log/cron"); }; #destination d_kern { file("/var/log/kern"); }; #destination d_mlal { usertty("*"); }; #destination d_all { file("/var/log/splunk"); }; log { source(s_sys); destination(d_mesg); }; #log { source(external); destination(d_hosts); }; From: Christian Turner Sent: Wednesday, August 3, 2016 11:53 AM To: 'syslog-ng(a)lists.balabit.hu' <syslog-ng(a)lists.balabit.hu> Subject: RE: sylog-ng filters not working Hi, I have the following filter configured; source src_devenv01 { udp(ip(0.0.0.0) port(514)); }; filter f_devenv01_04net { netmask(10.22.209.0/24); }; destination d_devenv_04net { file("/mnt/syslogng/p2alogs/DEVENV/04net-$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_devenv01); filter(f_devenv_04net); destination(d_devenv_04net); flags(final); }; However, the filter does not work, and the logs from this source all go to the generic logging destination. I perform an strace and I can see that the IP appears as expected, so I'm figuring I have a syntax error somewhere; [pid 28481] recvfrom(11, "<182>1 2016-08-03T10:27:50.645062-04:00 ::1 [[REDACTED]]..., 8192, 0, {sa_family=AF_INET, sin_port=htons(58785), sin_addr=inet_addr("10.22.209.10")}, [16]) = 265 Christian Turner

2 1

Re: [syslog-ng] sylog-ng filters not working
by Christian Turner 03 Aug '16

03 Aug '16

Hi, I have the following filter configured; source src_devenv01 { udp(ip(0.0.0.0) port(514)); }; filter f_devenv01_04net { netmask(10.22.209.0/24); }; destination d_devenv_04net { file("/mnt/syslogng/p2alogs/DEVENV/04net-$HOST-$YEAR$MONTH$DAY.log"); }; log { source(src_devenv01); filter(f_devenv_04net); destination(d_devenv_04net); flags(final); }; However, the filter does not work, and the logs from this source all go to the generic logging destination. I perform an strace and I can see that the IP appears as expected, so I'm figuring I have a syntax error somewhere; [pid 28481] recvfrom(11, "<182>1 2016-08-03T10:27:50.645062-04:00 ::1 [[REDACTED]]..., 8192, 0, {sa_family=AF_INET, sin_port=htons(58785), sin_addr=inet_addr("10.22.209.10")}, [16]) = 265 Christian Turner

4 5

please test syslog-ng 3.8
by Czanik, Péter 03 Aug '16

03 Aug '16

Hi, The first beta of syslog-ng 3.8 was released last week. There are now unofficial packages for the Debian (Ubuntu) / Fedora (RHEL) / (open)SUSE family of Linux distributions. I collected some of the highlights of the new release and also the packaging related changes. You can read it at https://czanik.blogs.balabit.com/2016/08/syslog-ng-3-8-what-changed/ Please test syslog-ng 3.8 and let us know not only your problems but also your success stories! Thanks, bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 135, Issue 14
by Hollósi Botond 01 Aug '16

01 Aug '16

Hi Balázs, Thank you. The '$SOURCEIP' what i need, combined with 'flags(no-parse)'. With this i can separate the messages by the source address contained in the ip packet header. config example: source s_net_0 { network( ip(0.0.0.0) port(600) transport(udp) flags(no-parse) ); }; destination d_file_0 { file("/var/log/remote_log/$R_MONTH/$R_DAY/$SOURCEIP/100/$user.log" owner(root) group(root) create-dirs(yes) perm(0700) dir-perm(0700)); }; log { source(s_net_0); destination(d_file_0); }; -- Üdvözlettel: Hollósi Botond Opennetworks Kft. Tel.: 06-1-9996000 Mobil: 06-20-4362032 2016-07-31 12:00 keltezéssel, syslog-ng-request(a)lists.balabit.hu írta: > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: Central netlog server for hosts behind NAT (Scheidler) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 30 Jul 2016 20:02:00 +0200 > From: Scheidler, Bal?zs <balazs.scheidler(a)balabit.com> > Subject: Re: [syslog-ng] Central netlog server for hosts behind NAT > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CANWQT2OUDC7T-8FBrg2g3zdk9nPsBAOZj8Wq-vWmzUeywxOQ1Q(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I am not sure I understand your usecase, and question. $HOST is populated > based on the host field within the message and senders are free to set that > to whatever they please. > > If that field is missing (which it might), syslog-ng fills that based on > the sender IP address. > > There are alternative macros (such as $SOURCEIP), which is the actual IP of > the datagram received by syslog-ng. But you can also play with $HOST > related syslog-ng options such as keep-hostname(). > > Could you try to rephrase your question? > Thanks > Bazsi > >

1 0

Central netlog server for hosts behind NAT
by Hollósi Botond 30 Jul '16

30 Jul '16

2 1

unofficial 3.8 beta1 openSUSE/Fedora/SLES/RHEL packages
by Czanik, Péter 28 Jul '16

28 Jul '16

Hi, My unofficial RPM packages for syslog-ng 3.8 beta1 are ready. Fedora / EPEL7: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ openSUSE / SLES12: https://build.opensuse.org/project/show/home:czanik:syslog-ng38 You can read more about what changed between 3.7 and 3.8 in the release announcement at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.0beta1 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng-3.8.0beta1 unofficial Debian/Ubuntu packages
by Budai, László 28 Jul '16

28 Jul '16

Hi, syslog-ng 3.8.0beta1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 8.0 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 * Ubuntu 15.10 * Ubuntu 16.04 (Debian 7 is not supported by OBS anymore. Precise: I haven't created packages yet... I'm thinking about dropping precise support... if someone would need it, just send a mail to me.) Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` You can replace Debian_8.0 to * xUbuntu_14.04 * xUbuntu_14.10 * xUbuntu_15.04 * xUbuntu_15.10 * Ubuntu 16.04 Available packages: syslog-ng syslog-ng-core syslog-ng-dbg syslog-ng-dev syslog-ng-mod-add-contextual-data syslog-ng-mod-amqp syslog-ng-mod-curl syslog-ng-mod-elastic syslog-ng-mod-geoip syslog-ng-mod-graphite syslog-ng-mod-hdfs syslog-ng-mod-http syslog-ng-mod-java syslog-ng-mod-java-common-lib syslog-ng-mod-journal syslog-ng-mod-json syslog-ng-mod-kafka syslog-ng-mod-kvformat syslog-ng-mod-mongodb syslog-ng-mod-python syslog-ng-mod-redis syslog-ng-mod-riemann syslog-ng-mod-smtp syslog-ng-mod-sql syslog-ng-mod-stomp Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.8.0beta1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

Coming from Graylog to Syslog-NG
by Roberto Carna 27 Jul '16

27 Jul '16

Dear, I've implemented Graylog to receive around 5000-6000 logs/sec, but I have several problems: INPUT greater than OUTPUT so the journal increase a lot and the system crash, mainly. Now I want to use Syslog-NG, because I used it in the past but for 100-200 logs/sec. Here is my questions: 1) If I use Syslog-NG with a flat text file to receive 5000-6000 logs/sec, using s fast disk from my storage, do you recommneded to me this option ??? 2) As an extra benefit, what is the best and simple way to have a graphical view of all the logs ??? Thanks a lot, and maybe I'll ask you again about this topic in accordance with your comments. Roberto

4 5

How to make auto archive
by Harsha S Aryan 26 Jul '16

26 Jul '16

Hi, Can any one let me know how to make auto archive for my logs using syslog-ng Regards, Aryan

1 0

[Syslog-ng] Log particular actions and send specific files to a remote server
by Savakh S 26 Jul '16

26 Jul '16

Hello everyone, I have two questions about syslog-ng: 1 - Is it possible to send to a remote syslog-ng server a particular log file, for example the following /var/log/network.log, and how we can do this? iface eth1 inet dhcp ... up echo $IFACE up $(date) >> /var/log/network.log down echo $IFACE down $(date) >> /var/log/network.log 2 - Is it possible to log in real time a change occured a network interface ? For example a new open port. More generally, how can we log actions naturally not covered by syslog-ng in order to send to a remote server? Thank you for any help.

3 2

initial rpm packaging of syslog-ng rust parser modules
by Czanik, Péter 21 Jul '16

21 Jul '16

Hi, As you might be aware, syslog-ng 3.8 -- which is still under development -- introduced the possibility of writing parser modules in Rust ( https://www.rust-lang.org/) There are some ready to use modules available at https://github.com/ihrwein/syslog-ng-rust-modules Three of these received now initial RPM packaging: regex-parser, actiondb-parser and correlation-parser. My packages are mostly untested, version numbers are messed up, documentation is not yet bundled, etc., but still in good enough shape for initial testing. You can find some documentation in the github repo under https://github.com/ihrwein/syslog-ng-rust-modules Packages are built from the latest git head syslog-ng and rust-modules sources. You can download the rpm packaged rust modules and syslog-ng from: Fedora / EPEL7 (x86_64): https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ openSUSE Leap / Tumbleweed: https://build.opensuse.org/project/show/home:czanik:syslog-ng38 Note: syslog-ng 3.8 is not yet released, so use of these packages in a production environment is not recommended. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

powered by syslog-ng
by Czanik, Péter 19 Jul '16

19 Jul '16

Hi, We have now a "Powered by syslog-ng" page on our website. It's available at https://syslog-ng.org/powered-by-syslog-ng/ If your project / company / product / etc. uses syslog-ng, we are happy to include you on the list too, just e-mail me directly! Some of the more interesting names from the list are Facebook, Kindle, FreeNAS and of course all the major Linux distributions. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

2 1