June 2017 - syslog-ng - lists.balabit.hu

unofficial syslog-ng 3.10.1 fedora/openSUSE/RHEL/SLES packages
by Czanik, Péter 19 Jun '17

19 Jun '17

Hi, The syslog-ng 3.10.1 release came a bit earlier than I expected, so I did not yet prepare dedicated repositories for the release. I do not want to leave you without RPM packages, so they are now available in my githead repo. Note: SLES 11 and RHEL 6 packages are currently not available: https://github.com/balabit/syslog-ng/issues/1403 You can read here how to use my githead repo: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ I hope to have dedicated repositories before leaving to Libre Software Meeting by the end of the month. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

unofficial syslog-ng 3.10.Debian/Ubuntu packages
by Laszlo Budai 19 Jun '17

19 Jun '17

Hi, syslog-ng 3.10.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 8.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng 3.10
by Laszlo Budai 19 Jun '17

19 Jun '17

syslog-ng-3.10.1<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1> Features * Support https in http (curl) module * Docker support : from now Dockerfile for CentOS7, Ubuntu Zesty and for Debian Jessie is part of our upstream * Add --database parameter for geoip template function * Metric improvements * add discarded messages for parsers * add matched/not matched counter for filters * add memory_usage counter to logqueue * add written counter * Written is a calculated counter which return the written messages by destinations. Written message is which was processed but not queued and not dropped. (written = processed - queued - dropped) * stats-counters: rename stored counter to queued * add global_allocated_logmsg_size counter for tracking memory logmsg related allocations * Add snmp-parser (v1, v2) * parses snmptrapd log * The parsed information is available as key-value pairs, which can be used/serialized (macros, format-json, etc.) in the log path. If you want to send the message in a structured way, you can disable the default message generation with the generate-message(no) option. * Add snmp-soure * available as an SCL block that containing a filesource and an SNMP parser modules: add snmptrapd parser * Add osquery source * available as an SCL block * It reads the osquery log file and parses with the JSON parser, creating name-vaule pairs with an .osquery. prefix by default. * Add cisco-parser * available as an SCL block * Add wildcard filesource * Add startdate template function * Add $(basename) and $(dirname) template functions * Add Kerberos support for HDFS destination * Add AUTH support for redis destination * Add map-value-pairs() parser * it can be used to map existing name-value pairs to a different set during processing, in bulk. Normal value-pairs expressions can be used, just like with value-pairs based destinations. * Extend Python language binding by Python parser * Add support for extract-stray-words() option in kv-parser() * stray words: those words that happen to be between key-value pairs and are otherwise not recognized either as keys nor as values. * Add $(context-values) template function * Add $(context-lookup) function * Add list related template functions * $(list-head list ...) returns the first element (unquoted) * $(list-nth NDX list ...) returns the specific element (unquoted) * $(list-concat list1 list2 ...) returns a list containing the concatenated list * $(list-append list elem1 elem2 ...) returns a list, appending elem1, elem2 ... * $(list-tail list ...) returns a list containing everything except for the first element * $(list-slice FROM:TO list ...) returns a list containing the slice [FROM:TO), Python style slice boundaries are supported (e.g. negative) * $(list-count ...) returns the number of elements in list * Add add query commands to syslog-ng-ctl * query list List names of counters which match the filter * query get Get names and values of counters which match the filter * query get --sum Get the sum of values of counters which match the filter * Support multiple servers in elasticsearch2-http destination * Implements elastic-v2 https in http mode * Add getent module (ported from incubator) * This module adds $(getent) that allows one to look up various NSS based databases, such as passwd, services or protocols. * Add support for IP_FREEBIND Bugfixes * Fix a libnet detection check error that caused problem configuring enable-spoof-source. * Avoid warnings about _DEFAULT_SOURCE on recent glibc versions With the glibc on zesty, using _GNU_SOURCE and not defining _DEFAULT_SOURCE results in a warning, avoid that by defining _DEFAULT_SOURCE as well. * Fix invalid database warning for geoip parser * Fix prefix() default in systemd-journal for new config versions * Fix a potential message loss in Riemann destination * Fix a potential crash in the Riemann destination when the client is not connected to the Riemann server. * Fix a possible add-contextual-data() related data loss in case of multiple reference to the same add-contextual-data parser in several logpaths. * Fix dbparser deadlock * Fix Python destination * open() was not called in every time_reopen() * python destination is not defined in stats output * Fix processed stats counter for afsocket * Fix stats source for pipes * Previously pipe source is shown as file * Fix csv-parser multithreaded support In some cases (when csv-parser attached to network source), the parser randomly filled the column macros with garbage. * Fix a message loss in case of filesource when syslog-ng was restarted and the log_msg_size > file size. * Fix a potential crash in cryptofuncs * Fix a potential crash in syslog-ng-ctl when no command line parameters was set. * Fix token duplication in the output of '--preprocess-into' * Fix UTF-8 support in syslog-ng-ctl * Fix a potential crash during X.509 certificate validation. * Fix a segfault in Python module startup * Fix a possible endless reading loop issue in case of multi-line filesource. * Fix soname for the http module from "curl" to "http" * Avoid openssl 1.1.0 deprecated APIs When openssl is built with --api=1.1 disable-deprecated, use of deprecated APIs results in build failure. Other changes * Increase processed counter by queued counter after reload or restart when diskqueue is used otherwise the newly added written counter would underflow. * Set the default time-zone to UTC for elasticsearch2 Elasticsearch and Kibana use UTC internally. * Add retries support for python destination * Prefer server side cipher suite order * Always include librabbitmq in the dist tarball * Always include ivykis in the dist tarball * Marking parse error locations with >@<. * Default log_msg_size is increased to 64Kbyte from 8Kb * Tons of syslog-debun improvements * Exit with 0 return code when --help is specified for syslog-ng-ctl * syslog-ng: make '--preprocess-into' foreground only * Add debug messages on log_msg_set_value() * Add more detail to filter evaluation related debug messages Notes to the Developers * Extract template perf test function to testlib * Print a debug message when logmsg passed to the Python side * Allow http module (curl) to be build with cmake * astylerc: allow continuation lines to start until column 60 * Move kv-scanner under syslog-ng/lib * scratch-buffers2: implement an alternative to current scratch buffers This new API is aimed a bit easier to use in situations where a throw away buffer is needed that will automatically be freed at the next message. It also gets does away with GTrashStack that is deprecated in recent glib versions. * Several refactors in stats module. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, eroen, Fabien Wernli, Gabor Nagy, Gergely Nagy, Jason Hensley, Laszlo Varady, Laszlo Budai, Mate Farkas, Noemi Vanyi, Peter Czanik, Peter Gervai, Todd C. Miller, Philip Prindeville, Zoltan Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1

1 0

Why Syslog-NG can't start?
by Jason Long 19 Jun '17

19 Jun '17

Hello. I installed Syslog-NG on CentOS 7 x64 and my configuration is : @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); ts_format(iso); encoding(UTF-8); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; destination d_separatedbyhosts{file("/var/log/$HOST/messages" owner("root") group("root") perm(0655) dir_perm(0755) create_dirs(yes)); }; log { source(s_netsyslog); destination(d_separatedbyhosts); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: But my Syslog-NG can't start and give me below error: # systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Mon 2017-06-19 05:33:42 EDT; 32s ago Docs: man:syslog-ng(8) Process: 20920 ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid (code=exited, status=1/FAILURE) Main PID: 20920 (code=exited, status=1/FAILURE) Status: "Starting up... (Mon Jun 19 05:33:41 2017" Jun 19 05:33:41 localhost.localdomain systemd[1]: Unit syslog-ng.service entered failed state. Jun 19 05:33:41 localhost.localdomain systemd[1]: syslog-ng.service failed. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Jun 19 05:33:42 localhost.localdomain systemd[1]: start request repeated too quickly for syslog-ng.service Jun 19 05:33:42 localhost.localdomain systemd[1]: Failed to start System Logger Daemon. Jun 19 05:33:42 localhost.localdomain systemd[1]: Unit syslog-ng.service entered failed state. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service failed. Jun 19 05:33:42 localhost.localdomain systemd[1]: start request repeated too quickly for syslog-ng.service Jun 19 05:33:42 localhost.localdomain systemd[1]: Failed to start System Logger Daemon. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service failed. How can I solve it? Thank you.

2 1

group name being a function of the $HOST macro?
by syslog-ng＠v3-1347.vlinux.de 18 Jun '17

18 Jun '17

Hello all, presently, I have a generic (so-to-say "zero configuration") destination in my syslog-ng.conf, destination d_host_month_facility_level {file("/var/log/$HOST/$YEAR-$MONTH.$FACILITY.$LEVEL"); }; , which works fine for my purposes. Presently, I have some scripts, processing log data, which have to run under "root", since the syslog files are all created with default owner("root") and perm("600"). I'd like to have such scripts running under less privileged accounts, that are member of a log source specific group. My first attempt was a destination d_host_month_facility_level { file("/var/log/$HOST/$YEAR-$MONTH.$FACILITY.$LEVEL" group("syslog_$HOST_r") perm(0640)); }; , but results in an error Error resolving group; group='syslog_$HOST_r' , which shows, that in the context of a "group()" parameter, the $HOST macro seem to be not evaluated. Does anybody here see another means, that would allow me to achieve something like the above without going away from my beloved generic single destination specification? Ideally, what I would like to achieve further to get an abstraction away from the hostname itself, is to have a mapping from $HOST to something that I can specify in a kind of conditional statement like if $HOST == "my_box" then groupname="user_machine_log_reader" else if $HOST == "some_of_my_routers" then groupname="wan_log_reader" else if $HOST == "some_other_of_my_routers" then groupname="firewall_log_reader" else groupname="$HOST" and finally, of course, a generic 'group("$groupname")' parameter to get it into position. Any ideas? Best regards and thanks.

1 0

Issue with timestamp
by Andrew 18 Jun '17

18 Jun '17

Hi I am new to syslog-ng and logging in general so apologies but I am having trouble understanding timestamps. I am trying to extract the timestamp inside the message and create a field called 'timestamp' for example. The message looks like this: "MESSAGE":"06:17-16:21:21 fw-1-1 ulogd[9300]: id=\"2001\" severity=\"info\" sys=\"SecureNet\" sub=\"packetfilter\" name=\"Packet dropped\" action=\"drop\" fwrule=\"60002\" initf=\"eth0\" outitf=\"eth1\" srcmac=\"4c:00:82:8a:f4:11\" dstmac=\"00:1a:8c:f0:3c:40\" srcip=\"1.1.1.1\" dstip=\"2.2.2.2\" proto=\"17\" length=\"143\" tos=\"0x00\" prec=\"0x00\" ttl=\"125\" srcport=\"51941\" dstport=\"161\" How can I create a field called 'timestamp' which contains the timestamp in the message but also adds the year to it and puts it into a better format? I am also using the kv-parser and json output looks good I have pairs of everything from the message but I also need a pair called timestamp="Jun 16 2017 16:21:21' for example. Hope I'm making sense here. Thanks for any help.

4 5

documentation for 3.10 available
by Czanik, Péter 16 Jun '17

16 Jun '17

Hi, Documentation for the upcoming syslog-ng 3.10 version is now available. If you want to check out what to expect in the next release, the summary of changes is at https://www.balabit.com/documents/syslog-ng-ose-3.10-guides/en/syslog-ng-os… If you also want to test the new features, I have RPM packages available compiled from git head snapshots. Installation instructions are available at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Insider 2017-06: HEPIX; Python parser; Turris; openSUSE; upcoming events;
by Czanik, Péter 15 Jun '17

15 Jun '17

Dear syslog-ng users, This is the 59th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS HEPIX and syslog-ng ------------------- One of the most important discoveries of this decade was the Higgs boson. But researchers at High Energy Physics and Nuclear Physics laboratories and institutes would have been unable to find the Higgs boson without the IT staff maintaining the computer infrastructure collecting and analyzing the massive amount of data generated during their experiments. HEPIX is a community that brings together these IT guys twice a year from around the world. This spring their event was hosted by the Wigner Research Centre for Physics in Budapest, where I also talked to faxmodem. He is both a HEPIX member and one of the most active syslog-ng community members: https://www.balabit.com/blog/helped-turning-higgs-boson-theory-reality/ Process trade data using Python ------------------------------- Upcoming syslog-ng Open Source and Premium Edition releases will feature a Python parser. Learn how to utilize this parser from our webinar. In the demo FIX data is transformed into easy-to-use name-value pairs, but the solution itself is not limited to trade data. You can access the recording of the webinar at https://pages2.balabit.com/fix-data-parser/ (requires registration). syslog-ng updated on Turris Omnia --------------------------------- The release candidate of version 3.7 of Turris OS – the OpenWRT-based operating system of Turris Omnia routers – is now available. Among many other changes, this updates syslog-ng from version 3.0 to 3.9. In other words, it adds about seven years’ worth of new syslog-ng features, including new parsers, filters, formatting options, destinations, and performance enhancements. https://www.balabit.com/blog/using-the-latest-syslog-ng-on-turris-omnia/ openSUSE conference 2017 ------------------------- Being a long-time openSUSE user, I visit the openSUSE conference not only to present on one of its components – syslog-ng – but also to meet friends and learn about new technologies and the plans for the upcoming year. Some talks, like those about Package Hub, were very interesting and important also from a syslog-ng perspective. Of course, I also joined a few talks for my personal interest, like the one on the new ARM devices supported by openSUSE. Read more about my experiences at https://www.balabit.com/blog/friends-syslog-ng-package-hub-arm-opensuse-con… UPCOMING EVENTS The next event where you can learn about syslog-ng: Libre Software Meeting (July 1-7): https://2017.rmll.info/en/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Changing message priority
by To Mitz 15 Jun '17

15 Jun '17

Hi, I am trying to use rewrite to change a message's PRIORITY. I am able to change the HOST to "info" using: rewrite r_rewrite_set{set("info", value("HOST") condition(filter(f_logstash)));}; But, when I try setting the PRIORITY to "info", using this: rewrite r_rewrite_set{set("info", value("PRIORITY") condition(filter(f_logstash)));}; But, when I try setting the PRIORITY to "info", using this: Error parsing rewrite expression, syntax error, unexpected LL_NUMBER, expecting LL_IDENTIFIER or LL_STRING in /etc/syslog-ng/f_messages.conf at line 8, column 27:included from /etc/syslog-ng.conf line 18, column 34rewrite r_rewrite_set{set(3, value("PRIORITY") condition(filter(f_logstash)));};^ Please advise on how I can fix this. Thanks.

3 2

date-parser not found
by Sandbox 13 Jun '17

13 Jun '17

Hi, I would like to use the date-parser. When i using the below parse it says "date-parser not found": parser haproxy_date { date-parser( format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") );. }; Plugin module not found in 'module-path'; module-path='/usr/lib/syslog-ng/3.5.6', module='date-parser' Error parsing parser expression, parser plugin date-parser not found in /etc/syslog-ng/conf.d/20-haproxy.conf at line 26, column 5: included from /etc/syslog-ng/syslog-ng.conf line 165, column 1 date-parser( ^^^^^^^^^^^ It's weird, because when i try to use the date-parser module in the log section: log { source(s_haproxy); filter(f_haproxy_info); date-parser(format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") ); destination(d_haproxy_info); }; it says: Error parsing log, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/conf.d/20-haproxy.conf at line 37, column 5: included from /etc/syslog-ng/syslog-ng.conf line 165, column 1 date-parser(format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") ); ^^^^^^^^^^^ In OSE version the date-parser module is not available, or i'm missing something? Thanks, Robert

3 3