- syslog-ng - lists.balabit.hu

linux kernel tuning
by Jim Hendrick 23 Jun '17

23 Jun '17

Hi all, What are your experiences with tuning the kernel for UDP, TCP, etc. How long does it take to see results from a single change? We are working issues with UDP drops, and have made fairly radical increases in buffer sizes with no apparent improvement, only to re-visit the system in a few days and seeing it cleaned up. "intuitively" I have always thought it should happen right away - but that is not what we are seeing here now. Thoughts? Thanks in advance! Jim

2 1

wildcard-file() source
by Czanik, Péter 22 Jun '17

22 Jun '17

Hi, Last week I ran a poll on twitter about the new features of syslog/ng 3.10 and the wildcard-file() source was voted the most. Starting with version 3.10, syslog-ng can collect messages from multiple text files. You do not have to specify file names one by one, just use a wildcard to select which files to read. This is especially useful when you do not know the file names by the time syslog-ng is started. This is often the case with web servers with multiple virtual hosts. From this blog you can learn how to get started using the wildcard-file() source. https://www.balabit.com/blog/reading-multiple-files-wildcard-file-source-sy… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

question Syslog-NG Set Thread Name ?
by rmkml 20 Jun '17

20 Jun '17

Hello, First, Big Thx everyone for very good Syslog-NG product! During middle/high load, it's always interesting for known what Jobs load Syslog-NG process, but if I think correctly, top -H not show what Thread Name is used ? Do you known it's possible Syslog-NG supporting Set Thread Name like pthread_setname_np ? Best Regards @Rmkml

1 0

template() parameter for apache-accesslog-parser
by Czanik, Péter 20 Jun '17

20 Jun '17

Hi, I ran into a situation, that an apache access log message arrives as a field in a CSV file. Rigth now the apache-accesslog-parser is always run on the MESSAGE template, but in this case I'd need to run this parser on csv.apache instead of MESSAGE. What do you think about the following change? It would keep the default behavior but enable me to use a different a template. Bye, CzP linux-pzl9:/usr/share/syslog-ng/include/scl/apache # diff -u /root/syslog-ng/scl/apache/apache.conf apache.conf --- /root/syslog-ng/scl/apache/apache.conf 2017-06-19 13:47:20.044547152 +0200 +++ apache.conf 2017-06-20 16:51:06.740528980 +0200 @@ -19,7 +19,7 @@ # COPYING for details. # ############################################################################# -block parser apache-accesslog-parser(prefix(".apache.")) { +block parser apache-accesslog-parser(prefix(".apache.") template("${MESSAGE}")) { channel { @@ -29,6 +29,7 @@ dialect(escape-double-char) flags(strip-whitespace) delimiters(" ") + template(`template`) quote-pairs('""[]') # field names match of that of Logstash columns("clientip", "ident", "auth", Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

tags not working
by Sandbox 19 Jun '17

19 Jun '17

Hi, I would like to transfer/store apache logs, but for some reason i cant tag them: I tried: source s_access { pipe("/var/log/apache2/pipe_access.log" tags("testtag"));. }; or source s_access { pipe("/var/log/apache2/pipe_access.log" program-override("testtag"));. }; Because it doesn't work with 'tags' i used the apache to format the message, i put the choosen word to the log format so it stores the correct place the log on the client and also on the server. Part of the log: - [19/Jun/2017:13:46:29 +0200] acc001 127.0.0.1 - - "GET /server-status?auto HTTP/1.1" 200 1781 If it possible i want to use 'tags' instead of manipulating apache logformats. I tried to read the pipes directly, but the above settings doesnt show, just the above log. I tried to tcpdump the traffic from client -> server (client side) but doesn't show any of the settings. It looks like for me, syslog-ng wont tag my logs. And i tried to read the stored log with file() and tag it, that doesn't work neither. What i am missing? Thanks, Robert

2 4

unofficial syslog-ng 3.10.1 fedora/openSUSE/RHEL/SLES packages
by Czanik, Péter 19 Jun '17

19 Jun '17

Hi, The syslog-ng 3.10.1 release came a bit earlier than I expected, so I did not yet prepare dedicated repositories for the release. I do not want to leave you without RPM packages, so they are now available in my githead repo. Note: SLES 11 and RHEL 6 packages are currently not available: https://github.com/balabit/syslog-ng/issues/1403 You can read here how to use my githead repo: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ I hope to have dedicated repositories before leaving to Libre Software Meeting by the end of the month. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

unofficial syslog-ng 3.10.Debian/Ubuntu packages
by Laszlo Budai 19 Jun '17

19 Jun '17

Hi, syslog-ng 3.10.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 8.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng 3.10
by Laszlo Budai 19 Jun '17

19 Jun '17

syslog-ng-3.10.1<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1> Features * Support https in http (curl) module * Docker support : from now Dockerfile for CentOS7, Ubuntu Zesty and for Debian Jessie is part of our upstream * Add --database parameter for geoip template function * Metric improvements * add discarded messages for parsers * add matched/not matched counter for filters * add memory_usage counter to logqueue * add written counter * Written is a calculated counter which return the written messages by destinations. Written message is which was processed but not queued and not dropped. (written = processed - queued - dropped) * stats-counters: rename stored counter to queued * add global_allocated_logmsg_size counter for tracking memory logmsg related allocations * Add snmp-parser (v1, v2) * parses snmptrapd log * The parsed information is available as key-value pairs, which can be used/serialized (macros, format-json, etc.) in the log path. If you want to send the message in a structured way, you can disable the default message generation with the generate-message(no) option. * Add snmp-soure * available as an SCL block that containing a filesource and an SNMP parser modules: add snmptrapd parser * Add osquery source * available as an SCL block * It reads the osquery log file and parses with the JSON parser, creating name-vaule pairs with an .osquery. prefix by default. * Add cisco-parser * available as an SCL block * Add wildcard filesource * Add startdate template function * Add $(basename) and $(dirname) template functions * Add Kerberos support for HDFS destination * Add AUTH support for redis destination * Add map-value-pairs() parser * it can be used to map existing name-value pairs to a different set during processing, in bulk. Normal value-pairs expressions can be used, just like with value-pairs based destinations. * Extend Python language binding by Python parser * Add support for extract-stray-words() option in kv-parser() * stray words: those words that happen to be between key-value pairs and are otherwise not recognized either as keys nor as values. * Add $(context-values) template function * Add $(context-lookup) function * Add list related template functions * $(list-head list ...) returns the first element (unquoted) * $(list-nth NDX list ...) returns the specific element (unquoted) * $(list-concat list1 list2 ...) returns a list containing the concatenated list * $(list-append list elem1 elem2 ...) returns a list, appending elem1, elem2 ... * $(list-tail list ...) returns a list containing everything except for the first element * $(list-slice FROM:TO list ...) returns a list containing the slice [FROM:TO), Python style slice boundaries are supported (e.g. negative) * $(list-count ...) returns the number of elements in list * Add add query commands to syslog-ng-ctl * query list List names of counters which match the filter * query get Get names and values of counters which match the filter * query get --sum Get the sum of values of counters which match the filter * Support multiple servers in elasticsearch2-http destination * Implements elastic-v2 https in http mode * Add getent module (ported from incubator) * This module adds $(getent) that allows one to look up various NSS based databases, such as passwd, services or protocols. * Add support for IP_FREEBIND Bugfixes * Fix a libnet detection check error that caused problem configuring enable-spoof-source. * Avoid warnings about _DEFAULT_SOURCE on recent glibc versions With the glibc on zesty, using _GNU_SOURCE and not defining _DEFAULT_SOURCE results in a warning, avoid that by defining _DEFAULT_SOURCE as well. * Fix invalid database warning for geoip parser * Fix prefix() default in systemd-journal for new config versions * Fix a potential message loss in Riemann destination * Fix a potential crash in the Riemann destination when the client is not connected to the Riemann server. * Fix a possible add-contextual-data() related data loss in case of multiple reference to the same add-contextual-data parser in several logpaths. * Fix dbparser deadlock * Fix Python destination * open() was not called in every time_reopen() * python destination is not defined in stats output * Fix processed stats counter for afsocket * Fix stats source for pipes * Previously pipe source is shown as file * Fix csv-parser multithreaded support In some cases (when csv-parser attached to network source), the parser randomly filled the column macros with garbage. * Fix a message loss in case of filesource when syslog-ng was restarted and the log_msg_size > file size. * Fix a potential crash in cryptofuncs * Fix a potential crash in syslog-ng-ctl when no command line parameters was set. * Fix token duplication in the output of '--preprocess-into' * Fix UTF-8 support in syslog-ng-ctl * Fix a potential crash during X.509 certificate validation. * Fix a segfault in Python module startup * Fix a possible endless reading loop issue in case of multi-line filesource. * Fix soname for the http module from "curl" to "http" * Avoid openssl 1.1.0 deprecated APIs When openssl is built with --api=1.1 disable-deprecated, use of deprecated APIs results in build failure. Other changes * Increase processed counter by queued counter after reload or restart when diskqueue is used otherwise the newly added written counter would underflow. * Set the default time-zone to UTC for elasticsearch2 Elasticsearch and Kibana use UTC internally. * Add retries support for python destination * Prefer server side cipher suite order * Always include librabbitmq in the dist tarball * Always include ivykis in the dist tarball * Marking parse error locations with >@<. * Default log_msg_size is increased to 64Kbyte from 8Kb * Tons of syslog-debun improvements * Exit with 0 return code when --help is specified for syslog-ng-ctl * syslog-ng: make '--preprocess-into' foreground only * Add debug messages on log_msg_set_value() * Add more detail to filter evaluation related debug messages Notes to the Developers * Extract template perf test function to testlib * Print a debug message when logmsg passed to the Python side * Allow http module (curl) to be build with cmake * astylerc: allow continuation lines to start until column 60 * Move kv-scanner under syslog-ng/lib * scratch-buffers2: implement an alternative to current scratch buffers This new API is aimed a bit easier to use in situations where a throw away buffer is needed that will automatically be freed at the next message. It also gets does away with GTrashStack that is deprecated in recent glib versions. * Several refactors in stats module. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, eroen, Fabien Wernli, Gabor Nagy, Gergely Nagy, Jason Hensley, Laszlo Varady, Laszlo Budai, Mate Farkas, Noemi Vanyi, Peter Czanik, Peter Gervai, Todd C. Miller, Philip Prindeville, Zoltan Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1

1 0

Why Syslog-NG can't start?
by Jason Long 19 Jun '17

19 Jun '17

Hello. I installed Syslog-NG on CentOS 7 x64 and my configuration is : @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); ts_format(iso); encoding(UTF-8); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; destination d_separatedbyhosts{file("/var/log/$HOST/messages" owner("root") group("root") perm(0655) dir_perm(0755) create_dirs(yes)); }; log { source(s_netsyslog); destination(d_separatedbyhosts); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: But my Syslog-NG can't start and give me below error: # systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Mon 2017-06-19 05:33:42 EDT; 32s ago Docs: man:syslog-ng(8) Process: 20920 ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid (code=exited, status=1/FAILURE) Main PID: 20920 (code=exited, status=1/FAILURE) Status: "Starting up... (Mon Jun 19 05:33:41 2017" Jun 19 05:33:41 localhost.localdomain systemd[1]: Unit syslog-ng.service entered failed state. Jun 19 05:33:41 localhost.localdomain systemd[1]: syslog-ng.service failed. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Jun 19 05:33:42 localhost.localdomain systemd[1]: start request repeated too quickly for syslog-ng.service Jun 19 05:33:42 localhost.localdomain systemd[1]: Failed to start System Logger Daemon. Jun 19 05:33:42 localhost.localdomain systemd[1]: Unit syslog-ng.service entered failed state. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service failed. Jun 19 05:33:42 localhost.localdomain systemd[1]: start request repeated too quickly for syslog-ng.service Jun 19 05:33:42 localhost.localdomain systemd[1]: Failed to start System Logger Daemon. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service failed. How can I solve it? Thank you.

2 1

group name being a function of the $HOST macro?
by syslog-ng＠v3-1347.vlinux.de 18 Jun '17

18 Jun '17

Hello all, presently, I have a generic (so-to-say "zero configuration") destination in my syslog-ng.conf, destination d_host_month_facility_level {file("/var/log/$HOST/$YEAR-$MONTH.$FACILITY.$LEVEL"); }; , which works fine for my purposes. Presently, I have some scripts, processing log data, which have to run under "root", since the syslog files are all created with default owner("root") and perm("600"). I'd like to have such scripts running under less privileged accounts, that are member of a log source specific group. My first attempt was a destination d_host_month_facility_level { file("/var/log/$HOST/$YEAR-$MONTH.$FACILITY.$LEVEL" group("syslog_$HOST_r") perm(0640)); }; , but results in an error Error resolving group; group='syslog_$HOST_r' , which shows, that in the context of a "group()" parameter, the $HOST macro seem to be not evaluated. Does anybody here see another means, that would allow me to achieve something like the above without going away from my beloved generic single destination specification? Ideally, what I would like to achieve further to get an abstraction away from the hostname itself, is to have a mapping from $HOST to something that I can specify in a kind of conditional statement like if $HOST == "my_box" then groupname="user_machine_log_reader" else if $HOST == "some_of_my_routers" then groupname="wan_log_reader" else if $HOST == "some_other_of_my_routers" then groupname="firewall_log_reader" else groupname="$HOST" and finally, of course, a generic 'group("$groupname")' parameter to get it into position. Any ideas? Best regards and thanks.

1 0