- syslog-ng - lists.balabit.hu

unofficial syslog-ng 3.10.1 fedora/openSUSE/RHEL/SLES packages
by Czanik, Péter 19 Jun '17

19 Jun '17

Hi, The syslog-ng 3.10.1 release came a bit earlier than I expected, so I did not yet prepare dedicated repositories for the release. I do not want to leave you without RPM packages, so they are now available in my githead repo. Note: SLES 11 and RHEL 6 packages are currently not available: https://github.com/balabit/syslog-ng/issues/1403 You can read here how to use my githead repo: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ I hope to have dedicated repositories before leaving to Libre Software Meeting by the end of the month. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

unofficial syslog-ng 3.10.Debian/Ubuntu packages
by Laszlo Budai 19 Jun '17

19 Jun '17

Hi, syslog-ng 3.10.1[1] packages are available in OBS repo[2]. List of supported OSs (i386/amd64): * Debian 8.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

syslog-ng 3.10
by Laszlo Budai 19 Jun '17

19 Jun '17

syslog-ng-3.10.1<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1> Features * Support https in http (curl) module * Docker support : from now Dockerfile for CentOS7, Ubuntu Zesty and for Debian Jessie is part of our upstream * Add --database parameter for geoip template function * Metric improvements * add discarded messages for parsers * add matched/not matched counter for filters * add memory_usage counter to logqueue * add written counter * Written is a calculated counter which return the written messages by destinations. Written message is which was processed but not queued and not dropped. (written = processed - queued - dropped) * stats-counters: rename stored counter to queued * add global_allocated_logmsg_size counter for tracking memory logmsg related allocations * Add snmp-parser (v1, v2) * parses snmptrapd log * The parsed information is available as key-value pairs, which can be used/serialized (macros, format-json, etc.) in the log path. If you want to send the message in a structured way, you can disable the default message generation with the generate-message(no) option. * Add snmp-soure * available as an SCL block that containing a filesource and an SNMP parser modules: add snmptrapd parser * Add osquery source * available as an SCL block * It reads the osquery log file and parses with the JSON parser, creating name-vaule pairs with an .osquery. prefix by default. * Add cisco-parser * available as an SCL block * Add wildcard filesource * Add startdate template function * Add $(basename) and $(dirname) template functions * Add Kerberos support for HDFS destination * Add AUTH support for redis destination * Add map-value-pairs() parser * it can be used to map existing name-value pairs to a different set during processing, in bulk. Normal value-pairs expressions can be used, just like with value-pairs based destinations. * Extend Python language binding by Python parser * Add support for extract-stray-words() option in kv-parser() * stray words: those words that happen to be between key-value pairs and are otherwise not recognized either as keys nor as values. * Add $(context-values) template function * Add $(context-lookup) function * Add list related template functions * $(list-head list ...) returns the first element (unquoted) * $(list-nth NDX list ...) returns the specific element (unquoted) * $(list-concat list1 list2 ...) returns a list containing the concatenated list * $(list-append list elem1 elem2 ...) returns a list, appending elem1, elem2 ... * $(list-tail list ...) returns a list containing everything except for the first element * $(list-slice FROM:TO list ...) returns a list containing the slice [FROM:TO), Python style slice boundaries are supported (e.g. negative) * $(list-count ...) returns the number of elements in list * Add add query commands to syslog-ng-ctl * query list List names of counters which match the filter * query get Get names and values of counters which match the filter * query get --sum Get the sum of values of counters which match the filter * Support multiple servers in elasticsearch2-http destination * Implements elastic-v2 https in http mode * Add getent module (ported from incubator) * This module adds $(getent) that allows one to look up various NSS based databases, such as passwd, services or protocols. * Add support for IP_FREEBIND Bugfixes * Fix a libnet detection check error that caused problem configuring enable-spoof-source. * Avoid warnings about _DEFAULT_SOURCE on recent glibc versions With the glibc on zesty, using _GNU_SOURCE and not defining _DEFAULT_SOURCE results in a warning, avoid that by defining _DEFAULT_SOURCE as well. * Fix invalid database warning for geoip parser * Fix prefix() default in systemd-journal for new config versions * Fix a potential message loss in Riemann destination * Fix a potential crash in the Riemann destination when the client is not connected to the Riemann server. * Fix a possible add-contextual-data() related data loss in case of multiple reference to the same add-contextual-data parser in several logpaths. * Fix dbparser deadlock * Fix Python destination * open() was not called in every time_reopen() * python destination is not defined in stats output * Fix processed stats counter for afsocket * Fix stats source for pipes * Previously pipe source is shown as file * Fix csv-parser multithreaded support In some cases (when csv-parser attached to network source), the parser randomly filled the column macros with garbage. * Fix a message loss in case of filesource when syslog-ng was restarted and the log_msg_size > file size. * Fix a potential crash in cryptofuncs * Fix a potential crash in syslog-ng-ctl when no command line parameters was set. * Fix token duplication in the output of '--preprocess-into' * Fix UTF-8 support in syslog-ng-ctl * Fix a potential crash during X.509 certificate validation. * Fix a segfault in Python module startup * Fix a possible endless reading loop issue in case of multi-line filesource. * Fix soname for the http module from "curl" to "http" * Avoid openssl 1.1.0 deprecated APIs When openssl is built with --api=1.1 disable-deprecated, use of deprecated APIs results in build failure. Other changes * Increase processed counter by queued counter after reload or restart when diskqueue is used otherwise the newly added written counter would underflow. * Set the default time-zone to UTC for elasticsearch2 Elasticsearch and Kibana use UTC internally. * Add retries support for python destination * Prefer server side cipher suite order * Always include librabbitmq in the dist tarball * Always include ivykis in the dist tarball * Marking parse error locations with >@<. * Default log_msg_size is increased to 64Kbyte from 8Kb * Tons of syslog-debun improvements * Exit with 0 return code when --help is specified for syslog-ng-ctl * syslog-ng: make '--preprocess-into' foreground only * Add debug messages on log_msg_set_value() * Add more detail to filter evaluation related debug messages Notes to the Developers * Extract template perf test function to testlib * Print a debug message when logmsg passed to the Python side * Allow http module (curl) to be build with cmake * astylerc: allow continuation lines to start until column 60 * Move kv-scanner under syslog-ng/lib * scratch-buffers2: implement an alternative to current scratch buffers This new API is aimed a bit easier to use in situations where a throw away buffer is needed that will automatically be freed at the next message. It also gets does away with GTrashStack that is deprecated in recent glib versions. * Several refactors in stats module. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, eroen, Fabien Wernli, Gabor Nagy, Gergely Nagy, Jason Hensley, Laszlo Varady, Laszlo Budai, Mate Farkas, Noemi Vanyi, Peter Czanik, Peter Gervai, Todd C. Miller, Philip Prindeville, Zoltan Pallagi https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.10.1

1 0

Why Syslog-NG can't start?
by Jason Long 19 Jun '17

19 Jun '17

Hello. I installed Syslog-NG on CentOS 7 x64 and my configuration is : @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); ts_format(iso); encoding(UTF-8); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; destination d_separatedbyhosts{file("/var/log/$HOST/messages" owner("root") group("root") perm(0655) dir_perm(0755) create_dirs(yes)); }; log { source(s_netsyslog); destination(d_separatedbyhosts); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: But my Syslog-NG can't start and give me below error: # systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Mon 2017-06-19 05:33:42 EDT; 32s ago Docs: man:syslog-ng(8) Process: 20920 ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid (code=exited, status=1/FAILURE) Main PID: 20920 (code=exited, status=1/FAILURE) Status: "Starting up... (Mon Jun 19 05:33:41 2017" Jun 19 05:33:41 localhost.localdomain systemd[1]: Unit syslog-ng.service entered failed state. Jun 19 05:33:41 localhost.localdomain systemd[1]: syslog-ng.service failed. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Jun 19 05:33:42 localhost.localdomain systemd[1]: start request repeated too quickly for syslog-ng.service Jun 19 05:33:42 localhost.localdomain systemd[1]: Failed to start System Logger Daemon. Jun 19 05:33:42 localhost.localdomain systemd[1]: Unit syslog-ng.service entered failed state. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service failed. Jun 19 05:33:42 localhost.localdomain systemd[1]: start request repeated too quickly for syslog-ng.service Jun 19 05:33:42 localhost.localdomain systemd[1]: Failed to start System Logger Daemon. Jun 19 05:33:42 localhost.localdomain systemd[1]: syslog-ng.service failed. How can I solve it? Thank you.

2 1

group name being a function of the $HOST macro?
by syslog-ng＠v3-1347.vlinux.de 18 Jun '17

18 Jun '17

Hello all, presently, I have a generic (so-to-say "zero configuration") destination in my syslog-ng.conf, destination d_host_month_facility_level {file("/var/log/$HOST/$YEAR-$MONTH.$FACILITY.$LEVEL"); }; , which works fine for my purposes. Presently, I have some scripts, processing log data, which have to run under "root", since the syslog files are all created with default owner("root") and perm("600"). I'd like to have such scripts running under less privileged accounts, that are member of a log source specific group. My first attempt was a destination d_host_month_facility_level { file("/var/log/$HOST/$YEAR-$MONTH.$FACILITY.$LEVEL" group("syslog_$HOST_r") perm(0640)); }; , but results in an error Error resolving group; group='syslog_$HOST_r' , which shows, that in the context of a "group()" parameter, the $HOST macro seem to be not evaluated. Does anybody here see another means, that would allow me to achieve something like the above without going away from my beloved generic single destination specification? Ideally, what I would like to achieve further to get an abstraction away from the hostname itself, is to have a mapping from $HOST to something that I can specify in a kind of conditional statement like if $HOST == "my_box" then groupname="user_machine_log_reader" else if $HOST == "some_of_my_routers" then groupname="wan_log_reader" else if $HOST == "some_other_of_my_routers" then groupname="firewall_log_reader" else groupname="$HOST" and finally, of course, a generic 'group("$groupname")' parameter to get it into position. Any ideas? Best regards and thanks.

1 0

Issue with timestamp
by Andrew 18 Jun '17

18 Jun '17

Hi I am new to syslog-ng and logging in general so apologies but I am having trouble understanding timestamps. I am trying to extract the timestamp inside the message and create a field called 'timestamp' for example. The message looks like this: "MESSAGE":"06:17-16:21:21 fw-1-1 ulogd[9300]: id=\"2001\" severity=\"info\" sys=\"SecureNet\" sub=\"packetfilter\" name=\"Packet dropped\" action=\"drop\" fwrule=\"60002\" initf=\"eth0\" outitf=\"eth1\" srcmac=\"4c:00:82:8a:f4:11\" dstmac=\"00:1a:8c:f0:3c:40\" srcip=\"1.1.1.1\" dstip=\"2.2.2.2\" proto=\"17\" length=\"143\" tos=\"0x00\" prec=\"0x00\" ttl=\"125\" srcport=\"51941\" dstport=\"161\" How can I create a field called 'timestamp' which contains the timestamp in the message but also adds the year to it and puts it into a better format? I am also using the kv-parser and json output looks good I have pairs of everything from the message but I also need a pair called timestamp="Jun 16 2017 16:21:21' for example. Hope I'm making sense here. Thanks for any help.

4 5

documentation for 3.10 available
by Czanik, Péter 16 Jun '17

16 Jun '17

Hi, Documentation for the upcoming syslog-ng 3.10 version is now available. If you want to check out what to expect in the next release, the summary of changes is at https://www.balabit.com/documents/syslog-ng-ose-3.10-guides/en/syslog-ng-os… If you also want to test the new features, I have RPM packages available compiled from git head snapshots. Installation instructions are available at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Insider 2017-06: HEPIX; Python parser; Turris; openSUSE; upcoming events;
by Czanik, Péter 15 Jun '17

15 Jun '17

Dear syslog-ng users, This is the 59th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS HEPIX and syslog-ng ------------------- One of the most important discoveries of this decade was the Higgs boson. But researchers at High Energy Physics and Nuclear Physics laboratories and institutes would have been unable to find the Higgs boson without the IT staff maintaining the computer infrastructure collecting and analyzing the massive amount of data generated during their experiments. HEPIX is a community that brings together these IT guys twice a year from around the world. This spring their event was hosted by the Wigner Research Centre for Physics in Budapest, where I also talked to faxmodem. He is both a HEPIX member and one of the most active syslog-ng community members: https://www.balabit.com/blog/helped-turning-higgs-boson-theory-reality/ Process trade data using Python ------------------------------- Upcoming syslog-ng Open Source and Premium Edition releases will feature a Python parser. Learn how to utilize this parser from our webinar. In the demo FIX data is transformed into easy-to-use name-value pairs, but the solution itself is not limited to trade data. You can access the recording of the webinar at https://pages2.balabit.com/fix-data-parser/ (requires registration). syslog-ng updated on Turris Omnia --------------------------------- The release candidate of version 3.7 of Turris OS – the OpenWRT-based operating system of Turris Omnia routers – is now available. Among many other changes, this updates syslog-ng from version 3.0 to 3.9. In other words, it adds about seven years’ worth of new syslog-ng features, including new parsers, filters, formatting options, destinations, and performance enhancements. https://www.balabit.com/blog/using-the-latest-syslog-ng-on-turris-omnia/ openSUSE conference 2017 ------------------------- Being a long-time openSUSE user, I visit the openSUSE conference not only to present on one of its components – syslog-ng – but also to meet friends and learn about new technologies and the plans for the upcoming year. Some talks, like those about Package Hub, were very interesting and important also from a syslog-ng perspective. Of course, I also joined a few talks for my personal interest, like the one on the new ARM devices supported by openSUSE. Read more about my experiences at https://www.balabit.com/blog/friends-syslog-ng-package-hub-arm-opensuse-con… UPCOMING EVENTS The next event where you can learn about syslog-ng: Libre Software Meeting (July 1-7): https://2017.rmll.info/en/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Changing message priority
by To Mitz 15 Jun '17

15 Jun '17

Hi, I am trying to use rewrite to change a message's PRIORITY. I am able to change the HOST to "info" using: rewrite r_rewrite_set{set("info", value("HOST") condition(filter(f_logstash)));}; But, when I try setting the PRIORITY to "info", using this: rewrite r_rewrite_set{set("info", value("PRIORITY") condition(filter(f_logstash)));}; But, when I try setting the PRIORITY to "info", using this: Error parsing rewrite expression, syntax error, unexpected LL_NUMBER, expecting LL_IDENTIFIER or LL_STRING in /etc/syslog-ng/f_messages.conf at line 8, column 27:included from /etc/syslog-ng.conf line 18, column 34rewrite r_rewrite_set{set(3, value("PRIORITY") condition(filter(f_logstash)));};^ Please advise on how I can fix this. Thanks.

3 2

date-parser not found
by Sandbox 13 Jun '17

13 Jun '17

Hi, I would like to use the date-parser. When i using the below parse it says "date-parser not found": parser haproxy_date { date-parser( format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") );. }; Plugin module not found in 'module-path'; module-path='/usr/lib/syslog-ng/3.5.6', module='date-parser' Error parsing parser expression, parser plugin date-parser not found in /etc/syslog-ng/conf.d/20-haproxy.conf at line 26, column 5: included from /etc/syslog-ng/syslog-ng.conf line 165, column 1 date-parser( ^^^^^^^^^^^ It's weird, because when i try to use the date-parser module in the log section: log { source(s_haproxy); filter(f_haproxy_info); date-parser(format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") ); destination(d_haproxy_info); }; it says: Error parsing log, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/conf.d/20-haproxy.conf at line 37, column 5: included from /etc/syslog-ng/syslog-ng.conf line 165, column 1 date-parser(format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") ); ^^^^^^^^^^^ In OSE version the date-parser module is not available, or i'm missing something? Thanks, Robert

3 3

testing new features, poll
by Czanik, Péter 13 Jun '17

13 Jun '17

Hi, syslog-ng 3.10 is coming soon. If you would like to test it and use one of the RPM-based distributions, there are some ready to use packages to get you started: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ I'd also appreciate if you could answer my quick poll on twitter which is your most waited feature in the upcoming syslog-ng version: https://twitter.com/PCzanik/status/874527761532104704 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

pdbtool Segmentation fault
by Evan Rempel 10 Jun '17

10 Jun '17

I am running a relatively recent githead (April 17) of 3.9.1. I have a single pattern that results in a segmentation fault when running pdbtool test --validate patterndb.xml patterndb can be found at http://web.uvic.ca/~erempel/syslog-ng/patterndb.xml As it stands now I realize that the pattern does NOT match the test, but it should not segfault. A single space removed from the pattern makes the test succeed. The working patterndb is at http://web.uvic.ca/~erempel/syslog-ng/patterndb2.xml Any help would be appreciated.

1 0

Multiple dates
by Sandbox 09 Jun '17

09 Jun '17

Hi, I just started to test, learn etc syslog-ng, my server configuration is really basic: Q: Can i filter (and mark them on client) the incoming logs, so i dont have to open multiple ports for different logs? #-----------------------------------Source--------------------------------- source s_network_testweb01_access { tcp(ip(192.168.56.48) port("16601")); }; source s_network_testweb01_error { tcp(ip(192.168.56.48) port("16602")); }; source s_network_testweb01_referer { tcp(ip(192.168.56.48) port("16603")); }; #------------------------------------Destination-------------------------- destination d_apache_access_testweb01 { file("/var/log/syslog-ng/apache2/$FULLHOST.access" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_access")); .... file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.access.$R_DAY" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_access_archive")); }; destination d_apache_error_testweb01 { file("/var/log/syslog-ng/apache2/$FULLHOST.error" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_error")); .... file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.error.$R_DAY" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_error_archive")); }; destination d_apache_referer_testweb01 { file("/var/log/syslog-ng/apache2/$FULLHOST.referer" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_referer")); .... file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.referer.$R_DAY" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_referer_archive")); }; #---------------------Connections-------------------------- log { source(s_network_testweb01_access); destination(d_apache_access_testweb01); }; log { source(s_network_testweb01_error); destination(d_apache_error_testweb01); }; log { source(s_network_testweb01_referer); destination(d_apache_referer_testweb01); }; The stored log: Q: Why does it store the date 3 times in every logs? Jun 8 08:20:11 192.168.7.30 133 <13>1 2017-06-08T08:20:11+02:00 testweb01 - - - [meta sequenceId="24"] :1 - - [08/Jun/2017:08:20:10 +0200] "GET / HTTP/1.1" 200 3004 Also my client configuration: #Log sources source s_apache_error { file("/var/log/apache2/error_testweb_log" follow-freq(1)); }; source s_apache_access { file("/var/log/apache2/access_testweb_log" follow-freq(1)); }; source s_apache_referer { file("/var/log/apache2/www_referer_log" follow-freq(1)); }; #LOG Destination destination d_network_access { syslog("192.168.56.48" transport("tcp") ip-protocol(4) port("16601") persist-name("testweb_access")); }; destination d_network_error { syslog("192.168.56.48" transport("tcp") ip-protocol(4) port("16602") persist-name("testweb_error")); }; destination d_network_referer { syslog("192.168.56.48" transport("tcp") ip-protocol(4) port("16603") persist-name("testweb_referer")); }; #LOG connections log { source(s_apache_access); destination(d_network_access); }; log { source(s_apache_error); destination(d_network_error); }; log { source(s_apache_referer); destination(d_network_referer); }; Thanks, Robert

2 4

very basic router
by Thufir Hawat 09 Jun '17

09 Jun '17

for this router: http://www.netgear.com/support/product/WNR1000v2 I don't see any option in the configuration for using syslog. Is there a work-around? The only way to grab logs is to either copy/paste or e-mail -- but the e-mail uses port 25, which is blocked. thanks, Thufir

2 4

admin guide in PDF
by Thufir Hawat 09 Jun '17

09 Jun '17

I can't seem to find the admin guide as a PDF. It's just html? thanks, Thufir

2 2

Re: [syslog-ng] Filter Not Working (too many or's?)
by james.r.hendrick 09 Jun '17

09 Jun '17

Not exactly what you asked but I have used netmask to simplify some filters. Although it probably won't be able to solve your problem. Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Szalai, Attila" <Attila.Szalai(a)morganstanley.com> Date: 6/6/17 8:07 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Filter Not Working (too many or's?) Hi, First of all, the content of the host() is a regular expression, so adding .* to the beginning and/or to the end of the expression adds nothing, just pain/slowness. Second, it would help a lot if we can see the actual error message. I found no obvious mistake, but because this is not the original line, maybe something lost in the translation. From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of wiskbroom(a)hotmail.com Sent: Tuesday, June 06, 2017 12:59 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] Filter Not Working (too many or's?) Here is an example of what I am trying to do, these hostnames are not real; the real ones have no common pattern. filter f_xyz { host("*.abca.*") or host(".*abcb.*") or host(".*abcc.*") or host(".*abcd.*") or host(".*abce.*") or host(".*abcf.*") or host(".*abcg.*") or host(".*abch.*"); }; The filter above is for any host containing a hostname with what is contained within the .* and *.; i.e. hostabca01 will be matched by host("*.abca.*") When I have this filter in my config, syslog fails to restart. Eyes hurt, obvious mistake? NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.

5 4

Duplicate header ?
by Scot 08 Jun '17

08 Jun '17

I seem to be getting a duplicate host header in my udp syslog input where the IP is printed twice. IP/IP any ideas where it comes from ? Same result for either of these. #source s_net_udp {udp(ip(0.0.0.0) port(514) keep_hostname(yes) so_rcvbuf(262142));}; #source s_net_udp {syslog(ip(10.189.252.62) port(514) transport("udp") flags(no-hostname) so_rcvbuf(262142));}; Jun 8 13:55:21 *192.168.10.10/192.168.10.10 <http://192.168.10.10/192.168.10.10> * fw-aplha %ASA-4-106 ..............

3 3

Filter Not Working (too many or's?)
by wiskbroom＠hotmail.com 06 Jun '17

06 Jun '17

Here is an example of what I am trying to do, these hostnames are not real; the real ones have no common pattern. filter f_xyz { host("*.abca.*") or host(".*abcb.*") or host(".*abcc.*") or host(".*abcd.*") or host(".*abce.*") or host(".*abcf.*") or host(".*abcg.*") or host(".*abch.*"); }; The filter above is for any host containing a hostname with what is contained within the .* and *.; i.e. hostabca01 will be matched by host("*.abca.*") When I have this filter in my config, syslog fails to restart. Eyes hurt, obvious mistake?

3 3

RPM packages from syslog-ng git head
by Czanik, Péter 06 Jun '17

06 Jun '17

Hi, Test the latest syslog-ng features and bug fixes! They are available for RPM platforms for easy installation. I build packages from git head snapshots regularly for the following Linux distribution versions: openSUSE SLES 12 Fedora RHEL/CentOS 7 You can read how to get started with these packages at: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Do stats cause memory leak
by Evan Rempel 06 Jun '17

06 Jun '17

We are running git head download post 3.9.1 and the memory footprint seems to continually grow. Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195 It it possible that all of these statistics take up a non-trivial amount of RAM? USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 10139760 ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng That is a 12GB footprint after 1 month of uptime. Evan.

4 8

docker based, local builds
by Scheidler, Balázs 05 Jun '17

05 Jun '17

Hi, I've just submitted a PR that makes it easy to build rpms and debs locally, out of a freshly checked out syslog-ng tree and/or a release tarball. If you download a release tarball (one that contains debian packaging files), all you need to build it to Ubuntu or Debian is: # this one builds the Ubuntu build image locally $ dbld/rules image-zesty $ dbld/rules deb-zesty But rpm-centos7 and deb-jessie also work. Check out the help screen for dbld/rules and the PR. This should make it pretty trivial to get binaries that you can deploy in production. This is just a bare-bones stuff, later on support can be added to a lot more distros. If you want to use it in a freshly cloned copy of syslog-ng, checked out from github, that will not have debian packaging (at least for now), you'll need to merge the "release" branch in order to get them. # this one gets the debian/rpm packaging locally $ git clone git@github.com:balabit/syslog-ng.git $ cd syslog-ng $ git merge origin/release $ dbld/rules image-centos7 $ dbld/rules rpm-centos7 I still don't like that the "release" branch lives separate, but I can't change that without discussing it with our Debian maintainer first. -- Bazsi

1 0

syslog 1024 byte limit work around
by Corey Davelaar 31 May '17

31 May '17

Hello, I'm new to the group and hopefully it's not a repeat question. I was guided to this mailing list by a balabit tech support rep. We use splunk on the back end of syslog-ng and we have some syslog sources that utilize very long syslog messages, that end up being over 1024 bytes. We switched from udp to tcp as the transport and now we are seeing the tail end of the syslog messages, but splunk gets confused by the inputs and treats each line as a separate syslog message and completely unrelated to the previous one, because that’s how syslog-ng is treating it. I was wondering if there is a way to tell syslog-ng that if a new packet doesn’t start with a standard syslog timestamp to treat it as a continuation of the previous message or somehow combine it with a previous message, utilizing a buffer type of setting. Thanks, Corey

3 5

Sawmill and Syslog-NG template() string
by Francesco Rolando 31 May '17

31 May '17

Hi all, I'm working with Sawmill log collector that it's able to import the Syslog-NG logs. I have to export logs with a specific template() to have them corretly imported into Sawmill. Here the regular expression used to match syslog-ng lines inside the Sawmill parser (that seem coded by "BalaBit IT Security"): '^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])T([0-9][0-9]:[0-9][0-9]:[0-9][0-9])[-+][0-9][0-9]:*[0-9][0-9] ([^ ]+) ([^ ]+) ([^ ]+) (.*)$' where matches are imported as: 1- date 2- time 3- logging_device 4- syslog_message_type 5- syslog_priority 6- v.syslog_message I have partially re-create the template() string but I still have a couple of doubts: template("${YEAR}-${MONTH}-${DAY}T${HOUR}:${MIN}:${SEC}+??:?? ${SOURCEIP} ${PROGRAM} ${PRIORITY} ${MSG}\n") Make sense $PROGRAM as "message type"? And do you have any ideas for the question marks? Any help is appreciated. Thank you. -- Saluti, Francesco.

2 2

Insider 2017-05: osquery; SELinux; disk queue; upcoming events
by Czanik, Péter 26 May '17

26 May '17

Dear syslog-ng users, This is the 58th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS osquery and syslog-ng --------------------- osquery allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. From this post, you will learn how to send log messages to osquery, read osquery logs using syslog-ng, and how to parse the JSON-based log messages of osquery, so selected fields can be forwarded to Elasticsearch or other destinations expecting name-value pairs. https://www.balabit.com/blog/endpoint-visibility-and-monitoring-using-osque… SELinux in enforcing mode ------------------------- Security-Enhanced Linux (SELinux) is a set of kernel and user-space tools enforcing strict access control policies. It is also the tool behind at least half of the syslog-ng problem reports. SELinux rules in Linux distributions cover all aspects of the syslog-ng configuration coming in the syslog-ng package available in the distribution. But as soon as an unusual port number or directory name is specified in the configuration, syslog-ng fails to work even with a perfectly legitimate configuration. While preventing unusual access is the main feature of SELinux, it also causes lots of headaches for unsuspecting administrators. Learn how you can use syslog-ng with SELinux in enforcing mode. https://www.balabit.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/ Processing messages stuck in the disk queue files ------------------------------------------------- When you change the configuration of a syslog-ng host that uses disk-based buffering (also called disk queue), syslog-ng may start new disk buffer files for the destinations that you have changed. Learn how you can flush log messages from the orphaned disk queue files from our new document: https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-dis… UPCOMING EVENTS You can learn about syslog-ng at a growing number of events: Big Data Universe: https://bdu.hu/ openSUSE conference: https://events.opensuse.org/conference/oSC17 Libre Software Meeting: https://2017.rmll.info/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 2

Re: [syslog-ng] Syslog-ng input for beats ? [SUMMARY01]
by Scot 26 May '17

26 May '17

Hi All, Just wanted to report my progress on this subject, and say thanks for your input. I'm pretty happy with this setup so far I'll try to keep it short but wanted to share. *tl;dnr* HOWTO send Windows log data through syslog-ng and get it perfectly parsed in splunk and elasticsearch. *Purpose*: Funnel Windows events from enterprise or dmz zones through an remote aggregation relay into syslog-ng, allowing syslog-ng to replicate and broker log data based on business needs. *Components*: *Beats*- Winlogbeat is one of the beats agents by Elastic.co (Elasticsearch) that run on windows hosts to offload events to Logstash in this case. *Logstash*: Like beats it is able to take plugin based input and convert it to a plugin based output. I use it like a syslog relay but for compressed json streams. *syslog-ng:* Primary data aggregation and rule based distribution to SPLUNK,ES and Archive file. Beats support plugin based outputs, the primary two are es and logstash. This solution uses a logstash instance running on the same VM as my rsyslog relay. The output from winlogbeat, running on all windows servers, is a compressed json data stream to its designated logstash relay IP:PORT. This looks like this winlogbeat winlogbeat winlogbeat ---> TCP:5044 --> Logstash --> TCP:9150 --> syslog-ng winlogbeat winlogbeat Systems local to the syslog-ng server also go through a logstash hop. Logstash ads a single aggregator tag to each json message and can also take rule based actions like VictorOps, or massage data inline. syslog-ng then directs a copy of the json stream to a splunk forwarder, archive file, elasticsearch and any other output supported by Balabit. *Advantage*: I can write rules to tier my data to best of class or lower cost data analytic solutions or even new ones with a syslog-ng. *sylog-ng.conf * # This listens for data from logstash output. source s_BEATS {network(port(9150) log-msg-size(65536) flags(no-parse));}; # Define the output to splunkforwarder. destination d_splunk_BEAT {network("splunkforwarderhost" template("$(format-json --scope nv-pairs -p @timestamp=${ISODATE} -x MESSAGE)\n") port(5140));}; # Send a copy to SPLUNK or filter f_splunk_beat log { source(s_BEATS); parser {json-parser();}; destination(d_splunk_BEAT); }; *logstash beats-pipeline.conf output to syslog-ng* output { tcp { host => "syslog-ng host" port => "9150" mode => "client" codec => "json_lines" } }

2 1