- syslog-ng - lists.balabit.hu

Insider 2017-08: 3.11 released; wildcard-file; docker; SCL; RMLL/LSM;
by Czanik, Péter 02 Aug '17

02 Aug '17

Dear syslog-ng users, This is the 60th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.11 released ----------------------- The latest version of syslog-ng, 3.11 is now available. The most important new feature is the GeoIP2 parser which builds on libmaxminddb providing both better performance and more detailed geographical information about IP addresses. AMQP destination now supports SSL. There are many more smaller features and bug fixes. For a complete list check the release announcement: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.11.1 Reading multiple files: wildcard file source ------------------------------- Starting with version 3.10, syslog-ng can collect messages from multiple text files. You do not have to specify file names one by one, just use a wildcard to select which files to read. This is especially useful when you do not know the file names by the time syslog-ng is started. This is often the case with web servers with multiple virtual hosts. https://www.balabit.com/blog/reading-multiple-files-wildcard-file-source-sy… Collecting Docker infrastructure logs ------------------------------------- Why use syslog-ng for collecting Docker logs? Docker already provides many drivers for logging, even for central log collection. On the other hand remote logging drivers arrive with a minimalist feature set and you are not able to use the “docker logs” command any more. To have the best of both worlds, you can use journald logging driver in Docker and use syslog-ng to read Docker logs from journald and forward log messages to your central log server or other destinations. You can even run syslog-ng itself in a Docker container, so you can use it on dedicated Docker host environments as well where it is not possible to install additional applications. https://www.balabit.com/blog/collecting-docker-infrastructure-logs-using-sy… The power of SCL ---------------- The syslog-ng configuration library (SCL) can help you to configure syslog-ng a lot more easily. These configuration snippets can hide away the complexity of collecting, parsing or storing log messages. >From this blog you can learn how to parse web server logs and store the results at a Logging as a Service (LaaS) provider in a structured form. You will use SCL both for message parsing and the LaaS destination, and also utilize the wildcard-file() source introduced in syslog-ng 3.10. https://www.balabit.com/blog/parsed-web-server-logs-cloud-syslog-ng-scl/ RMLL / Libre Software Meeting 2017 ---------------------------------- This year I participated again in the security track of the largest French open source conference, Libre Software Meeting (RMLL). “Participated” as I did not only give a talk on syslog-ng there, but also sat in to most of the presentations and had very good discussions both with visitors and fellow speakers. The organizers brought together talks from diverse IT security related fields, a very good opportunity for cross-pollination of ideas. https://www.balabit.com/blog/czp-rmll-libre-software-meeting-2017/ NEW RELEASES syslog-ng OSE 3.11.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.11.1 Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

OSE 3.11
by Laszlo Budai 01 Aug '17

01 Aug '17

3.11.1 Features * Add geoip2 parser and template function. It is based on the libmaxminddb(MaxMindDB). It will replace the old geoip parser and template function, so they are deprecated from 3.11 (but still available). * Add SSL support to AMQP. * Add template option to apache-accesslog-parser. * Add configurable event time to Riemann destination. * Add drop-unmatched() option to dbparser. * Add Ubuntu Xenial to the bundled docker images. * Support multi-instance support for Solaris 10 and 11. * Support multi-instance for systemd. * Add configurable timeout to HTTP destination. * Add prefix() option to cisco-parser. Bugfixes * Fix a memory usage counter underflow for threaded destination drivers and writers. * Fix a potential crash in AMQP. * Fix a potential crash during reload. * Fix a reload/shutdown issue. Under heavy load, worker might never exit from the fetch loop from the queue. * Fix a potential crash in afsocket destination during reload. * Fix a counter registration bug. In some cases not all the required counters are registered. * Fix a build issue on FreeBSD. * Fix a memory leak in diskq plugin. * Fix systemd-journal error codes validation. * Fix a potential crash in diskq when it is used with file destination and the file is reaped. * Fix a memory leak in HTTP destination * Fix ENABLE_DEBUG in dbparser. * Fix a unit tests that caused build issue on 32 bit platforms. Other changes * The eventlog library is part of syslog-ng from now. * Improve error messages when the config cannot be initialized. * Improve source suspended/resumed debug messages. * Rename syslog-debun to syslog-ng-debun. * Update manpages to v3.11 * Remove tgz2build directory. Notes to the Developers * Rewrite merge-grammar script from Perl to Python. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Fabien Wernli, Gabor Nagy, Giuseppe D'Anna, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Lorand Muzamel, Mate Farkas, Noemi Vanyi, Peter Czanik, Tamas Nagy, Tibor Bodnar, Tomasz Kazimierczak, Zoltan Pallagi

1 0

eStreamer src Integration
by Scot 01 Aug '17

01 Aug '17

Has anyone looked at sending Cisco eStreamer events to syslog-ng ? We have a couple Cisco Firepower management centers and I would rather use syslog-ng over sending directly to splunk so that we may use other integrations like elastic and our NMS. I have the eStreamer SDK on my syslog-ng server and wondered if anyone else has worked on this. Search of the user archive says no. Thanks Scot

3 3

syslog-ng poll: geographical information
by Czanik, Péter 01 Aug '17

01 Aug '17

Hi, One of the major features of the upcoming syslog-ng 3.11 will be a complete rewrite of GeoIP support. So this time I ask you in the poll if you use syslog-ng to look up geographical information for IP addresses: https://twitter.com/PCzanik/status/891985964788330497 If you use GeoIP, please include your use case in the comments or answer this e-mail. You can e-mail me directly, if for some reason you don't want to share it with the rest of the world. Please re-tweet the above tweet to reach as many syslog-ng users as possible! Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Upgrade from 3.10.1-1 to 3.11.1-1 fails on Ubuntu
by Clayton Dukes 31 Jul '17

31 Jul '17

I got this today on one of our Customer's systems, anyone else had this happen? Unpacking syslog-ng (3.11.1-1) over (3.10.1-1) ... dpkg: error processing archive /var/cache/apt/archives/syslog-ng_3.11.1-1_all.deb (--unpack): trying to overwrite '/lib/systemd/system/syslog-ng.service', which is also in package syslog-ng-core 3.11.1-1 Errors were encountered while processing: /var/cache/apt/archives/syslog-ng_3.11.1-1_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) [cid:image001.png@01D3078B.AD39C7D0] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D3078B.AD39C7D0]<https://twitter.com/logzilla>[cid:image003.png@01D3078B.AD39C7D0]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D3078B.AD39C7D0]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps!

4 8

Few messages are not getting forwarded to destination
by Harish Shetty 28 Jul '17

28 Jul '17

Hi I am using "syslog-ng-3.9.1-1.el6.x86_64" . Few messages are not getting forwarded to destination (Monitoring hosts). But messages are receiving in syslog-ng server from the client. As a tuning parameter I am using log_fifo_size (20000); flush_lines (0); time_reopen (10); For source s_tcp , source s_udp and source s_net so_rcvbuf(8388608) max-connections(200) log-iw-size(20000) Could you please let me know , how we can avoid the messages are not getting forwarded ti destination. Regards Harish Shetty

2 1

unknown symbol errors in jounalctl -xe
by vijay amruth 26 Jul '17

26 Jul '17

Hello All, hope everyone is doing great, I am unable to start syslog-ng, I see this messages... Any help is appreciated. Jul 25 11:36:14 server-1systemd[1]: Unit syslog.socket entered failed state. Jul 25 11:36:14 server-1systemd[1]: Unit syslog-ng.service entered failed state. Jul 25 11:36:14 server-1systemd[1]: syslog-ng.service failed. Jul 25 11:36:18 server-1kernel: dccp_diag: Unknown symbol dccp_hashinfo (err 0) Jul 25 11:36:18 server-1kernel: dccp_diag: Unknown symbol dccp_hashinfo (err 0) Jul 25 11:36:48 server-1systemd[1]: Listening on Syslog Socket. [root@server-1 tmp]# systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: activating (start) since Tue 2017-07-25 11:44:20 GMT+7; 36s ago Docs: man:syslog-ng(8) Main PID: 21822 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─21822 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid --fd-limit 50000 Jul 25 11:44:20 lvsinfosecsplunkcollector-1137760 systemd[1]: Starting System Logger Daemon... [root@server-1 tmp]# syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: date,json-plugin,syslogformat,add-contextual-data,affile,afprog,confgen,cryptofuncs,pseudofile,sdjournal,csvparser,graphite,cef,afmongodb,basicfuncs,system-source,afuser,dbparser,afstomp,afsocket,disk-buffer,kvformat,afamqp,linux-kmsg-format Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off -- Thanks, Vijay Amrut.

2 2

Windows Events multi-line
by Smith, Paul (Sr. Admin-InfoSec) 25 Jul '17

25 Jul '17

Hi Nagy, Thanks for checking that. That was probably one of the reasons it did not work. That is too bad since Windows servers using the syslog-ng agent are the source. I would have thought that would be working since bot products are from Balabit. Windows logs also send a lot of garbage with Information lines that it would also be good to remove since that is just going to fill up the log files with junk. I am getting 8-10 gigs per day from a single Windows Domain Controller and I need to add in about 25 more. The log file would be immense. I tested out using the syslog driver and was poking around with the multi-line-prefix and multi-line-garbage and they don't give any syntax errors, as long as I don't use the multi-line-mode option. Does syslog-ng ignore these options if the multi-line-mode option is not specified? Basically, do all the multi-line-xxx options not work like this? It seems like the only option that works for syslog or network is the flags(no-multi-line) flag. Is that the case? Are there any other alternatives? I would hate to write the syslogs to a file and then parse that file again to get the logs I want. That is double work for the server. But that seems like that might be the only option I have. Hmmm...maybe I can log to a VM syslog-ng and have that one parse the data first using the file driver and then I can relay to my main systems. Paul C Smith -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Thursday, July 13, 2017 7:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 147, Issue 14 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: Windows Events multi-line (Nagy) ---------------------------------------------------------------------- Message: 1 Date: Thu, 13 Jul 2017 10:50:03 +0200 From: Nagy, Gábor <gabor.nagy(a)balabit.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Windows Events multi-line Message-ID: <CAETAYnCft3KAyWF_5pB3g4+jodrn6+SpXTYJqqyfnFoUFzzaEw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello, Checking your configuration I see that you are using multi-line mode for a network source driver. Unfortunately, currently this is not supported, only the file() and pipe() source drivers support multi-line messages. I have checked the documentation and found that although it is mentioned which source drivers support multi-line mode (under flags option "no-multi-line"), the multi-line-* options are included in the network source driver options page. This is a documentation bug, which will be fixed soon. Sorry for the inconveniences. Best Regards, Gabor On Mon, Jul 10, 2017 at 8:15 PM, Smith, Paul (Sr. Admin-InfoSec) < Paul.C.Smith(a)snapon.com> wrote: > Hello, > > > > I’m having problems getting Windows events on a single line on > syslog-ng OSE. I’ve scoured the interwebs and not found what I need to > get this exact. I am guessing this is not an uncommon problem but I > can’t seem to find quite what I need. I am guessing I am just missing > some simple thing here. > > > > Here are my details. > > > > Using syslog-ng OSE 3.9.1 > > > > Have syslog-ng Windows Agent 6.0.6 running on Windows 2012 server > > > > Have a tcp source and that writes direct to the log file. Works fine > with no options set. > > > > Getting multiple lines per event. > > > > I’ve added what I think are the correct settings for multi-line, it > does not work. I don’t think it is the regex syntax, but something else? > > > > ___________ > > Syslog.ng.conf > > > > source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") > > multi-line-mode(regexp) > > multi-line-prefix("^[0-9]{3,5} > \s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][ > 0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") > > flags(no-parse)); }; > > > > ___________ > > Here is the error when trying to start syslog-ng or run syslog-ng –s: > > > > Error parsing afsocket, syntax error, unexpected LL_IDENTIFIER, > expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or > KW_DNS_CACHE in /etc/syslog-ng/syslog-ng.conf at line 38, column 76: > > > > source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") > multi-line-mode(regexp) multi-line-prefix("^[0-9]{3,5} > \s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][ > 0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") > flags(no-parse)); }; > > > ^^^^^^^^^^^^^^^ > > ___________ > > Sample of the event log part I am matching regex on: > > > > Jul 10 12:11:19 x.x.x.x 912 <133>1 2017-07-10T12:11:18-05:00 > computername Microsoft_Windows_security_auditing. 6260 - [win(a)18372.4 > EVENT_CATEGORY="Logoff" EVENT_FACILITY="16" EVENT_ID="4634" EVENT_LEVEL="0" > EVENT_NAME="Security" EVENT_REC_NUM="573516592" EVENT_SID="N/A" > EVENT_SOURCE="Microsoft Windows security auditing." EVENT_TASK="Logoff" > EVENT_TYPE="Success Audit" EVENT_USERNAME="domain\\userid"][meta > sequenceId="10817278" sysUpTime="-198876"] domain\userid: Security > Microsoft Windows security auditing.: [Success Audit] An account was > logged off. > > > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170713/826dd963/a…> ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng ------------------------------ End of syslog-ng Digest, Vol 147, Issue 14 ******************************************

2 1

positional arguments for blocks
by Fabien Wernli 21 Jul '17

21 Jul '17

Hi, Many native blocks have positional arguments, e.g. network(): network( "127.0.0.1", … ) Is it possible to set positional arguments to custom blocks?

2 1

Re: [syslog-ng] syslog-ng 3.10 fedora/rhel/centos packages
by alex 19 Jul '17

19 Jul '17

Hello, this is taken from syslog-ng 3.8.1 change log: "Minimal version of libdbi is set to 0.9.0" Correct me if I am wrong, but latest version of libdbi on CentOS 6.X is 0.8.3. How is syslog-ng 3.9 the latest compatible version with older CentOS distro? *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Czanik, Péter *Sent:* Wednesday, July 19, 2017 1:42 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng(a)lists.balabit.hu> *Subject:* Re: [syslog-ng] syslog-ng 3.10 fedora/rhel/centos packages Just tested, 3.10.1 does not compile on RHEL 6 (well, in practice the latest CentOS 6.X). It fails with: CCLD modules/dbparser/pdbtool/pdbtool ./lib/.libs/libsyslog-ng.so: undefined reference to `g_atomic_pointer_add' collect2: ld returned 1 exit status make[2]: *** [modules/dbparser/pdbtool/pdbtool] Error 1 make[1]: *** [all-recursive] Error 1 So, currently 3.9 is the latest syslog-ng compiling on old distros. Making sure, that syslog-ng compiles on old distros is on the roadmap of the team. Compiling on FreeBSD was fixed right after the 3.10 release, I hope RHEL 6 / SLES 11 will also compile again soon. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Mon, Jul 17, 2017 at 6:47 PM, Czanik, Péter <peter.czanik(a)balabit.com> wrote: Hi, I'll check RHEL 6 later this week. Personally I tested on SLES 11, as (open)SUSE is my primary testing platform. RHEL 6 might work, at least the later versions, as glib was updated. See https://github.com/balabit/ syslog-ng/issues/1403 for some background info. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Mon, Jul 17, 2017 at 5:12 PM, Evan Rempel <erempel(a)uvic.ca> wrote: RHEL 6 still has 2 years in the normal support lifecycle and even more if a extended support is purchased. IMHO syslog-ng should strive to have a current release that still compiles on these platforms until at least the end of standard support (2020). Evan. On 07/17/2017 07:36 AM, Czanik, Péter wrote: hi, Until now my unofficial syslog-ng 3.10 packages for Fedora / RHEL 7 / CentOS 7 were available from my "githead" repo. Today I created a dedicated 3.10 repository, as I plan to use the githead repo for creating builds from git HEAD again. It is available at https://copr.fedorainfracloud. org/coprs/czanik/syslog-ng310/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1