August 2017 - syslog-ng - lists.balabit.hu

RFC: incubator is to be discontinued
by Scheidler, Balázs 14 Aug '17

14 Aug '17

Hi, We had a discussion with our Debian maintainer about the fate of syslog-ng-incubator. For those of you who don't know, it is a set of syslog-ng modules not integrated to syslog-ng master for one reason or another. It is available from here: https://github.com/balabit/syslog-ng-incubator The problem with this repository is that it is not really maintained, causing compilation and functionality issues down the line. So our opinion is to go ahead and integrate stuff that can be brough up to speed. Once we've done what we could, we would discontinue the incubator altogether. Please speak up now, if you disagree, but please also be ready to help us out in maintenance in that case. If there are no disagreements in the following week, we would start to phasing out the incubator. We welcome volunteers to: 1) migrate code, OR 2) help with testing If you are the user of any of those plugins, letting us know at this point also helps making sure that no functionality is lost. Thanks. -- Bazsi

4 4

syslog parser modified in git head
by Scheidler, Balázs 10 Aug '17

10 Aug '17

Hi, The pull request #1617 has just been merged, which adds a slight modification to the way we parse RFC3164 syslog messages, by adding support for IPV6 addresses in the host field. Since changing that parser always makes me nervous (lot's of heuristics to support a number of different logging formats), it would be great if any of you having "strange" logging formats (appliances in general, Cisco, F5, Juniper, Huawei, etc) devices could validate that we didn't break anything major. The change is pretty focused, e.g. we accept an IPV6 address as a hostname if it matches the general format of IPv6 addresses. But the interpretation of the ':' character has slightly changed. This could cause hostnames and program names to be parsed incorrectly. Any feedback would be appreciated. -- Bazsi

1 0

Re: [syslog-ng] RFC: incubator is to be discontinued
by Balazs Scheidler 03 Aug '17

03 Aug '17

Thanks for the feedback. On Aug 2, 2017 21:42, "Czanik, Péter" <peter.czanik(a)balabit.com> wrote: Hi, As far as I can see, the kafka-c module is definitely in active use. I added Pyr -- the original author of the kafka module -- on CC. In addition I see once in a while that people are asking about grok and zmq. On the other hand I'm not sure how much functional these modules are (AFAIR the the source part of the ZMQ module was never really finished) and the grok module builds on a C library which is not actively maintained for many years. I don't recall people mentioning any of the other modules recently. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Sun, Jul 30, 2017 at 7:40 PM, Scheidler, Balázs < balazs.scheidler(a)balabit.com> wrote: > Hi, > > We had a discussion with our Debian maintainer about the fate of > syslog-ng-incubator. For those of you who don't know, it is a set of > syslog-ng modules not integrated to syslog-ng master for one reason or > another. > > It is available from here: > > https://github.com/balabit/syslog-ng-incubator > > The problem with this repository is that it is not really maintained, > causing compilation and functionality issues down the line. > > So our opinion is to go ahead and integrate stuff that can be brough up to > speed. Once we've done what we could, we would discontinue the incubator > altogether. > > Please speak up now, if you disagree, but please also be ready to help us > out in maintenance in that case. > > If there are no disagreements in the following week, we would start to > phasing out the incubator. > > We welcome volunteers to: > > 1) migrate code, OR > 2) help with testing > > If you are the user of any of those plugins, letting us know at this point > also helps making sure that no functionality is lost. > > Thanks. > > -- > Bazsi > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product= > syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

syslog-ng 3.11 on twitter
by Czanik, Péter 02 Aug '17

02 Aug '17

Hi, Please help spreading the word about the syslog-ng 3.11 also on Twitter. The syslog-ng OSE twitter account is available at https://twitter.com/sngose Please re-tweet one ore more of the following tweets: - 3.11 announcement: https://twitter.com/PCzanik/status/892315446325776386 - 3.11 documentation: https://twitter.com/PCzanik/status/892742991080763392 Depending on your Linux distribution preferences, there are also tweets about unofficial 3.11 packages: - fedora/RHEL7: https://twitter.com/PCzanik/status/892666279437271040 - openSUSE/SLES12: https://twitter.com/PCzanik/status/892648387400609793 - Debian/Ubuntu: https://twitter.com/PCzanik/status/892763031666556929 Tweets are from my own account, but you should follow @sngOSE account: https://twitter.com/sngose Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

syslog-ng 3.11.1 resources
by Czanik, Péter 02 Aug '17

02 Aug '17

Hi, Version 3.11.1 of syslog-ng OSE was released yesterday. Here I try to compile a list of useful resources in a hope that it does not end up in your spam folder due to the excess amount of URLs :) - List of new features and bugfixes: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.11.1 - source tgz: https://github.com/balabit/syslog-ng/releases/download/syslog-ng-3.11.1/sys… - unofficial Debian / Ubuntu packages: https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… - unofficial openSUSE / fedora / SLES / RHEL packages: https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… - documentation as single page HTML: https://www.balabit.com/documents/syslog-ng-ose-3.11-guides/en/syslog-ng-os… - for other formats go to https://www.balabit.com/support/documentation Choose “syslog-ng Open Source Edition” as product, select your version and your preferred format (single or multi-page HTML or PDF) For generic syslog-ng information go to https://syslog-ng.org/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Insider 2017-08: 3.11 released; wildcard-file; docker; SCL; RMLL/LSM;
by Czanik, Péter 02 Aug '17

02 Aug '17

Dear syslog-ng users, This is the 60th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.11 released ----------------------- The latest version of syslog-ng, 3.11 is now available. The most important new feature is the GeoIP2 parser which builds on libmaxminddb providing both better performance and more detailed geographical information about IP addresses. AMQP destination now supports SSL. There are many more smaller features and bug fixes. For a complete list check the release announcement: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.11.1 Reading multiple files: wildcard file source ------------------------------- Starting with version 3.10, syslog-ng can collect messages from multiple text files. You do not have to specify file names one by one, just use a wildcard to select which files to read. This is especially useful when you do not know the file names by the time syslog-ng is started. This is often the case with web servers with multiple virtual hosts. https://www.balabit.com/blog/reading-multiple-files-wildcard-file-source-sy… Collecting Docker infrastructure logs ------------------------------------- Why use syslog-ng for collecting Docker logs? Docker already provides many drivers for logging, even for central log collection. On the other hand remote logging drivers arrive with a minimalist feature set and you are not able to use the “docker logs” command any more. To have the best of both worlds, you can use journald logging driver in Docker and use syslog-ng to read Docker logs from journald and forward log messages to your central log server or other destinations. You can even run syslog-ng itself in a Docker container, so you can use it on dedicated Docker host environments as well where it is not possible to install additional applications. https://www.balabit.com/blog/collecting-docker-infrastructure-logs-using-sy… The power of SCL ---------------- The syslog-ng configuration library (SCL) can help you to configure syslog-ng a lot more easily. These configuration snippets can hide away the complexity of collecting, parsing or storing log messages. >From this blog you can learn how to parse web server logs and store the results at a Logging as a Service (LaaS) provider in a structured form. You will use SCL both for message parsing and the LaaS destination, and also utilize the wildcard-file() source introduced in syslog-ng 3.10. https://www.balabit.com/blog/parsed-web-server-logs-cloud-syslog-ng-scl/ RMLL / Libre Software Meeting 2017 ---------------------------------- This year I participated again in the security track of the largest French open source conference, Libre Software Meeting (RMLL). “Participated” as I did not only give a talk on syslog-ng there, but also sat in to most of the presentations and had very good discussions both with visitors and fellow speakers. The organizers brought together talks from diverse IT security related fields, a very good opportunity for cross-pollination of ideas. https://www.balabit.com/blog/czp-rmll-libre-software-meeting-2017/ NEW RELEASES syslog-ng OSE 3.11.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.11.1 Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

OSE 3.11
by Laszlo Budai 01 Aug '17

01 Aug '17

3.11.1 Features * Add geoip2 parser and template function. It is based on the libmaxminddb(MaxMindDB). It will replace the old geoip parser and template function, so they are deprecated from 3.11 (but still available). * Add SSL support to AMQP. * Add template option to apache-accesslog-parser. * Add configurable event time to Riemann destination. * Add drop-unmatched() option to dbparser. * Add Ubuntu Xenial to the bundled docker images. * Support multi-instance support for Solaris 10 and 11. * Support multi-instance for systemd. * Add configurable timeout to HTTP destination. * Add prefix() option to cisco-parser. Bugfixes * Fix a memory usage counter underflow for threaded destination drivers and writers. * Fix a potential crash in AMQP. * Fix a potential crash during reload. * Fix a reload/shutdown issue. Under heavy load, worker might never exit from the fetch loop from the queue. * Fix a potential crash in afsocket destination during reload. * Fix a counter registration bug. In some cases not all the required counters are registered. * Fix a build issue on FreeBSD. * Fix a memory leak in diskq plugin. * Fix systemd-journal error codes validation. * Fix a potential crash in diskq when it is used with file destination and the file is reaped. * Fix a memory leak in HTTP destination * Fix ENABLE_DEBUG in dbparser. * Fix a unit tests that caused build issue on 32 bit platforms. Other changes * The eventlog library is part of syslog-ng from now. * Improve error messages when the config cannot be initialized. * Improve source suspended/resumed debug messages. * Rename syslog-debun to syslog-ng-debun. * Update manpages to v3.11 * Remove tgz2build directory. Notes to the Developers * Rewrite merge-grammar script from Perl to Python. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Fabien Wernli, Gabor Nagy, Giuseppe D'Anna, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Lorand Muzamel, Mate Farkas, Noemi Vanyi, Peter Czanik, Tamas Nagy, Tibor Bodnar, Tomasz Kazimierczak, Zoltan Pallagi

1 0

eStreamer src Integration
by Scot 01 Aug '17

01 Aug '17

Has anyone looked at sending Cisco eStreamer events to syslog-ng ? We have a couple Cisco Firepower management centers and I would rather use syslog-ng over sending directly to splunk so that we may use other integrations like elastic and our NMS. I have the eStreamer SDK on my syslog-ng server and wondered if anyone else has worked on this. Search of the user archive says no. Thanks Scot

3 3

syslog-ng poll: geographical information
by Czanik, Péter 01 Aug '17

01 Aug '17

Hi, One of the major features of the upcoming syslog-ng 3.11 will be a complete rewrite of GeoIP support. So this time I ask you in the poll if you use syslog-ng to look up geographical information for IP addresses: https://twitter.com/PCzanik/status/891985964788330497 If you use GeoIP, please include your use case in the comments or answer this e-mail. You can e-mail me directly, if for some reason you don't want to share it with the rest of the world. Please re-tweet the above tweet to reach as many syslog-ng users as possible! Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0