November 2019 - syslog-ng - lists.balabit.hu

Insider 2019-11: logging to Elasticsearch; PE 6 to 7 upgrade; Elastic 7; in-list(); off-line deb; Splunk conf;
by Peter Czanik (pczanik) 14 Nov '19

14 Nov '19

Dear syslog-ng users, This is the 76th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Logging to Elasticsearch made simple with syslog-ng --------------------------------------------------- Elasticsearch is gaining momentum as the ultimate destination for log messages. You can store arbitrary name-value pairs coming from structured logging or message parsing and you can use Kibana as a search and visualization interface. You can use syslog-ng as an all in one solution (system and application logs, message parsing, filtering and queuing) to collect and forward log messages to Elasticsearch: https://www.syslog-ng.com/community/b/blog/posts/logging-to-elasticsearch-m… Upgrading syslog-ng PE from version 6 to 7 ------------------------------------------ Learn the major steps necessary to upgrade your system from syslog-ng Premium Edition version 6 to 7. As you will see, it is no more difficult than any other major software version upgrade, and after the upgrade you can start using all the new and useful features that are available in version 7. https://www.syslog-ng.com/community/b/blog/posts/upgrading-syslog-ng-pe-fro… syslog-ng and Elasticsearch 7: getting started on RHEL/CentOS ------------------------------------------------------------- Version 7 of the Elastic Stack, packed with new features and improved performance, has now been available for some time. Elasticsearch is not the only one to have come up with a major new version recently: starting with version 3.21, syslog-ng features a new Elasticsearch destination driver (based on the http() destination) that does not require Java. In most cases, it is more resource friendly than the Java-based driver and it is definitely easier to configure. It also has the benefit that unlike the Java-based driver, it can be included in Linux distributions. This is a quick how-to guide to get you started with syslog-ng and Elasticsearch 7 on RHEL/CentOS 7. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-elasticsearc… Handling lists in syslog-ng: the in-list() filter ------------------------------------------------- Recently, a number of quite complex configurations came up while syslog-ng users were asking for advice. Some of these configurations were even pushing the limits of syslog-ng (regarding the maximum number of configuration objects). As it turned out, these configurations could be significantly simplified using the in-list() filter, one of syslog-ng’s lesser known features. https://www.syslog-ng.com/community/b/blog/posts/handling-lists-in-syslog-n… Offline syslog-ng DEB package installer --------------------------------------- “How can I install the unofficial syslog-ng packages on a machine without Internet access?” This question has been raised several times recently. As it entails more than simply downloading the repository containing the packages, syslog-ng lead developer Laszlo Budai created a script that solves the problem for Debian and Ubuntu users. The script downloads syslog-ng along with its dependencies using a container, and produces an archive containing all DEB packages necessary to install syslog-ng as well as a simple script responsible for the installation. Using a container in this case might seem like over complication, but it is still the easiest way to ensure that all dependencies are included in the archive. https://www.syslog-ng.com/community/b/blog/posts/offline-syslog-ng-deb-pack… syslog-ng at Splunk .conf 2019 ------------------------------ The syslog-ng team participated at Splunk .conf 2019 and came home with positive experiences. They met enthusiastic users, who use syslog-ng in front of Splunk as log management layer. Many of them already using the HTTP event collector and use syslog-ng to load-balance log messages among collectors. They also demonstrated how syslog-ng can collect over one million log messages a second over UDP. https://twitter.com/OneIdentity/status/1187852500222693377 WEBINARS * You can achieve 1m+ UDP messages per second ingestion challenge: https://www.syslog-ng.com/webcast-ondemand/you-can-achieve-1m-udp-messages-… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 3

syslog-ng in EPEL 8
by Peter Czanik (pczanik) 13 Nov '19

13 Nov '19

Hi, Version 3.23.1 of syslog-ng is now available in EPEL 8. It has most, but not all of the syslog-ng features available. Missing features: * kafka destination, C implementation, as librdkafka is too old * Java destination drivers (kafka, elastic, hdfs), as sources necessary to compile them are not in the distro. Note, that Elastic is supported through elastic-http() * tcp wrapper, as support was removed from RHEL * riemann does not have sub-second precision, as client in EPEL is too old As missing dependencies arrived to EPEL 8, right now the official EPEL 8 syslog-ng package has more features enabled, than my own experimental EPEL 8 package 🙂 Once 3.25.1 is released, I hope to support RHEL/CentOS 8 from my regular syslog-ng repositories. The difference is not as big as it was before missing dependencies became available, but I still need to figure out some RPM spec magic 🙂 EPEL 8 stays at 3.23 (no version update is allowed there). There are plans to allow new versions into EPEL with each new minor RHEL release, but seeing is believing. It was promised for too many years. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 7

parse rancher logs from syslog-ng files
by Tech Gurus 13 Nov '19

13 Nov '19

Hi , I have setup where rancher 2.3 send logs ogs to syslog-ng server , I am getting several GB per hour and write them to text file. The log format are dynamic , general form are [timestamp SourceContainer-host-IP Source-Program list-of-nested-json-key/value-pairs] .. I got the elasticsearch forwarding wokring with config below, however all json kv gets written to message field.. I need to be able to : * Parse json kv into diffeent fields to elastic and flatten the json fields. * Use one field to create the index on destination elastic like "program" or custom added field on source side like "k8s-cluster-name" Thanks for help John destination d_elasticsearch_http { elasticsearch-http( url("https://elastricsearch:9200/_bulk") template("$(format-json --scope rfc5424 --scope dot-nv-pairs --rekey .* --shift 1 --scope nv-pairs --exclude DATE --key ISODATE @timestamp=${ISODATE})") index("syslog-ng-test") #time-zone("UTC") type("") workers(4) batch-lines(100) #batch-timeout(10000) #timeout(10) tls( #ca-dir("/etc/syslog-ng/conf.d") ca-file("/etc/syslog-ng/conf.d/root-ca.pem") cert-file("/etc/syslog-ng/conf.d/admin.pem") key-file("/etc/syslog-ng/conf.d/admin.key") peer-verify(no) ) ); };

2 1

Building syslog with cmake as external project
by Alexandre Santos 12 Nov '19

12 Nov '19

Hi All, I am trying to build syslog-ng with cmake to add it to a custom Linux distribution. The problem is that when the syslog-ng is compiled it gets the RPATH of the location where it has been generated: *root@5526d87140af:/home/src/**chassisx/build# ldd syslog-ng-install/sbin/syslog-* *ng linux-vdso.so.1 (0x00007fff151ed000) libsyslog-ng.so.3.24.1 => /home/src/target/build/**syslog-ng-install/lib/* *libsyslog-ng.so.3.24.1 (0x00007f81baf89000) libeventlog.so => /home/src/target/build/**syslog-ng-install/lib/* *libeventlog.so (0x00007f81bad84000) libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/* *libgmodule-2.0.so.0 (0x00007f81bab80000) libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/* *libgthread-2.0.so.0 (0x00007f81ba97e000) libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-* *2.0.so.0 (0x00007f81ba66a000) libivykis.so.0 => /home/src/target/build/**syslog-ng-install/lib/* *libivykis.so.0 (0x00007f81ba45b000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.* *so.3 (0x00007f81ba1e8000) libresolv.so.2 => /lib/x86_64-linux-gnu/* *libresolv.so.2 (0x00007f81b9fd1000) libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/* *libssl.so.1.1 (0x00007f81b9d65000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/* *libcrypto.so.1.1 (0x00007f81b98cc000) libsecret-storage.so => /home/src/target/build/**syslog-ng-install/lib/* *libsecret-storage.so (0x00007f81b96c8000) libpthread.so.0 => /lib/x86_64-linux-gnu/* *libpthread.so.0 (0x00007f81b94ab000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.* *6 (0x00007f81b910c000) /lib64/ld-linux-x86-64.so.2 (0x00007f81bb48e000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.**so.2 (0x00007f81b8f08000)* I am using the cmake file in attachment. Can you help me? Thanks in advance, Alex

2 4

Insert SequenceNumber into FileName
by William Luiz Ribeiro Vasconcelos Da Silva 12 Nov '19

12 Nov '19

Good Morning Is this possible put some kind of sequence Number or some kind ID number at the fileName, when the syslog-ng is writing the file???? If it's possible, how to do this? Tks, Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson, Brazil Mobile +55 11 97979-9886 wsilva_ericsson(a)timbrasil.com.br<mailto:wsilva_ericsson@timbrasil.com.br> www.ericsson.com<http://www.ericsson.com/> [Descri??o: Descri??o: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informa??es privilegiadas e/ou de car?ter confidencial, n?o podendo ser retransmitida sem autoriza??o do remetente. Se voc? n?o ? o destinat?rio ou pessoa autorizada para receb?-la, informamos que o seu uso, divulga??o, c?pia ou arquivamento s?o proibidos. Portanto, se voc? recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conte?do. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

Certificate valid, but invalid purpose
by W3EUU 12 Nov '19

12 Nov '19

Running syslog-ng 3.22 OSE on Fedora 31 installation. I am in the process of implementing TLS encryption on the remote links. I have the links working, including the mutual authentication function. However, I am getting the warning "Certificate valid, but purpose is invalid" on startup. Since it is a warning, it does not seem to be affecting the functionality. The certificates are self signed, with TLS Web Server Authentication and TLS Web Client Authentication extensions, among others. I am unable to find any documentation defining what additional extensions are needed to silence this warning. Can anyone tell me what additional is required?

2 1

Syslog-ng don't Listening UDP packets PORT(514) - Problem History
by William Luiz Ribeiro Vasconcelos Da Silva 08 Nov '19

08 Nov '19

Good Morning, Yesterday, we performed the first test of syslog-ng 3.24 OSE on a RHEL 7.6 VM in our customer's production environment. And we identified a very strange behavior. We saw through tcpdump that packets were being received by the network card but were not captured by syslog-ng. We have a UDP packet send simulator, and when I was pointing my simulator to another IP on the same machine(the users' connection IP), syslog-ng does the capping and consequently the treatment we put into the syslog-ng.conf file. What kind of TROUBLESHOOTING we can do in order to identify the reason for syslog-ng to capture packets on the users' connection IP, but for the application connection IP, it doesn't identify anything. It is informed by the client at the beginning of our project that syslog should always capture all UDP packets received by the application IP. This is the configuration of the "source" we have on this machine in our client's productive environment. source s_model { #udp(port(514)); udp(ip(10.46.105.34) port(514)); # udp(); #network(transport("udp")); #network( # ip("10.46.105.34") # transport("udp") #); }; Tks, Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

3 3

Syslog-ng don't Listening UDP packets PORT(514)
by William Luiz Ribeiro Vasconcelos Da Silva 08 Nov '19

08 Nov '19

Hello guys, We are doing a test in a VM RHEL 7.6, and the syslog-ng OSE 3.24, do Not capturing UDP packets. This is my configuration: ######################################################################################### source s_net { #udp(port(514)); udp(ip("10.46.105.34") port(514)); #udp(ip("10.25.17.4") port(514)); # udp(); #network(transport("udp")); #network( # ip("10.46.105.34") # transport("udp") #); }; destination d_dest { file("/syslog_out/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}${SEC}_GMT${TZ}_${RCPTID}.log" template("${PROGRAM};${MESSAGE};\n") owner(syslog) group(syslog) perm (0755) ); }; log { source(s_net ); destination(d_dest ); }; ########################################################################################### What kind of configuration i have to do, or what I need to check in OS, to do the capturing UDP packets. Tks, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

syslog-ng-stable RPM repo
by Peter Czanik (pczanik) 04 Nov '19

04 Nov '19

Hi, Until now I had a single request for a syslog-ng-stable repo, which always contains the latest stable release of syslog-ng. But the discussion about my cleanup plans brought up the topic again, and Bazsi wrote me, that there were more such requests coming from the Splunk Slack channel. So I did it: * fedora/rhel/centos: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-stable/ * openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng-stable In theory it should be the exact copy of my unofficial 3.23.1 packages and some fixes backported for openSUSE Tumbleweed. New syslog-ng versions will first appear in versioned repositories, as usual. I plan to push packages to the stable repo only after a week or two delay, so there is time find major bugs, have a .2 release if necessary. Once we have more experience that it works as expected, I'll update my blogs and the syslog-ng website. Until that I'm waiting for your feedback, especially once 3.24.1 comes out. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 2

GLib missing when running compilation with cmake
by Alexandre Santos 04 Nov '19

04 Nov '19

Hi All, I got the following error, when compiling with cmake: ... -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") CMake Error at /usr/share/cmake-3.7/Modules/FindPackageHandleStandardArgs.cmake:138 (message): Could NOT find GLIB (missing: GLIB_INCLUDE_DIRS GLIB_LIBRARIES GLIB_GMODULE_LIBRARIES GLIB_GTHREAD_LIBRARIES) Call Stack (most recent call first): /usr/share/cmake-3.7/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE) cmake/Modules/FindGLIB.cmake:111 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) CMakeLists.txt:168 (find_package) -- Configuring incomplete, errors occurred! See also "/thanos/src/syslog-ng/build/CMakeFiles/CMakeOutput.log". See also "/thanos/src/syslog-ng/build/CMakeFiles/CMakeError.log". Nevertheless, I have the following GLibs installed: root@5526d87140af:/asantos/src/syslog-ng-git# apt list libglib* Listing... Done libglib-object-introspection-perl/oldstable 0.042-1 amd64 libglib-perl/oldstable 3:1.324-1 amd64 libglib2.0-0/oldstable,now 2.50.3-2+deb9u1 amd64 [installed,automatic] libglib2.0-0-dbg/oldstable 2.50.3-2+deb9u1 amd64 libglib2.0-bin/oldstable 2.50.3-2+deb9u1 amd64 libglib2.0-cil/oldstable 2.12.40-2 amd64 libglib2.0-cil-dev/oldstable 2.12.40-2 amd64 libglib2.0-data/oldstable,now 2.50.3-2+deb9u1 all [installed,automatic] libglib2.0-dev/oldstable 2.50.3-2+deb9u1 amd64 libglib2.0-doc/oldstable 2.50.3-2+deb9u1 all libglib2.0-tests/oldstable 2.50.3-2+deb9u1 amd64 libglib3.0-cil/oldstable 2.99.3-2+b1 amd64 libglib3.0-cil-dev/oldstable 2.99.3-2+b1 amd64 libglibmm-2.4-1v5/oldstable 2.50.0-1 amd64 libglibmm-2.4-dev/oldstable 2.50.0-1 amd64 libglibmm-2.4-doc/oldstable 2.50.0-1 all root@5526d87140af:/asantos/src/syslog-ng-git# Can you explain me what is happening? Thanks, Alex

2 2