September 2020 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Rewrite JSON field without re-formating via custom template
by Nirgil 29 Sep '20

29 Sep '20

Hello all, sorry to bother you, but I'm bit lost ;] Trying to accomplish the following: Have a JSON inside syslog message and I need to rewrite, or better to remove, one selected particular field, it contains lot of data and I just want to drop this one field before forwarding to different destination. I have created a parser for JSON: parser p_json { channel { parser { json-parser (prefix(".json.")); }; }; }; Now I need remove or change a JSON field, lets say payload. So I created the following rewrite rule and tried multiple options to remove or change the content rewrite { unset(value("${.json.payload}")); unset(value(".json.payload")); groupunset(values("${.json.payload}")); set("TEST", value(".json.payload")); }; This is working on default fields ${HOST}, etc, but not for JSON fields. Message can be reformatted via template ie: template json_template { template("${DATE} ${HOST} $PROGRAM: {\"payload\"\:\""${.json.payload}"\"}") }; In that case, json.payload can be affected by appropriate rewrite rule. But I'd like to affect the json.payload field inside the original ${MSG} without re-formatting whole message via custom template, because the original message has lot of optional fields, other inner JSON's and it is not so simple to made an universal template for this. template json_template { template("${DATE} ${HOST} $PROGRAM: ${MSG}") }; Is there a way how to remove JSON field from original message (${MSG}) without reformatting the whole message via custom template ? Thanks for any thoughts ! Sample of Message: Sep 23 15:22:48 hostname myapplication: {"payload":"verylongpayload","field1":"value",.."lot of other fields, included inner JSON"...}

3 4

Syslog-NG within Kubernetes
by Allen Olivas 22 Sep '20

22 Sep '20

Hello all, I've been working on a containerized and orchestrated environment for a little while now (learning how it all works and slowly implementing it). One step I need to add is Syslog-NG. I"ve copied down the Syslog-NG Docker files and attempted adding these to a deployment in Kubernetes. Syslog-NG appears to be running successfully in Docker but in Kubernetes the logs I get say: syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' I'm not entirely sure where to turn too from this point. I've looked online for any ideas or resolution to this issue but so far non seem to work for my use case. I figured maybe someone on the mailing list might have an Idea? Specifically I'm trying to find out what that error means, and how best to orchestrate a containerized Syslog-NG. Thanks!

2 1

Can I manually setup a Syslog server?
by Turritopsis Dohrnii Teo En Ming 20 Sep '20

20 Sep '20

Subject: Can I manually setup a Syslog server? Good day from Singapore, I have just implemented and setup the following for an investment company (company name is confidential) in Singapore on 12 Sep 2020 Saturday Singapore Time. (1) Installed and configured Kiwi Free Syslog Server 9.7.0 (by Solarwinds) on the Active Directory Domain Controller (Windows Server 2016 Standard). (2) Configured Cisco ASA 5506-X Firewall's Logging to send and transmit syslog messages to Kiwi Free Syslog Server 9.7.0 on the Windows Server. (3) Added a firewall rule to open UDP port 514 on the Symantec Endpoint Protection Manager (SEPM) on Windows Server 2016. So the question I have is, can I manually setup and configure a Syslog Server with GUI using open source syslog daemon, open source tools and any Linux distro? Thank you very much. -- -----BEGIN EMAIL SIGNATURE----- The Gospel for all Targeted Individuals (TIs): [The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html ******************************************************************************************** Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020): [1] https://tdtemcerts.wordpress.com/ [2] https://tdtemcerts.blogspot.sg/ [3] https://www.scribd.com/user/270125049/Teo-En-Ming -----END EMAIL SIGNATURE-----

2 2

Insider 2020-09: Prometheus; proxy; ESK;
by Peter Czanik (pczanik) 09 Sep '20

09 Sep '20

Dear syslog-ng users, This is the 84th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Using a proxy with the http() destination ----------------------------------------- The http() destination is quickly becoming one of the most often used destinations within syslog-ng. You might already be using it even if you are not aware of it. Quite a few syslog-ng destination drivers are actually just configuration snippets in the syslog-ng configuration library (SCL), utilizing the http() destination in the background. Just think about elasticsearch-http(), different Logging as a Service (LasS) providers, or slack(). Starting with syslog-ng version 3.28.1 you can also reach these services when there is a proxy server between syslog-ng and your destination. https://www.syslog-ng.com/community/b/blog/posts/using-a-proxy-with-the-htt… Prometheus: syslog-ng exporter ------------------------------ Recently Prometheus became one of the most used open source monitoring solutions. Quite a few people asked if a syslog-ng exporter is available. It is not part of syslog-ng, but there are numerous implementations available on GitHub. Now that Prometheus is part of the openSUSE Leap 15.2 release, which is the Linux distribution running on my laptop, I gave it a try. From this blog, you can learn how to compile the syslog-ng exporter for Prometheus yourself and get it working with Prometheus. https://www.syslog-ng.com/community/b/blog/posts/prometheus-syslog-ng-expor… Jump-starting ESK: Elasticsearch, syslog-ng and Kibana ------------------------------------------------------ If you want to test drive syslog-ng or just want to learn something new, I recommend you checking out the BLACK ESK project. By running a single script, you can set up a containerized test environment, complete with Elasticsearch, Kibana and a syslog-ng server. All network connections among them are encrypted and the syslog-ng configuration showcases many interesting syslog-ng features, including PatternDB and JSON parsing, GeoIP, in-list filtering and the new Elasticsearch destination. Once it is installed, all you need are some logs directed at this server and a browser to reach Kibana. You can learn a lot from reading through the setup scripts and the different configuration files. https://www.syslog-ng.com/community/b/blog/posts/jump-starting-esk-elastics… NEW RELEASES * Version 3.29.1 released: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.29.1 WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Multi-line messages across socket
by Alexandre Santos 07 Sep '20

07 Sep '20

Hi, I have some multi-line messages that are being broken, into single log messages, when they are sent over some remote host. In my configuration, I have one syslog-ng sending messages to other syslog-ng over unix-domain-socket. (check configurations in attachment). My goal is to have content of "/tmp/test1_udp_file.log" equal to the content of "/var/log/netconf-commands.log". Do know what am I missing? Any help is appreciated. Thanks in advance. Alex *</usr/sbin/syslog-ng -Fvde>* [2020-09-07T15:57:17.711583] Incoming log entry from journal; message='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' [2020-09-07T15:57:17.711760] json-parser(): no marker at the beginning of the message, skipping JSON parsing ; input='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB', marker='@cim:' [2020-09-07T15:57:17.712016] Reliable disk-buffer state saved; filename='/tmp/syslog-ng-00002.rqf', qdisk_length='0' [2020-09-07T15:57:17.712046] Initializing destination file writer; template='/var/log/netconf-command.log', filename='/var/log/netconf-command.log' [2020-09-07T15:57:17.712150] Outgoing message; message='<158>1 2020-09-07T15:57:17.449+00:00 localhost root 25947 - - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\x0a' [2020-09-07T15:57:17.712299] Outgoing message; message='<158>1 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\x0a' *</var/log/netconf-commands.log>* <158>1 2020-09-07T15:57:17.449+00:00 localhost root 25947 - - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 with newline BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB *<ip vrf exec MGMT /usr/sbin/syslog-ng -Fvde --cfgfile=/etc/syslog-ng/mgmt-syslog-ng.conf --pidfile=/var/lib/syslog-ng/mgmt-syslog-ng.pid --persist-file=/var/lib/syslog-ng/mgmt-syslog-ng.persist --control=/var/lib/syslog-ng/mgmt-syslog-ng.ctl>* [2020-09-07T15:57:17.712514] Incoming log entry; line='<158>1 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 ' [2020-09-07T15:57:17.712762] Incoming log entry; line='with newline BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' [2020-09-07T15:57:17.712997] Outgoing message; message='<158>1 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - [meta sequenceId="2"] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 \x0a' [2020-09-07T15:57:17.713250] Outgoing message; message='<158>1 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - [meta sequenceId="2"] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00 \x0a' *</tmp/test1_udp_file.log>* <158>1 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - [meta sequenceId="2"] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Test at 2020-09-07T15:57:17,448482284+00:00

2 1

Re: [syslog-ng] message being consistently dropped
by Gabor Nagy (gnagy) 01 Sep '20

01 Sep '20

Hi, Your logs get dropped by the filter "facility(local1)". The filter does not match on the above log because it cannot be parsed by the syslog-protocol (RFC5424) due to the unescaped brackets in the SDATA. Due to the failed parsing, your log message has been altered to visualize the parsing errors location: <43>1 2020-09-01T18:28:24+02:00 localhost syslog-ng 2514 - - Error processing log message: <141>1 2020-08-04T08:10:58.769127-05:00 fqdn.example.com apmd 12374 01490113:5: [F5@12276 hostname="fqdn" errdefs_msgno="01490113:5:" partition_name="Common" session_id="1c95e1e7" Access_Profile="/Common/blah" Partition="Common" Session_Id="1c95e1e6" Session_Variable_Name="session.machine_info.last.net_adapter.list.[0>@<].mac_address" Session_Variable_Value="3C:D9:2B:33:9A:8E"] /Common/<blah>:Common:1c95e1e6: session.machine_info.last.net_adapter.list.[0].mac_address is 3C:D9:2B:33:9A:8E The question is, do you need to parse the message before storing it into a log file? Because we can disable parsing on the source side and the whole message is put into the $MESSAGE macro. I guess you need some basic parsing, as you filter the messages based on facility. Do you use this source in other log paths as well? I'm thinking in a workaround currently, as so far I didn't found a way to overcome this issue. I would like to emphasize that F5 log format violates the RFC5424 described protocol. This should be reported to them. Regards, Gabor ________________________________ From: Wilson, Jonathan <jonathan.wilson(a)vumc.org> Sent: Tuesday, September 1, 2020 14:39 To: Gabor Nagy (gnagy) <Gabor.Nagy(a)oneidentity.com> Subject: Re: [syslog-ng] message being consistently dropped CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Good morning Gabor, thank you for the info on my syslog-ng issue. Unfortunately the device in question is “canned” (an F5) and I am pretty sure I cannot change the format it logs with. Is there a simple way to pre-process them so that they don’t get dropped by syslog-ng? Thanks, Jon Jon Wilson | Principal System Engineer, IT Service Management | Information Technology | Vanderbilt University Medical Center jonathan.wilson(a)vumc.org<mailto:jonathan.wilson@vumc.org> | phone: 615-440-7895 | fax: 615-323-2181

1 0

Re: [syslog-ng] syslog-ng is ignoring a network source
by Bruns, Benjamin 01 Sep '20

01 Sep '20

Hello Laci, Thanks a lot! It turned out that my second firewall does not come through to netcat on port 514, even if I turned off the firewall completely. It's all running a Security Onion host and now I'm wondering if the IDS/IPS is fooling me and is blocking the traffic (but I don't notice any alerts related to the IP of the second firewall ...). However, any other ideas? Mit freundlichen Grüßen / Best regards Benjamin Bruns IT Security Manager CYPP GmbH Gotenstrasse 15 20097 Hamburg Germany Telefon: +49 40 237 34-285 Mobil: +49 160 3230 655 Mail: benjamin.bruns(a)cypp.de<mailto:benjamin.bruns@cypp.de> www.cypp.de<http://www.cypp.de/> | www.plathgroup.com<http://www.plathgroup.com/> Sitz der Gesellschaft: Hamburg HRB 132781 Amtsgericht Hamburg Ust-ID DE 298033875 Vertretungsberechtigter Geschäftsführerin: Maya von Holdt Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, benachrichtigen Sie den Versender bitte per E-Mail oder telefonisch und löschen Sie die Nachricht unverzüglich. This message is strictly confidential and intended solely for the use of the addressee. It is not allowed to copy or disseminate this message. Please notify the sender by e-mail or telephone if you have received this message by mistake and delete this message immediately. Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 28. August 2020 13:41 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] syslog-ng is ignoring a network source Hello Ben, nc -u -l -p 514 -u : use UDP -l : listen in server mode, instead of sending -p : defines port number (If you need, you can also specify the local IP address to bind to with the -s option.) Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Bruns, Benjamin <Benjamin.Bruns(a)cypp.de<mailto:Benjamin.Bruns@cypp.de>> Sent: Friday, August 28, 2020 13:29 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] syslog-ng is ignoring a network source CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello Laci, how do I check that by using netcat? Cheers, Ben Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 28. August 2020 12:21 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Betreff: Re: [syslog-ng] syslog-ng is ignoring a network source Hello Ben, our experience shows that having logs in Wireshark doesn't necessary means that they reach the applications. (Wireshark captures packets on the interface level.) So as a first step I would recommend to clarify that the logs from the second host indeed reaches Syslog-ng. (For UDP logs netcat usually is enough.) Once it is clear that those logs reaches the application level, we can focus on debugging Syslog-ng. By starting Syslog-ng with the following options, it would be much easier to examine the flow of messages: syslog-ng -Fdevt -F : start it in the foreground -d : debug mode -e : log messages to stderr -v : increases verbository -t : also enable trace messages Note: With these options enabled, Syslog-ng will produce a LOT of messages. So if you can turn off other logging sources temporary, than it will be much more easier to read those logs. At this point you should start to receive this kind of debug messages, which will indicate that Syslog-ng received the log messages from your host: [2020-08-28T10:09:43.289660] Incoming log entry; line='hello world' >From this point the easiest way is to start with a minimal config, and build up your final configuration step by step. Checking incoming logs in each steps. Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Bruns, Benjamin <Benjamin.Bruns(a)cypp.de<mailto:Benjamin.Bruns@cypp.de>> Sent: Friday, August 28, 2020 11:53 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] syslog-ng is ignoring a network source CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello! I try to send syslogs from two firewalls to my syslog-ng host. The first one worked immediately, but logs of the second firewall seems to be ignored. Both syslogs come in via UDP on port 514 and I can see them in Wireshark on my syslog-ng host, but they disappear for my second firewall in a black hole. Both have Logstash as their destination configured. Any ideas? Thanks in advance! Cheers, Ben

2 3

Re: [syslog-ng] syslog-ng is ignoring a network source
by Bruns, Benjamin 01 Sep '20

01 Sep '20

Hello Laci, Question: I can successfully read the logs of the firewall that is located in the same network that my logserver is, but the second firewall is sending the logs from a different network segment to the log server. Does this causes the problem? Again: I can see the traffic on my network interface, but netcat doesn't receive anything on port 514 (for the second firewall). Does iptables fools me somehow? Mit freundlichen Grüßen / Best regards Benjamin Bruns IT Security Manager CYPP GmbH Gotenstrasse 15 20097 Hamburg Germany Telefon: +49 40 237 34-285 Mobil: +49 160 3230 655 Mail: benjamin.bruns(a)cypp.de<mailto:benjamin.bruns@cypp.de> www.cypp.de<http://www.cypp.de/> | www.plathgroup.com<http://www.plathgroup.com/> Sitz der Gesellschaft: Hamburg HRB 132781 Amtsgericht Hamburg Ust-ID DE 298033875 Vertretungsberechtigter Geschäftsführerin: Maya von Holdt Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, benachrichtigen Sie den Versender bitte per E-Mail oder telefonisch und löschen Sie die Nachricht unverzüglich. This message is strictly confidential and intended solely for the use of the addressee. It is not allowed to copy or disseminate this message. Please notify the sender by e-mail or telephone if you have received this message by mistake and delete this message immediately. Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 28. August 2020 13:41 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] syslog-ng is ignoring a network source Hello Ben, nc -u -l -p 514 -u : use UDP -l : listen in server mode, instead of sending -p : defines port number (If you need, you can also specify the local IP address to bind to with the -s option.) Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Bruns, Benjamin <Benjamin.Bruns(a)cypp.de<mailto:Benjamin.Bruns@cypp.de>> Sent: Friday, August 28, 2020 13:29 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] syslog-ng is ignoring a network source CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello Laci, how do I check that by using netcat? Cheers, Ben Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 28. August 2020 12:21 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Betreff: Re: [syslog-ng] syslog-ng is ignoring a network source Hello Ben, our experience shows that having logs in Wireshark doesn't necessary means that they reach the applications. (Wireshark captures packets on the interface level.) So as a first step I would recommend to clarify that the logs from the second host indeed reaches Syslog-ng. (For UDP logs netcat usually is enough.) Once it is clear that those logs reaches the application level, we can focus on debugging Syslog-ng. By starting Syslog-ng with the following options, it would be much easier to examine the flow of messages: syslog-ng -Fdevt -F : start it in the foreground -d : debug mode -e : log messages to stderr -v : increases verbository -t : also enable trace messages Note: With these options enabled, Syslog-ng will produce a LOT of messages. So if you can turn off other logging sources temporary, than it will be much more easier to read those logs. At this point you should start to receive this kind of debug messages, which will indicate that Syslog-ng received the log messages from your host: [2020-08-28T10:09:43.289660] Incoming log entry; line='hello world' >From this point the easiest way is to start with a minimal config, and build up your final configuration step by step. Checking incoming logs in each steps. Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Bruns, Benjamin <Benjamin.Bruns(a)cypp.de<mailto:Benjamin.Bruns@cypp.de>> Sent: Friday, August 28, 2020 11:53 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] syslog-ng is ignoring a network source CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello! I try to send syslogs from two firewalls to my syslog-ng host. The first one worked immediately, but logs of the second firewall seems to be ignored. Both syslogs come in via UDP on port 514 and I can see them in Wireshark on my syslog-ng host, but they disappear for my second firewall in a black hole. Both have Logstash as their destination configured. Any ideas? Thanks in advance! Cheers, Ben

2 1

message being consistently dropped
by Wilson, Jonathan 01 Sep '20

01 Sep '20

We are running syslog-ng, open source edition, version 3.22.1, on RedHat Enterprise version 7.8. Among many other data sources, we receive syslog data from an F5 device that acts as a VPN server. One type of message that it sends us is consistently not making it into the log (though everything else, to my knowledge, does.) I wonder if anyone sees something wrong with the message that would prevent its being parsed/stored. The message (reconstructed from a packet capture, identification fields redacted or changed): <141>1 2020-08-04T08:10:58.769127-05:00 <fqdn here> apmd 12374 01490113:5: [F5@12276 hostname="fqdn" errdefs_msgno="01490113:5:" partition_name="Common" session_id="1c95e1e7" Access_Profile="/Common/blah" Partition="Common" Session_Id="1c95e1e6" Session_Variable_Name="session.machine_info.last.net_adapter.list.[0].mac_address" Session_Variable_Value="3C:D9:2B:33:9A:8E"] /Common/<blah>:Common:1c95e1e6: session.machine_info.last.net_adapter.list.[0].mac_address is 3C:D9:2B:33:9A:8E I inserted the line breaks to help Outlook. Note that although the message appears to be RFC5424-compliant I have discovered that it does not have whatever magic headers let syslog-ng handle it as structured data transparently. As a result, I cannot use the "syslog" transport but have to use the generic "network" transport and then use flags "syslog-protocol". This works OK for everything else from this source. The message doesn't appear in our syslog logs at all. It's consistent. The relevant parts of the syslog-ng configuration look like: options { ts_format(iso); use_dns(yes); dns_cache(1000); dns_cache_expire(30); use_fqdn(yes); keep_hostname(no); chain-hostnames(no); keep_timestamp(yes); stats_level(1); stats_freq(3600); normalize_hostnames(yes); log_fifo_size(10000); flush_lines(1000); log-msg-size(65536); trim-large-messages(yes); threaded(yes); time_reap(10); create_dirs(yes); dir_group(esmapp); dir_owner(esmapp); group(esmapp); owner(esmapp); perm(0644); }; source s_F5VPN_source { network( transport("tcp") ip(0.0.0.0) port(11000) keep-timestamp(yes) flags(no-multi-line,syslog-protocol) max-connections(50) log_iw_size(10000) log-fetch-limit(20) ); }; filter f_F5VPN_client { netmask(<blah>) and facility(local1); }; destination d_F5VPN_client { file( "/home/syslog/F5VPN_client.log" ts_format(rfc3164) ); }; log { source(s_F5VPN_source); filter(f_F5VPN_client); destination(d_F5VPN_client); flags(flow-control); }; Regards, Jon Jon Wilson | Principal System Engineer, IT Service Management | Information Technology | Vanderbilt University Medical Center jonathan.wilson(a)vumc.org<mailto:jonathan.wilson@vumc.org> | phone: 615-440-7895 | fax: 615-323-2181

2 1