November 2020 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

poll: web interface for syslog-ng
by Peter Czanik (pczanik) 27 Nov '20

27 Nov '20

Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 3

Re: [syslog-ng] poll: web interface for syslog-ng
by Clayton Dukes 25 Nov '20

25 Nov '20

Well naturally, my vote is for LogZilla 😉 Peter, I should probably get you an updated description of the product, it has grown a lot! Here are a few useful links: LogZilla in 4 minutes https://www.logzilla.net/logzilla-in-4-minutes.html Automation/Orchestration https://www.logzilla.net/tcat.html ELK Revealed https://www.logzilla.net/the-real-cost-of-elk.html [photo] [signature_543644359] Clayton Dukes CEO, LogZilla Corporation [signature_672330100]<http://www.linkedin.com/in/lzcdukes/> [signature_753363620]<http://twitter.com/logzilla> [signature_14708953] (855) LogZilla<tel:(855)%20LogZilla> [signature_1738705128] 919-600-3198<tel:919-600-3198> [signature_579604246] cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> [signature_1641853981] logzilla.net<http://logzilla.net/> [signature_1306807084] 4819 Emperor Blvd., Suite 400, Raleigh, NC, 27703 [signature_560762648]<https://calendly.com/teamneo/logzilla-neo-discovery> Schedule a 15 minute demo<https://calendly.com/teamneo/logzilla-neo-discovery> [signature_457621985]<https://youtu.be/yGQqkXegdoo> [signature_39202885]LZ in 4 Minutes<https://youtu.be/yGQqkXegdoo> LogZilla Corporation delivers a Centralized Log Management (CLM) platform, designed for IT Operations, Security, and Risk Management leaders who want to gain better incident investigation capabilities by capturing all network and security related logs into a single log collection platform. [Logo Description automatically generated] From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of "Peter Czanik (pczanik)" <Peter.Czanik(a)oneidentity.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Date: Wednesday, November 25, 2020 at 9:46 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] poll: web interface for syslog-ng Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslo…>) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsyslog-ng…> https://twitter.com/PCzanik<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Logzilla Corporation may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.

1 0

syslog-ng 3.30 Debian/Ubuntu packages (OBS)
by Laszlo Budai 20 Nov '20

20 Nov '20

Hi, syslog-ng 3.30.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 * Ubuntu 20.10 (build is still in progress in OBS) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-geat install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.30.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

issues with starting syslog-ng from systemd
by Russell Fulton 20 Nov '20

20 Nov '20

version 3.26.1 — ubuntu 14.04 I can start syslog-ng from the command line without issues (apart form a warning to stderr) but when I try and start it from systemd it starts up (it prints the warning message to the logs) and then exits silently. This version was compiled locally i.e. it was not installed from a package. The .service file was adapted from a sample in the contrib directory. I also had the same behaviour from and other .service file that I put together myself. It is presumably some to do with the environment. I did try starting it with -i (interactive) in the hope that it would spit something out on the console but instead it hung and I had to ^C it Any thoughts on how to figure out what is going on? ===================================== here are two different attempts (first commented) rful011@secmgrprd02:~$ cat /etc/systemd/system/syslog-ng.service #[Unit] #Description=syslog next generation #After=network.target auditd.service #[Service] #EnvironmentFile=/etc/default/syslog-ng #ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf #ExecReload=/usr/local/sbin/syslog-ng-ctl reload #Restart=on-failure #RestartPreventExitStatus=255 #Type=notify #[Install] #WantedBy=multi-user.target #Alias=syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) Conflicts=emergency.service emergency.target Wants=network.target network-online.target After=network.target network-online.target [Service] Type=notify EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf ExecReload=/usr/local/sbin/syslog-ng-ctl reload StandardOutput=journal StandardError=journal Restart=on-failure [Install] WantedBy=multi-user.target

3 2

syslog-ng-3.30.1
by Laszlo Budai 19 Nov '20

19 Nov '20

3.30.1 Highlights * filter template function * support proxy-protocol<https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> Features * kafka (C implementation): * Added template support to topic(). * Added fallback-topic() option, which will be used, if the templated topic() yields an invalid topic name. (#3372<https://github.com/syslog-ng/syslog-ng/pull/3372>) * transport: add proxy-protocol support http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt Example config, click to expand! ([#3437<https://github.com/syslog-ng/syslog-ng/pull/3437>](https://github.com//pull/3437<https://github.com/syslog-ng/syslog-ng/pull/3437>)) * filter: new template function The new introduced filter template function will allow filtering lists based on a filter expression. For example this snippet removes odd numbers log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * file, network, program destinations: : new truncate_size option introduced to truncate an output message to a specified max size. default value is -1 (disabled). network("127.0.0.1" truncate_size(100)); new stats counters: dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_count;1 dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_bytes;1 (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * network: add FreeBSD support for the so_reuseport(yes) the same as in linux SO_REUSEPORT (FreeBSD uses SO_REUSEPORT_LB flag). (#3438<https://github.com/syslog-ng/syslog-ng/pull/3438>) * date-parser: %z accepts local timezone std format as well (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * syslog-format: accepting longer sdata keys Triggered by #3197<https://github.com/syslog-ng/syslog-ng/issues/3197> At the end of the discussion in the mentioned issue, we decided to change the parser and accept longer than 32 character ID's. (#3244<https://github.com/syslog-ng/syslog-ng/pull/3244>) * systemd-journal: add namespace() option This option accepts a string which is identical to the --namespace option of journalctl. For systems defining this option with a systemd version older than v245 a warning is issued. (#3358<https://github.com/syslog-ng/syslog-ng/pull/3358>) Bugfixes * date-parse: %Z should parse the same timezones as %z not just local and gmt (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * python: printing the exception instead of None (if compiled with clang) (#3405<https://github.com/syslog-ng/syslog-ng/pull/3405>) * network/udp: message was lost (not sent) if it was too large, and a time reopen amount of time needed to expire to send the next message lowering the thoughtput. now it is truncated at 65507. (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * tlscontext: support IPv6 X509v3 Subject Alternative Name Fixes #3465<https://github.com/syslog-ng/syslog-ng/issues/3465> (#3466<https://github.com/syslog-ng/syslog-ng/pull/3466>) * map: pass $_ to if correctly. Prior this patchset, if did not receive $_ correctly. After this change, these configurations will work: log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(map $(if ('$(% $_ 2)' eq '0') 'even' 'odd') $INPUT)'\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * systemd-journal: add namespace to the persist name (#3407<https://github.com/syslog-ng/syslog-ng/pull/3407>) * syslog-ng: fixed numerous spelling mistakes in messages generated by syslog-ng (#3398<https://github.com/syslog-ng/syslog-ng/pull/3398>) * network: fix TLS certificate hostname verification when using failover() servers For TLS certificate hostname verification, the certificate's hostname needs to be compared to the configured hostname of the primary and each failover server. syslog-ng used always the primary server's name incorrectly. (#3447<https://github.com/syslog-ng/syslog-ng/pull/3447>) * afsocket: syslog-ng fails to bind() after config revert When having a program source or destination and a network destination in the config, if we reload with an invalid config, syslog-ng crashes, as it cannot init the old network source, because its address is in use. (#3416<https://github.com/syslog-ng/syslog-ng/pull/3416>) * syslog-ng-ctl: when syslog-ng gets stuck on executing a heavy stats-ctl command, should be able to do a graceful shutdown when it is requested. (#3349<https://github.com/syslog-ng/syslog-ng/pull/3349>) * json-parser: fix parsing 64 bit numbers (currently 32 bit was a limit) (#3403<https://github.com/syslog-ng/syslog-ng/pull/3403>) * usertty(): on each tty open error an error mesage and a 10 minutes long disabling of the usertty() destination has been added. Until now, the usertty() destination were only disabled for blocking write() calls. (#3473<https://github.com/syslog-ng/syslog-ng/pull/3473>) Notes to developers * Proxy protocol support added to loggen. Four new options added to loggen to suppport the proxy protocol: * --proxied : Generate PROXY protocol v1 header * --proxy-src-ip : Set the source IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-dst-ip : Set the destination IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-src-port : Set the source port for the PROXY protocol v1 header. If not specified a random port generated in the range 5000-10000. * --proxy-dst-port : Set the destination port for the PROXY protocol v1 header. If not specified the port number 514 will be used. (#3462<https://github.com/syslog-ng/syslog-ng/pull/3462>) * bison: Minimum required version is now 3.4.2. You still only need bison, if you are building from git source or changing the grammar in the released source tarball. (#2526<https://github.com/syslog-ng/syslog-ng/pull/2526>) * Template evaluation related function signatures changed. A new structure LogTemplateEvalOptions is introduced to group parameters together. (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) Other changes * json-parser: change every per message logs that was higher than debug to debug (#3401<https://github.com/syslog-ng/syslog-ng/pull/3401>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Boris Korzun, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Viktor Juhasz, Vivin Peris, Zoltan Pallagi, bjoe2k4

1 0

syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 13 Nov '20

13 Nov '20

Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe

2 1

Insider 2020-10: LaaS; PAN-OS; Security Onion; New Relic;
by Peter Czanik (pczanik) 12 Nov '20

12 Nov '20

Dear syslog-ng users, This is the 86th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS How to use syslog-ng with LaaS and why? --------------------------------------- The first Logging as a Service (LaaS) I learned about many years ago was created by Loggly. Of course there are many more LaaS providers now. While most services also provide their own clients for sending log messages, many of them also document sending log messages to them using syslog-ng. >From this blog, you can learn about the advantages of using syslog-ng with LaaS, or with any cloud based SIEM solution or a Managed Security Service Provider (MSSP), and also how easy it is to create a configuration block for a new provider. While creating a configuration block takes some time, the simplification it provides can make your life easier on the long term. https://www.syslog-ng.com/community/b/blog/posts/how-to-use-syslog-ng-with-… Parsing PAN-OS logs using syslog-ng ----------------------------------- Version 3.29 of syslog-ng was released recently including a user-contributed feature: the panos-parser(). It is parsing log messages from PAN-OS (Palo Alto Networks Operating System). Unlike some other networking devices, the message headers of PAN-OS syslog messages are standards-compliant. However, if you want to act on your messages (filtering, alerting), you still need to parse the message part. The panos-parser() helps you create name-value pairs from the message part of the logs. >From this blog you can learn why it is useful to parse PAN-OS log messages and how to use the panos-parser(). https://www.syslog-ng.com/community/b/blog/posts/parsing-pan-os-logs-using-… Syslog-ng and Security Onion ---------------------------- One of the most interesting projects utilizing syslog-ng is Security Onion, a free and open source Linux distribution for threat hunting, enterprise security monitoring, and log management. It is utilizing syslog-ng for log collection and log transfer and uses the Elastic stack to store and search log messages. Even if you do not use its advanced security features, you can still use it for centralized log collection and as a nice web interface for your logs. But it is also worth getting acquainted with its security monitoring features, as it can show you useful insights about your network. Best of all, Security Onion is completely free and open source, with commercial support available for it. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-security-oni… How to Use Syslog-ng with New Relic ----------------------------------- Learn how to use the http() destination to send logs to New Relic: https://blog.newrelic.com/product-news/how-to-use-syslog-ng-with-new-relic/ WEBINARS * Improve SIEM performance and cut SIEM costs with log management solutions: https://www.syslog-ng.com/event/improve-siem-performance-and-cut-siem-costs… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Using usertty
by Alexandre Santos 04 Nov '20

04 Nov '20

Hi, I am trying to use usertty(*) to send log all messages with severity equal or higher than critical to every user logged. But I am not getting any messages in serial port or ssh. I am sending the configurations and the debug log in attachment. Can you help me to understand what is happening? Thanks in advance, Alex

2 7

Requesting help with Grouping-by function
by Maciek Solnicki 04 Nov '20

04 Nov '20

Hi All, I would like to ask for help with usage of grouping-by function. Goal is to correlate logs from auditd. Logs produced by auditd are multiline, but events share some number in key with name *msg. *This in theory should allow aggregation of logs with the same *msg *key, but I cannot get it right. Here is my config: *log {* * source {* * file('/var/log/audit/audit.log' flags(no-parse));* * };* * parser {* * linux-audit-parser(prefix(".auditd."));* * };* * parser {* * grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );* * };* * destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};* After implementing above config, output logs look like this, prefix is added as expected and format is set o JSON, but no aggregation happens: *{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}* *{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}* Syslog-ng version I'm using: *syslog-ng 3 (3.29.1)* *Config version: 3.29* *Installer-Version: 3.29.1* *Revision: 3.29.1-1* *Compile-Date: Aug 28 2020 12:13:25* *Module-Directory: /usr/lib/syslog-ng/3.29* *Module-Path: /usr/lib/syslog-ng/3.29* *Include-Path: /usr/share/syslog-ng/include* *Available-Modules: pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis* *Enable-Debug: off* *Enable-GProf: off* *Enable-Memtrace: off* *Enable-IPv6: on* *Enable-Spoof-Source: on* *Enable-TCP-Wrapper: on* *Enable-Linux-Caps: on* *Enable-Systemd: on* Can anyone please tell what am I doing wrong ? Kind regards Maciej

4 8