- syslog-ng - lists.balabit.hu

Problem with syslog-ng OpenSource and SELinux
by QUINONERO Paul (AREVA) 24 Aug '18

24 Aug '18

Hello everyone, I have an issue with syslog-ng on a Centos7 server. I configure a destination to use a python script for parsing my message, but, due to SELinux enforced mode the script is not called. Do you know any solution to fix this problem (I need to keep SELinux enforced)? Thanks in advance

3 3

Insider 2018-08: 3.16 & 3.17; Splunk; IoT security; Telegram; Throttling;
by Czanik, Péter 23 Aug '18

23 Aug '18

Dear syslog-ng users, This is the 68th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.16 & 3.17 are released ---------------------------------- Version 3.16 & 3.17 of syslog-ng are now available. Support to send log messages to Telegram was added. You can also use failback mode when you have failover enabled and send your logs to your primary log server as soon as it is available again. For a complete list of changes, check https://github.com/balabit/syslog-ng/releases/. For binary packages, check https://syslog-ng.com/3rd-party-binaries. Using the syslog-ng Store Box (SSB) in front of Splunk ------------------------------------------------------ The syslog-ng application was used for many years as a log collection layer in front of Splunk. But why use a full-blown log management appliance – like SSB – with a graphical user interface instead of a simple command line application? I learned the answers at Red Hat Summit while talking to my fellow Balabit engineers and booth visitors. https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-store-box-… IoT security: logging --------------------- Recently SANS published a brand new white paper about the Internet of Things: “Stopping IoT-based Attacks on Enterprise Networks”. IoT devices have been around in the networks of enterprises for many years, just think about network-connected printers. But recently, both the number and variety of these devices skyrocketed and enterprises now have to embrace everything from BYOD phones to smart lamps. In this blog post, I would like to highlight a critical aspect of how you can protect your organization from potential IoT-based attacks: logging. https://www.syslog-ng.com/community/b/blog/posts/iot-security-logging Telegram destination in syslog-ng --------------------------------- Getting started with the Telegram destination of syslog-ng is not an easy and straightforward process, but it is well worth the efforts. If you do not know Telegram yet, Telegram is a cloud-based messaging application known for its security and speed. Best of all, it is a free app without ads. No wonder that Telegram is also used by many system administrators. With the Telegram destination of syslog-ng introduced in version 3.16, we intend to help your work as a sysadmin. As a result, you can now receive critical log messages in real-time on your mobile or desktop Telegram client. https://www.syslog-ng.com/community/b/blog/posts/telegram-destination-syslo… hook-commands: easy driver setup -------------------------------- The hook-commands() option of syslog-ng makes it easy to execute external commands when a driver is started or stopped. For example, you can open a port in the firewall when a network source is started and close it once syslog-ng is shut down. Or you could also run a command each time syslog-ng is reloaded. From this blog, you can learn how to start and stop auditing file system changes using auditctl from syslog-ng, then collect, parse and filter the results, and finally send them to Elasticsearch for storage and further analysis. https://www.syslog-ng.com/community/b/blog/posts/hook-commands-easy-driver-… Throttling log messages ----------------------- One of the main advantages of syslog-ng is that it is high performance and low on resource usage. Why throttle the messages then? There are three main reasons – licensing, performance, and bandwidth – all outside of syslog-ng. From this blog, you can learn how to identify use cases for the throttling of log messages, read about a possible drawback, and finally get a sample configuration. https://www.syslog-ng.com/community/b/blog/posts/throttling-log-messages Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng with multiple destinations
by swald.ext＠orange.com 22 Aug '18

22 Aug '18

Hello, We are a syslog-ng configuration with multiple destinations : source s_remote_udp { udp(ip(0.0.0.0) port(514)); }; source s_remote_tcp { tcp(ip(0.0.0.0) port(514) max-connections(1000) so_keepalive(yes)); }; destination d_SIEM_ { udp("10.0.1.145" port(514)); }; destination d_BOX_ { file("/var/log/box/FLOW_$R_MONTH$R_DAY$R_HOUR$R_MIN.log" create_dirs(yes) dir_perm(0755) dir_owner(root) dir_group(root) owner(root) group(root)); }; filter f_SIEM_ { match("FLOW:"); }; log { source(s_remote_udp); source(s_remote_tcp); filter(f_SIEM_); destination(d_BOX_); destination(d_SIEM_);flags(flow-control,final); }; We can see kind of errors : /var/log/syslog-ng.log : Aug 4 23:34:39 host1 syslog-ng[16290]: I/O error occurred while reading; fd='32', error='Connection reset by peer (104)' We lose some logs. We have a syslog-ng version 3.2. Can you help please please ? How can I diagnose the problem? How do I get to the root of the problem? Maybe debugging syslog-ng? Thanks in advance. Best regards. Sebastian. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

2 3

Hosts before DNS
by Oleg 21 Aug '18

21 Aug '18

I am using the latest version of ose ng, and have an issue I am trying to resolve. We have hosts that resolve to multiple names via round robin dns. So ng is capturing logs from all of those, depending on how it was resolved during during the connection. For those types of hosts, I would like to configure ng to use hosts first, and fall back to dns resolution. Tried different combinations of configs, but it does not work this way. It either uses the hosts, or it does dns lookup. Thanks in advance for any tips on resolving this. Example: 10.0.0.1 resolve to www, app1, ftp I want to call it webserver in /etc/hosts, and if the entry matches, ng would just use that name. right now it creates 3 separate log files for the same host based on the name it's able to resolve at lookup.

4 3

Re: [syslog-ng] multi-line logs and program/facility filters
by Michael Thénault 14 Aug '18

14 Aug '18

Ok, well that cannot be a solution for us because of different reasons: performance (embedded environment), probability to add bugs ... I guess we'll have to go reconsider keeping syslogd which doesn't have this issue. Regards, Michael

4 5

documentation for snmp destination driver
by Armando Mártires 14 Aug '18

14 Aug '18

Hello everyone, I'm trying to forward some log messages through SNMP with syslog-ng 3.16 buy I can't seem to find the documentation for the snmp() driver, can anyone help me with this? Thank you, Armando Mártires

2 1

found a release-critical bug, 3.17.2
by Laszlo Budai 10 Aug '18

10 Aug '18

Hi, we are in the middle of an investigation of a partial-write issue and found a bug that can lead to message loss in some circumstances, even when reliable diskbuffer is used. We marked this bug as release critical. Fortunately we fixed the issue a few minutes ago, so we are going to release 3.17.2 that contains the fix ( https://github.com/balabit/syslog-ng/pull/2224 ). We don't have an automated testcase for this, actually it is in progress(this is why we didn't catch the bug), but sharing the manual reproduction: * listen with netcat (and use pv tool to make partial write happen) nc -l 61002 | pv -L 40 * run syslog-ng with the following config: @version: 3.17 options { time_reopen(3); stats_level(1); }; source s_network { network(ip("127.0.0.1") port(5555)); }; destination d_network { network("127.0.0.1" disk_buffer(mem-buf-size(1000) disk-buf-size(1048576) reliable(yes) dir('/tmp/dq')) ip_protocol(4) port(6666) so-sndbuf(4096)); }; log { source(s_network); destination(d_network); flags(flow-control); }; * send logs loggen -r 100000 -I 200 127.0.0.1 10001 --inet * wait a few seconds then stop netcat * result: syslog-ng will drops all the incoming messages regards, Laszlo Budai

1 1

multi-line logs and program/facility filters
by Michael Thénault 10 Aug '18

10 Aug '18

Hello, I have an Issue with syslog-ng 3.16.1 and multi-line logs. I try to configure per-application filters using either the program name or a facility. The applications use the traditional syslog() from syslog.h. When an application logs multiple lines, only the first line is filtered. Indeed, the program name or facility is only applied to the first line. Example : $ logger -t testprog "line1 line2 line3" $ cat /var/log/messages 2018-08-10T16:26:14.000000+02:00 testprog: line1 2018-08-10T16:26:14.899505+02:00 line2 2018-08-10T16:26:14.899505+02:00 line3 The log source is unix-stream("/dev/log" ); What can I do to fix this ? Thanks in advance for your help. Thanks & Regards, Michael

2 1

unofficial packages for syslog-ng (Debian/Ubuntu)
by Laszlo Budai 10 Aug '18

10 Aug '18

Hi, syslog-ng 3.17.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 In case of Debian 9.0, armv7l architecture also supported, so you can install it to your Raspberry Pi. Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

5 5

OSE 3.17.1
by Laszlo Budai 09 Aug '18

09 Aug '18

3.17.1 Features * Client side failback mode (#2183<https://github.com/balabit/syslog-ng/pull/2183>) * New linux-audit() source as SCL (#2186<https://github.com/balabit/syslog-ng/pull/2186>) * Decorating generated configuration (#2218<https://github.com/balabit/syslog-ng/pull/2218>) * Introduce ewmm() source (#2199<https://github.com/balabit/syslog-ng/pull/2199>, #2209<https://github.com/balabit/syslog-ng/pull/2209>) * Add parsing of Cisco unified call manager (#2134<https://github.com/balabit/syslog-ng/pull/2134>) * Mandatory parameters for cfg-block (SCL) (#2088<https://github.com/balabit/syslog-ng/pull/2088>) Bugfixes * dqtool cat print backlog items (#2213<https://github.com/balabit/syslog-ng/pull/2213>) * Rewind backlog in case of stateless LogProtoClient (#2214<https://github.com/balabit/syslog-ng/pull/2214>) * Filter out incorrectly parsed sudo logs (#2208<https://github.com/balabit/syslog-ng/pull/2208>) * Minor fixes related to client-lib-dir, loggen and eventlog (#2204<https://github.com/balabit/syslog-ng/pull/2204>) * Minor stats-query fixes and refactor (#2193<https://github.com/balabit/syslog-ng/pull/2193>) * Reliable disk buffer non-empty backlog (#2192<https://github.com/balabit/syslog-ng/pull/2192>) * Fix pip package versions on older distro releases (dbld) (#2188<https://github.com/balabit/syslog-ng/pull/2188>) * Fix a groupset/groupunset and map-value-pairs() crash (#2184<https://github.com/balabit/syslog-ng/pull/2184>) * Make g_atomic_counter_set() atomic and update minimum glib version to 2.26 (#2182<https://github.com/balabit/syslog-ng/pull/2182>) * Aligning java related SCLs with mandatory parameters (#2160<https://github.com/balabit/syslog-ng/pull/2160>) * Loggen minor fixes (#2150<https://github.com/balabit/syslog-ng/pull/2150>) * grab-logging should be installed as a header (#2151<https://github.com/balabit/syslog-ng/pull/2151>) * Fix possible underflow of memory_usage (afsql, logqueue-fifo) (#2140<https://github.com/balabit/syslog-ng/pull/2140>) * Fix SELinux module version inconsistency (#2133<https://github.com/balabit/syslog-ng/pull/2133>) * Fix CMake unit test compilation (no-pie) (#2137<https://github.com/balabit/syslog-ng/pull/2137>) * Fix possible crash in syslog-parser() (#2128<https://github.com/balabit/syslog-ng/pull/2128>) * Disable ack for mark mode (#2129<https://github.com/balabit/syslog-ng/pull/2129>) * Fixing a Telegram destination bug with bot id (#2142<https://github.com/balabit/syslog-ng/pull/2142>) * All drivers should support inner destination or source plugins (#2143<https://github.com/balabit/syslog-ng/pull/2143>) * Fix default file and directory creation ownership (#2119<https://github.com/balabit/syslog-ng/pull/2119>) * Fix global "center;;received" counter when systemd-journal() is used (#2121<https://github.com/balabit/syslog-ng/pull/2121>) * Link everything to libsecret-storage (#2100<https://github.com/balabit/syslog-ng/pull/2100>) * Inform about the right dns-cache() configuration (warning message typo) (#2145<https://github.com/balabit/syslog-ng/pull/2145>) * Adjusting window size for internal mark mode (#2146<https://github.com/balabit/syslog-ng/pull/2146>) * Fix memory leaks in disk-buffer() (#2153<https://github.com/balabit/syslog-ng/pull/2153>) * Fix undefined behavior in log multiplexer (#2154<https://github.com/balabit/syslog-ng/pull/2154>) * Fix static linking mode (autotools) (#2155<https://github.com/balabit/syslog-ng/pull/2155>) * Fix internal mark mode infinite loop with old ivykis (#2157<https://github.com/balabit/syslog-ng/pull/2157>) * Fix missing normalize flags (#2162<https://github.com/balabit/syslog-ng/pull/2162>) * Keep JVM running on reload if once configured (#2164<https://github.com/balabit/syslog-ng/pull/2164>, #2211<https://github.com/balabit/syslog-ng/pull/2211>) * Fix a race condition (suspend) in LogSource (#2167<https://github.com/balabit/syslog-ng/pull/2167>) * Add @requires json-plugin to the cim() parser (#2181<https://github.com/balabit/syslog-ng/pull/2181>) * Added exclude_kmsg option to system source (#2166<https://github.com/balabit/syslog-ng/pull/2166>) * Fix padding template function (#2174<https://github.com/balabit/syslog-ng/pull/2174>) * Leak & invalid memory access (#2173<https://github.com/balabit/syslog-ng/pull/2173>) * FreeBSD 11.2 builderror SOCK_STREAM (#2170<https://github.com/balabit/syslog-ng/pull/2170>) * Add ref-counted TLSVerifier struct (use after free fix) (#2168<https://github.com/balabit/syslog-ng/pull/2168>) Other changes * Improve loggen's file message parser (#2205<https://github.com/balabit/syslog-ng/pull/2205>) * syslog-ng-debun improvements (#2201<https://github.com/balabit/syslog-ng/pull/2201>) * Goodbye "goto relex" (refactor) (#2198<https://github.com/balabit/syslog-ng/pull/2198>) * Refactor the callback registration mechanism of WildcardFileReader (#2185<https://github.com/balabit/syslog-ng/pull/2185>) * Extended Linux capabilities detection (pkg-config) (#2169<https://github.com/balabit/syslog-ng/pull/2169>) * Add atomic gssize (#2159<https://github.com/balabit/syslog-ng/pull/2159>) * Lower the message level of @requires to debug (#2147<https://github.com/balabit/syslog-ng/pull/2147>) * macOS warning elimination (#2139<https://github.com/balabit/syslog-ng/pull/2139>) * Remove a misleading rewrite-related debug message (#2132<https://github.com/balabit/syslog-ng/pull/2132>) * Minor updates to SELinux policy installer script (#2127<https://github.com/balabit/syslog-ng/pull/2127>) * More robust GLib detection (CMake) (#2125<https://github.com/balabit/syslog-ng/pull/2125>) * Logthreaded nonfunctional changes (#2123<https://github.com/balabit/syslog-ng/pull/2123>) * Confgen and pragma improvements (#2122<https://github.com/balabit/syslog-ng/pull/2122>) * Flush before stopping syslog-ng (functional tests) (#2216<https://github.com/balabit/syslog-ng/pull/2216>) * Port unit tests into criterion (test_filters_netmask6, test_findeom, csv_parser, patternDB) (#2217<https://github.com/balabit/syslog-ng/pull/2217>, #2175<https://github.com/balabit/syslog-ng/pull/2175>, #2118<https://github.com/balabit/syslog-ng/pull/2118>) * Libtest refactors (#2149<https://github.com/balabit/syslog-ng/pull/2149>) * Add missing files to the source tarball (#2114<https://github.com/balabit/syslog-ng/pull/2114>) * Better python binary detection (#2092<https://github.com/balabit/syslog-ng/pull/2092>) Notes to the developers * LogThreadedDestDriver batching (#2063<https://github.com/balabit/syslog-ng/pull/2063>) * Add sqlite3 and riemann to dbld devshell (#2210<https://github.com/balabit/syslog-ng/pull/2210>) * Make mock-transport inheritable (#2120<https://github.com/balabit/syslog-ng/pull/2120>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, Bernie Harris, Bertrand Jacquin, Gabor Nagy, Gergely Nagy, German Service Network, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Szigetvari Janos, Terez Nemes, Viktor Juhasz. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. github.com regards, Laszlo Budai

1 0