- syslog-ng - lists.balabit.hu

ACLs with syslog-ng
by Faine, Mark R. (MSFC-IS40)[NICS] 19 May '21

19 May '21

RHEL 7 and 8 syslog-ng 3.19 and 3.31 We have always set permissions on directories that we want the Splunk universal forwarder to be able to read as root:splunk 640, but now security doesn't like this and wants everything under /var/log to always be root:root except for some specific exceptions. We had tried to solve this with an ACL in the past, however, syslog-ng always seems to clobber the ACL, even when it's the default ACL on the folder. Is this a known issue, is there a way to get syslog-ng to play nice with ACLs. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

2 4

Insider 2021-05: 3.32; Fortigate; Windows event collector; FreeBSD; Pi;
by Peter Czanik (pczanik) 13 May '21

13 May '21

Dear syslog-ng users, This is the 91st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.32 is now available ------------------------------- Highlights include multi-threaded connection pooling to MongoDB servers and template support for its collection() option. OpenBSD support is now fixed. For a complete list of changes, check the release notes at: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.32.1 Parsing Fortigate logs and other syslog-ng 3.31 news ---------------------------------------------------- Version 3.31 of syslog-ng has been released recently. One of its most user-visible features is the parser for Fortigate logs, yet another networking vendor that produces log messages not conforming to syslog specifications. Parsing Fortigate logs builds upon the new no-header flag of syslog-ng combined with the key-value and date parsers. Other features include a new silent message option for the Telegram destination and automatic directory creation for disk-buffer files. https://www.syslog-ng.com/community/b/blog/posts/parsing-fortigate-logs-and… Collecting Windows EventLog without installing extra software on Windows -------------------------------------------- Talking to users at both IT security and operations related conferences, they confessed the same: the less software the better. Less software to learn, maintain and follow in bug trackers. The Windows Event Collector (WEC) component of syslog-ng Premium Edition (PE) brings this to the extremes: it enables Windows machines to act as log sources without installing any software. Just enable the WEC component of syslog-ng PE on your Linux server and enjoy the logs flowing in. From this blog you can learn how to set up a test environment for WEC. I will also show you an easier way to prepare the certificates used, help you avoid some of the failures, and point you to alternative ways, where necessary. https://www.syslog-ng.com/community/b/blog/posts/windows-event-collector-fo… syslog-ng on the FreeBSD wiki ----------------------------- My original blog about getting started with syslog-ng on FreeBSD became outdated. The content was updated and extended a bit, and with some help from the FreeBSD community, it is now available on the FreeBSD wiki at https://wiki.freebsd.org/Ports/sysutils/syslog-ng Syslog-ng & Pi day ------------------ We had March 14th recently, or as many geeks refer to it: Pi day. On this occasion, I would like to show you a syslog-ng configuration, which prints a huge π on the screen, and two recent articles that feature syslog-ng on the Raspberry Pi. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-pi-day WEBINARS * syslog-ng, Swiss Army Knife of Log Management: https://www.syslog-ng.com/event/syslogng-swiss-army-knife-of-log-management… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Changing message transfer for rfc6587
by Hartmann Benedict - Maschinenfabrik Reinhausen 12 May '21

12 May '21

Hello, at the moment I am sending messages via destination d_syslog_visu_conf_IETF { syslog("`IPADRESS`" transport(TCP) port(`PORT`) `TLS_CERT`) ; }; This approach will lead to the rfc6587 standard. The messages are collected via visual syslog. Is it possible to change the message transfer mode from "octet counting" (leading message length) to "non-transparent-framing" (no leading message length) via config? Best regards Benedict Hartmann Maschinenfabrik Reinhausen GmbH General Manager / Geschäftsführer: Dr. Nicolas Maier-Scheubeck, Wilfried Breuer, Holger Michalka Chairman of the Board / Vors. des Aufsichtsrats: Hans-Jürgen Thaus Registered Office / Sitz der Gesellschaft: Regensburg Local Court Regensburg / Amtsgericht Regensburg HRB 3687 This e-mail and any attachments contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. This e-mail was scanned for viruses, vandals and malicious content.

2 1

syslog-ng mail list is fixed
by Attila Szakacs (aszakacs) 10 May '21

10 May '21

Hello fellow syslog-ng users, Apologies for the past few weeks' syslog-ng mail list outage. We have had some issues with our mail server, but everything seems to be resolved now. If you still cannot send mails to syslog-ng(a)lists.balabit.hu, please write a mail to me directly: attila.szakacs(a)oneidentity.com. Cheers, Attila

1 0

syslog-ng-3.32.1
by Peter Kokai (pkokai) 10 May '21

10 May '21

3.32.1 ====== ## Highlights * `mongodb()`: add `workers()` support (multi-threaded connection pooling) The MongoDB driver now supports the `workers()` option, which specifies the number of parallel workers to be used. Workers are based on the connection pooling feature of the MongoDB C library. This increases the throughput of the MongoDB destination driver. Example: ``` destination { mongodb( uri("mongodb://hostA,hostB/syslog?replicaSet=my_rs&wtimeoutMS=10000&socketTimeoutMS=10000&connectTimeoutMS=10000&serverSelectionTimeoutMS=5000") collection("messages") workers(8) ); }; ``` ([#3621](https://github.com/syslog-ng/syslog-ng/pull/3621)) * `mongodb()`: template support for the `collection()` option The `collection()` option of the MongoDB destination driver now accepts templates, for example: ``` destination { mongodb( uri("mongodb://host/syslog") collection("${HOST}_messages") ); }; ``` ([#3621](https://github.com/syslog-ng/syslog-ng/pull/3621)) ## Features * `time-reopen`: Support the `time-reopen()` option on the driver level for the following drivers: * sources: `example-diskq-source`, `python-fetcher` * destinations: `amqp`, `example-destination`, `file`, `http`, `mongodb`, `network`, `pipe`, `program`, `pseudofile`, `python`, `redis`, `riemann`, `smtp`, `sql`, `stomp`, `syslog`, `tcp`, `tcp6`, `udp`, `udp6`, `unix-dgram`, `unix-stream`, `usertty` ([#3585](https://github.com/syslog-ng/syslog-ng/pull/3585)) * `csv-parser()`: add drop-invalid() option along with the already existing flag with the same name. This is to improve the consistency of the configuration language. ([#3547](https://github.com/syslog-ng/syslog-ng/pull/3547)) * `usertty() destination`: Support changing the terminal disable timeout with the `time-reopen()` option. Default timeout change to 60 from 600. If you wish to use the old 600 timeout, add `time-reopen(600)` to your config in the `usertty()` driver. ([#3585](https://github.com/syslog-ng/syslog-ng/pull/3585)) * `syslog-parser()`: add a new drop-invalid() option that allows the use of syslog-parser() in if statements. Normally a syslog-parser() injects an error message instead of failing. ([#3565](https://github.com/syslog-ng/syslog-ng/pull/3565)) ## Bugfixes * date-parser: if the timestamp pattern did not covered a field (for example seconds) that field had undefined value The missing fields are initialized according to the following rules: 1) missing all fields -> use current date 2) only miss year -> guess year based on current year and month (current year, last year or next year) 3) the rest of the cases don't make much sense, so zero initialization of the missing field makes sense. And the year is initialized to the current one. ([#3615](https://github.com/syslog-ng/syslog-ng/pull/3615)) * Fix compilation issues on OpenBSD syslog-ng can now be compiled on OpenBSD. ([#3661](https://github.com/syslog-ng/syslog-ng/pull/3661)) * loggen: debug message printed wrong plugin name (ssl-plugin instead of socket_plugin) ([#3624](https://github.com/syslog-ng/syslog-ng/pull/3624)) * tls: fixup EOF detection issue in tls (before 3.0 version) syslog-ng error message: "I/O error occurred while reading; fd='13', error='Success (0)'" ([#3618](https://github.com/syslog-ng/syslog-ng/pull/3618)) * kafka: the config() block couldn't contain option that is already a keyword in syslog-ng (example: retries) ([#3658](https://github.com/syslog-ng/syslog-ng/pull/3658)) * templates: fixed error reporting when invalid templates were specified The `amqp()`, `file()` destination, `sql()`, `stomp()`, `pdbtool`, and `graphite()` plugins had template options that did not report errors at startup when invalid values were specified. ([#3660](https://github.com/syslog-ng/syslog-ng/pull/3660)) ## Packaging * bison: minimum version of bison is bumped to 3.7.6 ([#3547](https://github.com/syslog-ng/syslog-ng/pull/3547)) * java-modules: the minimum version of gradle changed from 2.2 to 3.4 ([#3645](https://github.com/syslog-ng/syslog-ng/pull/3645)) * light: add to the release tarball ([#3613](https://github.com/syslog-ng/syslog-ng/pull/3613)) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Ming Liu, Norbert Takacs, Peter Kokai, Todd C. Miller, Yi Fan Yu, Zoltan Pallagi

1 0

Syslog-ng + Consuming syslog alerts via FIFO: Alarms being lost, no logs related anywhere
by Richter Vitali Guedes 16 Apr '21

16 Apr '21

This is my first time posting here. I hope you can help me. We have 8 collector servers running syslog probes. They also run a collector application. Syslog-ng receives all alarms through UDP Ports, filter them, and write a fifo. It also forwards these alarms through UDP to another server. We are using this another server that receives these forwarded messages as a validation point. There are some filter rules for the pipe delivery, and no filters for UDP delivery. But we could see several not filtered alarms being received at the UDP receiver server, but not being received by the collector application, that reads from the fifo (/opt/solution/src1.pipe). Collector logs have no error lines. I tried rewrititng syslog-ng configuration file, in order to use flow-control, that, according to syslog-ng documentation, is necessary for high volume environments (as this is the case here, 500-1000 alarms per second). I applied it yesterday, together with net.core.rmem_max=1024000, and the scenario got even worst. We noticed missing messages on the UDP destination server and on the fifo. Dropped packets counter on ifconfig was increasing quickly. We rolled back to the previous syslog-ng configuration. Below, you will find the first version of the syslog-ng.conf file, without flow-control (several names , filters and addresses were changed in sake of privacy): @version:3.19 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; template t_customer { template("${DATE} ${SOURCEIP} ${HOST} ${MSGHDR} ${MESSAGE}\n"); }; # UDP Source destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; #log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; #Debug filter f_debu_1 { not level(debug); }; #Information filter f_info_1 { not match("PROGRAM-" value("MESSAGE")) }; filter f_info_2 { not match("Multicast Client" value("MESSAGE")); }; filter f_info_3 { not match("executeCommand" value("MESSAGE")); }; filter f_info_4 { not match("has the privilege to execute" value("MESSAGE")); }; filter f_info_5 { not match("Telnet " value("MESSAGE")); }; #Warning filter f_warn_1 { not match("LSP went Up" value("MESSAGE")); }; filter f_warn_2 { not match("LSP went Down" value("MESSAGE")); }; filter f_warn_3 { not match("Subscriber created" value("MESSAGE")); }; filter f_warn_4 { not match("Subscriber deleted" value("MESSAGE")); }; filter f_warn_5 { not match("BGP-WARN" value("MESSAGE")); }; filter f_warn_6 { not match("DHCPS-WARNING" value("MESSAGE")); }; filter f_warn_7 { not match("DHCP-WARNING" value("MESSAGE")); }; #Notice filter f_noti_1 { not match("iCDR\[[0-9]+\]:"); }; #Error filter f_erro_1 { not match("FAN started working" value("MESSAGE")); }; filter f_erro_2 { not match("FAN is not working" value("MESSAGE")); }; #Alert filter f_aler_1 { not match("LDP virtual tunnel went" value("MESSAGE")); }; #Critical filter f_crit_1 { not match("traps dropped" value("MESSAGE")); }; filter f_crit_2 { not match("upstream E1 signals" value("MESSAGE")); }; filter f_crit_3 { not match(",ABCDE," value("MESSAGE")); }; filter f_crit_4 { not match(",XYZW," value("MESSAGE")); }; filter f_crit_5 { not match(" for ABCDE-[0-9]" value("MESSAGE")); }; source s_dummy{ udp(ip(0.0.0.0) port(5140)); }; source src1{ udp(ip(0.0.0.0) port(514)); }; source s_10514{ udp(ip(0.0.0.0) port(10514)); }; source src2{ udp(ip(0.0.0.0) port(515)); }; destination d_src1 { pipe("/opt/solution/src1.pipe" template(t_customer)); }; destination d_src2 { pipe("/opt/solution/src2.pipe" template(t_customer)); }; destination d_syslogCustom{ pipe("/opt/solution/syslog_custom.pipe" template(t_customer)); }; destination d_udp_dst{ syslog("192.168.1.151" transport("udp") port(514) spoof_source(no)); }; filter f_src2 { host("r(core|distribution|management).*") or host("^abcd.*") or host("^d2") or host("src2") or host("xyz1") or host("^jkl") or host("asddf") or host("zxcvb"); }; log { source(src1); source(s_10514); filter(f_debu_1); filter(f_info_1); filter(f_info_2); filter(f_info_3); filter(f_info_4); filter(f_info_5); filter(f_warn_1); filter(f_warn_2); filter(f_warn_3); filter(f_warn_4); filter(f_warn_5); filter(f_warn_6); filter(f_warn_7); filter(f_noti_1); filter(f_erro_1); filter(f_erro_2); filter(f_aler_1); filter(f_crit_1); filter(f_crit_2); filter(f_crit_3); filter(f_crit_4); filter(f_crit_5); destination(d_src1); }; log { source(src2); filter(f_src2); destination(d_src2); }; log { source(src1); destination(d_udp_dst); }; That's all. I Also tried using flow-control. The main achanges are below: source src1{ # udp( network( #new driver syntax transport("udp") #new driver syntax ip(0.0.0.0) port(514) max-connections(3000) ); }; source s_10514{ # udp( network( #new driver syntax transport("udp") #new driver syntax ip(0.0.0.0) port(10514) max_connections(3000) ); }; source src2{ # udp( network( #new driver syntax transport("udp") #new driver syntax ip(0.0.0.0) port(515) max_connections(3000) ); }; destination d_src1 { pipe( "/opt/solution/src1.pipe" template(t_customer) log_fifo_size(600000) ); }; destination d_src2 { pipe( "/opt/solution/src2.pipe" template(t_customer) log_fifo_size(300000) ); }; destination d_udp_dst{ syslog( "192.168.1.151" transport("udp") port(514) spoof_source(no) log_fifo_size(300000) ); }; And the situation got even worst. So, I would like to ask you: How can I fix it? I need to be sure that all data received is forwarded to their destinations, respecting the filters, and without any information loss. I'm quite sure all data is being received, I've checked it using tcpdump, no udp packet loss during transmission. The most important sessions are: log { source(src1); source(s_10514); filter(f_debu_1); filter(f_info_1); filter(f_info_2); filter(f_info_3); filter(f_info_4); filter(f_info_5); filter(f_warn_1); filter(f_warn_2); filter(f_warn_3); filter(f_warn_4); filter(f_warn_5); filter(f_warn_6); filter(f_warn_7); filter(f_noti_1); filter(f_erro_1); filter(f_erro_2); filter(f_aler_1); filter(f_crit_1); filter(f_crit_2); filter(f_crit_3); filter(f_crit_4); filter(f_crit_5); destination(d_src1); }; log { source(src1); destination(d_udp_dst); }; Thanks a lot, and sorry for the huge message, Richter <https://www.linkedin.com/company/icarotech>

2 3

Insider 2021-04: Grafana; Windows agent; BSD;
by Peter Czanik (pczanik) 15 Apr '21

15 Apr '21

Dear syslog-ng users, This is the 90th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Grafana, Loki, syslog-ng: jump-starting a new logging stack ----------------------------------------------------------- Talking to syslog-ng users, I found that many of them plan to take a closer look at Grafana, due to the upheaval around the change of licensing terms for Elastic. Luckily, it is now possible to jump-start the complete, new logging stack – including Grafana, Loki, syslog-ng and tools to monitor this stack – with a single command. All you need to do is to point a couple of syslog clients at the included syslog-ng server and open Grafana in your browser. Of course, this setup is far from being production-ready, but it can speed up preparing a test environment for you. From this blog, you can learn how to install Grafana, Loki, syslog-ng stack, how to forward your log messages there, and how to check the results in Grafana. https://www.syslog-ng.com/community/b/blog/posts/grafana-loki-syslog-ng-jum… When to use the syslog-ng agent for Windows? -------------------------------------------- You can collect log messages from a Windows host in multiple ways using syslog-ng. For large scale installations the easiest is to use the Windows Event Collector (WEC) component of syslog-ng Premium Edition (PE). This way you don’t have to install any new client software on the Windows side, just point the WEC to the destionation to send their log messages. Please note that WEC only works for Windows EventLog. If you need to collect log messages from text files, you need to install the syslog-ng agent for Windows on your hosts. For example, web servers often log to files instead of Windows EventLog. Let’s review how to do a standalone installation of syslog-ng agent for Windows and then see the differences between using the legacy (RFC3164) and the new (RFC5424) syslog protocol. https://www.syslog-ng.com/community/b/blog/posts/when-to-use-the-syslog-ng-… Syslog-ng on BSDs ----------------- My FOSDEM presentation in the BSD devroom showcased what is new in sudo and syslog-ng and explained how to install or compile these software yourself on FreeBSD. Not only am I a long time FreeBSD user (started with version 1.0 in 1994) I also work on keeping the syslog-ng port in FreeBSD up to date. But soon after my presentation I was asked what I knew about other BSDs. And – while I knew that all BSDs have syslog-ng in their ports system – I realized I had no idea about the shape of those ports. For this article I installed OpenBSD, DragonFlyBSD and NetBSD to check syslog-ng on them. Admittedly, they are not in the best shape: they contain old versions, some do not even start or are unable to collect local log messages. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-on-bsds WEBINARS * Scale log collection from Windows endpoints with WEC clustering in syslog-ng https://www.syslog-ng.com/event/scale-log-collection-from-windows-endpoints… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Try again: What's wrong with my config?
by Dan Egli 13 Apr '21

13 Apr '21

Okay. I'm completely stumped on this, and just before the list went down I was hoping someone could help me with this. I'm trying to break everything out of the monolithic /var/log/messages and place each service in it's own log file. To that extent, I created the following config file: syslog-ng config: @version: 3.30 @include "scl.conf" options { threaded(yes); chain_hostnames(no); stats_freq(43200); mark_freq(3600); }; source src { system(); internal(); }; filter samba { program("samba") or program("nmbd") or program("smbd"); }; filter sshd { program("ssdhd"); }; filter syslog { not filter(sshd) and not filter(samba); }; destination console { file("/dev/tty12"); }; destination messages { file("/var/log/messages"); }; destination sshd_log { file("/var/log/sshd/sshd.log"); }; log { source(src); filter(sshd); destination(sshd_log); flags(final); }; log { source(src); filter(syslog); destination(console); }; log { source(src); filter(syslog); destination(messages); }; so, as I understand the logic. The three log { } lines do this: log { source(src); filter(sshd); destination(sshd_log); flags(final); }; Anything from sshd gets written to the /var/log/sshd/sshd.log. Nothing else goes here. log { source(src); filter(syslog); destination(console); }; Anything that is not from sshd, not from smbd, not from sabma and not from nmbd goes to the /dev/tty12 device log { source(src); filter(syslog); destination(messages); }; likewises for /var/log/messages. Is my understanding correct? If so, WHY do I see ssh log entries in /var/log/messages? And how do I stop it!? sshd messages should ONLY show up in /etc/sshd/sshd.log. jupiter ~ # grep sshd /var/log/messages | head -n 2 Apr 13 00:00:50 jupiter sshd[14721]: Received disconnect from <IP> port 18726:11: [preauth] Apr 13 00:00:50 jupiter sshd[14721]: Disconnected from <IP> port 18726 [preauth] Thanks!

3 3

Re: [syslog-ng] Try again: What's wrong with my config?
by Dan Egli 13 Apr '21

13 Apr '21

I wish I knew how is was happening. On 4/13/2021 12:26 AM, Dan Egli wrote: > > Okay. I'm completely stumped on this, and just before the list went > down I was hoping someone could help me with this. > > I'm trying to break everything out of the monolithic /var/log/messages > and place each service in it's own log file. To that extent, I created > the following config file: > >

1 0

Re: [syslog-ng] Try again: What's wrong with my config?
by Dan Egli 13 Apr '21

13 Apr '21

crap. Not that bug again. Ignore this, and watch for my completed message. Sorry. On 4/13/2021 12:12 AM, Dan Egli wrote: > > Okay. I'm completely stumped on this, and just before the list went > down I was hoping someone could help me with this. >

1 0