April 2017 - syslog-ng - lists.balabit.hu

RSS memory grows when disk is full
by Arnob Ghosh 20 Apr '17

20 Apr '17

Hi, We are currently using syslog-ng version 3.3.6 . We are observing RSS memory growing issue for syslog-ng when destination disk is full. Is that an expected behavior , as the incoming message gets queued up in system RAM and thus RSS also will grow for syslog-ng process. Please correct me if I am wrong in understanding the same. Thanks, Arnob

1 0

down network during reload leads to blocking syslog() calls
by Nathan Parrish 18 Apr '17

18 Apr '17

Hi there, we are using syslog-ng 3.6.4 on Linux. we had an incident where the network port used for remote logging was inadvertently disabled for a couple hours, and during this time a critical process which logged via syslog() calls experienced threads hanging for seconds to minutes at a time. after some investigation and reproduction efforts, it looks like the problem is a combination of: - remote logging to server specified by hostname (vs. IP) - loss of management interface (e.g. that used for syslog traffic and DNS resolution) - log rotate triggering syslog-ng reload when I reproduce the problem (seems to take some considerable amount of logging load and a minute or so), I can see that writes to /dev/log hang: root@escort-ct0:/etc# strace -ttT logger test ^C 19:38:46.064830 execve("/usr/bin/logger", ["logger", "test"], [/* 26 vars */]) = 0 <0.000135> ... 19:38:46.069757 socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 1 <0.000015> 19:38:46.069806 connect(1, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = 0 <0.000010> 19:38:46.069850 sendto(1, "<13>Mar 20 19:38:46 root: test", 30, MSG_NOSIGNAL, NULL, 0) = 30 <11.627190> 19:38:57.697110 close(1)= 0 <0.000490> meanwhile I see the main thread timing out name resolution: 19:48:18.214182 stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=148, ...}) = 0 <0.000008> 19:48:18.214234 socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 31 <0.000012> 19:48:18.214279 connect(31, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.15.83.11")}, 16) = 0 <0.000024> 19:48:18.214332 poll([{fd=31, events=POLLOUT}], 1, 0) = 1 ([{fd=31, revents=POLLOUT}]) <0.000006> 19:48:18.214370 sendto(31, "\343\235\1\0\0\1\0\0\0\0\0\0\rescdev-syslog\3dev\vpurestorage\3com\0\0\1\0\1", 51, MSG_NOSIGNAL, NULL, 0) = 51 <0.000047> 19:48:18.214444 poll([{fd=31, events=POLLIN}], 1, 5000) = 0 (Timeout) <5.005040> this repeats with the other DNS servers configured. I cannot reproduce the issue if I configure an IP instead of hostname for the syslog server. we are using UDP and no flow-control configuration, with the expectation that syslog() will never block. and, indeed, until the reload, it works as expected. however, after reload I guess we re-establish the socket for the remote connection, requiring us to resolve the hostname; I don’t pretend to understand how this ultimately backs up processing of /dev/log (note that internal() and kernel messages are coming through just fine during this time). I guess my question is whether this is a known/expected issue, and/or if there’s a resolution other than specifying remote syslog servers by IP or hardcoding the name resolution in /etc/hosts and pointing to that with dns-cache-hosts(). basically, I’d like syslog-ng to simply give up if it can’t resolve remote syslog server hostnames, rather than allow this to interfere with servicing of /dev/log, with ramifications to callers. thanks in advance, nathan

2 1

Insider 2017-04: Docker; Splunk HEC; Suricata; videos; upcoming events;
by Czanik, Péter 13 Apr '17

13 Apr '17

Dear syslog-ng users, This is the 57th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Central log server in Docker ---------------------------- Containerization, and Docker in particular, changed the way we distribute and run applications. Your central log server can also run in a Docker container. If you wish to deploy your log server running syslog-ng in a Docker container, it is available as a ready-to-use image from the Docker Hub, already passing 500K pulls. Read how at https://www.balabit.com/blog/central-log-server-docker/ Feeding the Splunk HTTP event collector --------------------------------------- Using the HTTP destination of syslog-ng, you can feed the Splunk HTTP event collector (HEC) with log messages. This can simplify your logging architecture, because there is no need to store data in files and to use a Splunk forwarder. Read more about how to configure it on the Splunk website at https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregate… syslog-ng PE 7.0.2 available ---------------------------- The latest syslog-ng Premium Edition release adds two interesting new features: you can write message parsers and template functions in Python and the monitoring() source allows you to granularly select which statistics of syslog-ng PE you want to monitor. https://www.balabit.com/blog/monitor-your-syslog-ng-architecture-easier/ Collecting and parsing Suricata logs ------------------------------------ You can use syslog-ng to collect and parse the JSON-based log messages of Suricata. Learn how you can send these logs to Loggly or Elasticsearch for further analysis or configure simple alerting within syslog-ng: https://www.balabit.com/blog/collecting-and-parsing-suricata-logs-using-sys… Tutorial videos --------------- Patrick Bailey (https://twitter.com/whiteboardcoder) created a couple of tutorial videos about syslog-ng. These cover installation and initial configuration on Ubuntu, opening a network port for collecting log messages and sending JSON-based log messages: - https://www.youtube.com/watch?v=DrfBU9nBeoE - https://www.youtube.com/watch?v=YAYfBteY0kg - https://www.youtube.com/watch?v=Uejb0agO2NU UPCOMING EVENTS You can learn about syslog-ng at a growing number of events: HEPIX Spring Workshop: https://indico.cern.ch/event/595396/ Big Data Universe: https://bdu.hu/ openSUSE conference: https://events.opensuse.org/conference/oSC17 NEW RELEASES syslog-ng PE 7.0.2: https://www.balabit.com/documents/syslog-ng-pe-7.0-guides/en/syslog-ng-pe-g… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

syslog-ng buffer and reload
by Fabien Wernli 13 Apr '17

13 Apr '17

Hi, I'm using the disk queue to buffer the writes to elasticsearch. I notices the following behaviour: When reloading the configuration (`syslog-ng-ctl reload`), syslog-ng stops processing incoming messages (they appear as `dropped` in stats) and starts emptying the queue. It only starts accepting new messages when the queue is completely empty. I understand this is probably an expected behaviour, but in the following scenario (I just experienced) it poses a problem: 1. some application goes bananas logging at zillions of events per second 2. syslog-ng queue starts filling up 3. crazy app identified: I modify syslog-ng.conf in order to filter out the app 4. syslog-ng-ctl reload 5. syslog-ng starts dropping all new messages and emptying the queue 6. I have to wait for the queue to be empty (which can take a long time) Wouldn't it be saner to continue accepting messages when intercepting the HUP?

4 6

is it exist some way to extract the filename listened by file() ?
by Jorge Pereira 08 Apr '17

08 Apr '17

Hi Team, Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload. My current approach is. 1. I have the below destination() receiving the file name as a parameter. <snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip> 2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn() <snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; }; log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; }; ......................... </snip> But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()? -- Jorge Pereira

3 7

set mysql charset & collation
by Nik Ambrosch 04 Apr '17

04 Apr '17

I’m having a hard time getting the charset and collation of mysql tables set with syslog-ng: it needs to be utf8 (or utf8mb4) and from the documentation I’m under the impression that I can do something like this in order to set session variables: session-statements("set character_set_database=utf8mb4","set collation_database=utf8mb4_unicode_ci”) …however it’s not working - new tables are still being created as latin1. As far as I’m aware there’s also not a way to specify the charset/collation of a new table via syslog-ng either, right? Any ideas?

1 0