- syslog-ng - lists.balabit.hu

syslog-ng running but not writing logs
by buckingham＠nym.hush.com 27 Nov '17

27 Nov '17

Hello forum members, Having spent a couple of weeks on this problem I thought someone might be able to help getting syslog-ng 3.9.1 to write log files on OpenBSD 6.2. The O/S was a fresh install, then syslog-ng and lastly BIND. No other packages except syslog-ng and BIND dependencies have been installed. No binaries or libs from BIND conflict or replace any from the syslog-ng installation. The .conf file was copied over from an older OpenBSD system, also running an older WORKING syslog-ng (3.1.1). The required changes to the .conf were made so that syslog-ng 3.9.1 with supervisor starts and remains running. By killing the syslog-ng process I was able to test that the supervisor process restarts syslog-ng. The rc.d script starts, stops and restarts syslog-ng as expected. /usr/local/sbin/syslog-ng -s returns no errors, indicating that the config file is sane. In /var/log the only thing it writes are syslog-ng start/stop messages. In an effort to localise the problem, a stub syslog-ng.conf was created with the following contents: @version: 3.9.1 source s_local { unix-dgram("/dev/log"); }; # destination catchall { file(/var/log/catchall); }; log { source(s_local); destination(catchall); }; The permissions and ownership on /var, /var/log, /var/run, and /etc are correct according to the OpenBSD /etc/mtree/special file. Syslog-ng starts retaining root permissions (default) and creates /dev/log if it does not exist. The syslog_ng_flags I use are: "-R /var/run/syslog_ng.persist -c /var/run/syslog_ng.ctl -p /var/run/syslog_ng.pid"Have tried running without any flags to see if that would help, but did not help. The output of syslog-ng -V is: syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: affile,afprog,afsocket,afsql,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,curl,date,dbparser,disk-buffer,geoip-plugin,graphite,kvformat,linux-kmsg-format,pseudofile,system-source,add-contextual-data,json-plugin,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off On my hunt I foud a message on NARKIVE where a user was having the same problem with syslog-ng 3.6 on OpenBSD 5.9, there was no solution. There was a reply saying: "With the changes in 5.6 using sendsyslog(2), only syslogd picks up local syslog. Search the openbsd-ports list for syslog-ng to see some comments on it.". After a further search I turned up nothing that would help. Have also tried to compile syslog-ng 3.12 from source to see if that might help solve the problem, but no success on OpenBSD. Many thanks in advance. CB

2 2

Follow-up: syslog-ng changes owner and permissions on a file source (syslog-ng Digest, Vol 151, Issue 20)
by Jim Segrave 24 Nov '17

24 Nov '17

We tried putting a file source with owner and permissions in our syslog-ng configuration: file("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log" flags(no-parse) program-override("custom-test") owner(-1) group(-1) perm(-1)); And at startup we get: Error parsing affile, syntax error, unexpected KW_OWNER, expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or KW_DNS_CACHE in /etc/syslog-ng/conf.d/remote_syslog.conf at line 4, column 123: file("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log" flags(no-parse) program-override("custom-test") owner(-1) group(-1) perm(-1)); ^^^^^ So the documentation is correct, the options are not valid on file sources. But it would be a good thing if they were -- Jim Segrave jes(a)j-e-s.net

1 0

Insider 2017-11: netdata; stats to graphite; logstash to syslog-ng;
by Czanik, Péter 23 Nov '17

23 Nov '17

Dear syslog-ng users, This is the 63rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Collecting netdata metrics -------------------------- netdata is a system for distributed real-time performance and health monitoring. You can use syslog-ng to collect and filter data provided by netdata and then send it to Elasticsearch for long-term storage and analysis. The aim is to send both metrics and logs to an Elasticsearch instance, and then access it via Kibana. https://www.balabit.com/blog/sending-netdata-metrics-syslog-ng-elasticsearc… syslog-ng statistics to Graphite -------------------------------- syslog-ng makes available various types of statistics. Data is available in a couple of forms: emitted regularly from the internal() source of syslog-ng or obtained using the syslog-ng-ctl utility from the command line. Due to the format that the internal source or the “stats” option of syslog-ng-ctl uses, it is not easy to send statistical data to Graphite or anywhere else. The syslog-ng-ctl utility provides a flexible “query” option, (available in recent versions of syslog-ng) which uses an easy-to-parse output format. Using the “jo” utility by @JPMens, you can convert the output to JSON, which can then be parsed by syslog-ng and forwarded to Graphite. https://www.balabit.com/blog/collecting-syslog-ng-statistics-to-graphite/ Sending logs from Logstash to syslog-ng ---------------------------------- Logstash adds a new syslog header to log messages before forwarding them to a syslog server. In the case of syslog messages, it is problematic as there will be two syslog headers in the message. Using syslog-ng for everything logging related in an Elasticsearch environment can considerably simplify your architecture. Still, there are situations, when Filebeats and Logstash are already deployed and you need some logs from Logstash in syslog-ng. Learn how you can remove the extra syslog header. https://www.balabit.com/blog/sending-logs-logstash-syslog-ng/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

syslog-ng changes owner and permissions on a file source
by Jim Segrave 22 Nov '17

22 Nov '17

We are running syslog-ng on a Centos-7 server: root@usenetmonitor-fe01.am4:[~]# /usr/sbin/syslog-ng -V syslog-ng 3 (3.12.1) Installer-Version: 3.12.1 Revision: Compile-Date: Nov 3 2017 15:15:50 Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: afuser,cef,linux-kmsg-format,sdjournal,confgen,map-value-pairs,json-plugin,affile,kvformat,tfgetent,date,afstomp,csvparser,add-contextual-data,cryptofuncs,system-source,syslogformat,afsocket,afprog,pseudofile,afamqp,snmptrapd-parser,dbparser,xml,stardate,tags-parser,graphite,basicfuncs,disk-buffer Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on When we tried to use syslog-ng to treat a file as a source, it changes the owner and group to root and the permissions to 0664, which is undesireable, as the file must remain writeable by the apache server. This is the source definition in syslog.conf when we were testing: source s_dev { file("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log" flags(no-parse) program-override("custom-test")); }; with syslog-ng stopped, here's the status of the file: root@usenetmonitor-fe01:[~]# ls -lt /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log -rw-r----- 1 apache apache 0 Nov 21 11:00 /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log As soon as syslog-ng is started it becomes root@usenetmonitor-fe01:[~]# ls -lt /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log -rw-r--r-- 1 root root 0 Nov 21 11:00 /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log Checking with strace, I find 16258 open("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 24 Followed shortly therafter by: 16258 fchown(24, 0, -1) = 0 16258 fchown(24, -1, 0) = 0 16258 fchmod(24, 0664) = 0 (and no, fd 24 hasn't been closed and re-opened) It's unclear to me why syslog-ng would have any reason to change ownership or permissions of file sources. As there are no options for file sources to set what it would change it to, it seems highly undesireable that this happens. The only workaround is to set global options to disable changing ownership or permissions: perm(-1); owner(-1); group(-1); and then specify those options for every destination file which is a maintenance burden Is there some other way to stop syslog-ng from latering files used as sources? -- Jim Segrave jes(a)j-e-s.net

2 1

syslog-ng events in log timestamping wrong
by Contreras, Gerald (DPS) 22 Nov '17

22 Nov '17

Hi All, I am very new to syslog-ng via Cygwin. I am having trouble with MWG proxy logs where individual events inside the log are being timestamped at the rotation time (timestamping at 00:00-00:01AM). This happens intermittently so the problem isn't constant. I have verified via packet captures that the timestamp is correct up until syslog-ng begins to handle the event and write it to the log. My syslog-ng.conf is as follows =============================================================== @version: 3.2 @include "scl.conf" options { create_dirs(yes); use_dns(yes); dns_cache_hosts(/etc/hosts); dns_cache_expire(3600); chain_hostnames(0); log_fifo_size(512); perm(0644); dir_perm(0755); time_reopen(10); time_reap(360); use_fqdn(no); keep_hostname(no); stats_freq(0); }; source s_network { udp(port("514")); }; destination s_files { file("/cygdrive/d/syslog/$HOST/$HOST-$MONTH-$DAY.log" create_dirs(yes)); }; log { source(s_network); destination(s_files); }; ============================================================== Thanks. Any help or direction would be much appreciated Important Notice: If you have received this email by mistake, please advise the sender and delete the message and attachments immediately. This email, including attachments, may contain confidential, sensitive, legally privileged and/or copyright information. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. Please consider the environment before printing this email.

3 3

Hourly Log rotate
by craig bowser 20 Nov '17

20 Nov '17

So according to https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng- ose-guide-admin/html/example-logrotate.html using this format: destination d_sorted { file("/var/log/remote/${HOST}/${YEAR}_${MONTH}_${DAY}.log" create-dirs(yes)); }; I can create logs folders and files based on the timestamp. And this is working nicely for us... With a caveat. We are getting a TON of logs, so we want to rotate hourly and archive quickly. My config is this: destination d_msg { file("/var/log/message_${YEAR} _${MONTH}_${DAY}_${HOUR}.log"); }; But instead of creating one file per hour according to the time the event is received, it is creating files based on the timestamp of the event (which, while useful for discovering and tracking down machines with time synch problems, is not so useful for managing log files on the syslog server). Is there a way to make it create files based on the time the event is received and NOT the timestamp of the event? Thanks.

2 2

Cisco ACS logs truncated
by Scot 15 Nov '17

15 Nov '17

Hi, Has anyone worked with ACS logs and solved the message header limit ? We can get syslog working but as expected the message gets truncated. Local logs on the ACS have the entire payload. Thinking there may be a way to script a log fetch or something. Thanks Scot

4 8

(no subject)
by craig bowser 15 Nov '17

15 Nov '17

I installed syslog-ng on an offline CentOS 6.x machine by first downloading all required rpms using this command: yumdownloader --resolve syslog-ng There were two files needed, that did not download and I downloaded them separately using: yumdownloader eventlog-0.2.13-1.el6.x86_64.rpm and: yumdownloader libnet-1.1.6-7.el6.x86_64.rpm These are the files I moved to the offline server: 17,684 eventlog-0.2.13-1.el6.i686.rpm 17,804 eventlog-0.2.13-1.el6.x86_64.rpm 122,940 gamin-0.1.10-9.el6.i686.rpm 1,738,964 glib2-2.28.8-9.el6.i686.rpm 4,570,040 glibc-2.12-1.209.el6_9.2.i686.rpm 117,400 libgcc-4.4.7-18.el6.i686.rpm 60,464 libnet-1.1.6-7.el6.i686.rpm 59,196 libnet-1.1.6-7.el6.x86_64.rpm 111,688 libselinux-2.0.94-7.el6.i686.rpm 309,920 libstdc++-4.4.7-18.el6.i686.rpm 161,168 nss-softokn-freebl-3.14.3-23.3.el6_8.i686.rpm 201,096 pcre-7.8-7.el6.i686.rpm 452,548 syslog-ng-3.2.5-4.el6.i686.rpm 451,092 syslog-ng-3.2.5-4.el6.x86_64.rpm 63,536 tcp_wrappers-libs-7.6-58.el6.i686.rpm 74,284 zlib-1.2.3-29.el6.i686.rpm Once I moved all these files to the offline server into their own folder, I ran: rpm –ivh *.rpm to install. I then checked the config file, made a couple of tweaks, then started syslog-ng using: service syslog-ng start I got the following errors: Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql' Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql' [ OK ] Syslog-ng was now running, just with these two errors. A quick google pointed to the problem, syslog-ng is looking for the sql plugin and the solution: yum install syslog-ng-libdbi But when I tried to run yumdownloader syslog-ng-libdbi I got "Nothing to download" And when I tried yum install --downloadonly syslog-ng-libdbi I got "Nothing to do" So here are my questions: 1. Will syslog-ng run just fine receiving udp and sending udp with those errors and without that module? If, so, I don’t care if I don’t get that module. 2. Is it possible to run syslog-ng and disable that module in the run command? 3. Can I remove syslog-ng and re-install disable that module on install? 4. I’d rather not recompile if I don’t have to, but I found instructions for how to do that. Any tips would be appreciated. Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

2 2

Install Issue
by Elphick, Matthew Thomas 08 Nov '17

08 Nov '17

Hello, I am trying to install syslog-ng OSE on RHEL 7 but I get the following error. Transaction check error: file /etc/logrotate.d/syslog from install of syslog-ng-3.12.1-2.el7.centos.x86_64 conflicts with file from package rsyslog-8.24.0-12.el7.x86_64 I can't remove rsyslog from the system as other people us it. What is the best way to resolve this issue? Thanks for your help. Matthew Elphick, Network Engineering Marsh & McLennan Companies Global Technology Infrastructure (MGTI) | Enterprise Networking Services Tower Place, London, EC3R 5BU O +44 (0)20 73573667 | M +44 (0)7392 122399 | matthew.elphick(a)mmc.com<mailto:matthew.elphick@mmc.com> ________________________________ Marsh Ltd. Registered in England and Wales Number: 1507274 Registered office 1 Tower Place West, Tower Place, London, EC3R 5BU. Marsh Ltd is authorised and regulated by the Financial Conduct Authority. This message and any attachments are confidential. If you have received this message in error please delete it from your system. If you require any assistance please notify the sender. Thank you.

4 6

logstash to syslog-ng
by Czanik, Péter 08 Nov '17

08 Nov '17

Hi, In the past few weeks I have seen questions about sending logs from Logstash to syslog-ng, so I tried it myself and put together a blog with simple configurations to help you: https://www.balabit.com/blog/sending-logs-logstash-syslog-ng/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 3