- syslog-ng - lists.balabit.hu

unidirectional unix-stream
by Fabien Wernli 11 May '17

11 May '17

Hi, I'm trying to use the unix-stream destination to send data to another application which opens the socket in line mode. The problem is that both ends complain about the connection breaking all the time: syslog-ng[31002]: Syslog connection broken; fd='24', server='AF_UNIX(/var/run/otherapp.sock)', time_reopen='60' otherapp[123445]: failed to read from socket #10: Connection reset by peer I tracked down the problem to the fact that otherapp sends some data after receiving a line from syslog-ng: "data ok", and syslog-ng doesn't seem to like that. If I set up a unidirectional proxy (that filters out upstream "data ok") between syslog-ng and otherapp everything works fine: socat -u UNIX-LISTEN:/tmp/proxy.sock,fork UNIX-CONNECT:/var/run/otherapp.sock Am I doing something wrong here, or is this a known limitation of syslog-ng?

3 9

is it possible to use if() ... else statements on syslog-ng 3.7?
by Jorge Pereira 11 May '17

11 May '17

hi team, is it possible to use if() ... else statements on syslog-ng 3.7? I am trying to do something like. if ($value >= X) ...... else ...... -- Jorge Pereira

6 11

CDATA in patterndb test_message
by John Kristoff 08 May '17

08 May '17

I wonder if it might be possible to be able to instruct the patternsdb parser to ignore markup for the test_message. This is a minor quibble, but I do have some messages that I parse that could use something like a <![CDATA[]]> wrapper around the test_message text. If I do a 'pdbtool test --validate' against a database that includes the offending test_message I get an error, but if I wrap it in a CDATA it seems to run without error, although the parser doesn't actually seem to see that wrapped test message. I can also verify the test message with 'pdbtool match -M' command, so this isn't a big deal, just thought I'd bring attention to it and see if maybe I'm missing something or if this might be something to handle in a future release. John

1 0

syslog-ng wont start - please help
by vijay amruth 06 May '17

06 May '17

Hello Peter/ team, I am stuck with issue since four days, spent hours and hours, I am unable find a cause, I compiled syslog-ng from source and resolved all the dependencies, when I get to start it, it wont start. here is the out put, please have a look at it. Any help is appreciated. OS version : CentOS Linux release 7.3.1611 (Core) [root@my-host ~]# systemctl start syslog-ng.service Job for syslog-ng.service failed because a timeout was exceeded. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. [root@my-host ~]# systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: activating (start) since Thu 2017-05-04 16:13:44 PDT; 29s ago Docs: man:syslog-ng(8) Main PID: 5089 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─5089 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid May 04 16:13:44 my-host systemd[1]: Starting System Logger Daemon... [root@my-host ~]# journalctl -xe May 04 16:10:01 my-host systemd[1]: Starting Session 348 of user root. -- Subject: Unit session-348.scope has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit session-348.scope has begun starting up. May 04 16:10:21 my-host nslcd[1289]: [18832f] <group/member="root"> failed to bind to LDAP server ldap://ldap.vip.slc.e May 04 16:10:21 my-host nslcd[1289]: [18832f] <group/member="root"> no available LDAP server found: Can't contact LDAP May 04 16:10:21 my-host nslcd[1289]: [18832f] <group/member="root"> no available LDAP server found: Server is unavailab May 04 16:10:21 my-host CROND[5035]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 04 16:10:23 my-host polkitd[676]: Registered Authentication Agent for unix-process:5039:17396299 (system bus name : May 04 16:10:23 my-host systemd[1]: Stopping Flush Journal to Persistent Storage... -- Subject: Unit systemd-journal-flush.service has begun shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-journal-flush.service has begun shutting down. May 04 16:10:23 my-host systemd-journal[2711]: Journal stopped -- Subject: The journal has been stopped -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The system journal process has shut down and closed all currently -- active journal files. May 04 16:10:23 my-host systemd-journal[5046]: Runtime journal is using 16.0M (max allowed 391.1M, trying to leave 586. May 04 16:10:23 my-host systemd-journal[5046]: Runtime journal is using 16.0M (max allowed 391.1M, trying to leave 586. May 04 16:10:23 my-host systemd-journald[2711]: Received SIGTERM from PID 1 (systemd). May 04 16:10:23 my-host systemd-journal[5046]: Journal started -- Subject: The journal has been started -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The system journal process has started up, opened the journal -- files for writing and is now ready to process requests. May 04 16:10:23 my-host polkitd[676]: Unregistered Authentication Agent for unix-process:5039:17396299 (system bus name May 04 16:10:23 my-host systemd[1]: Starting Flush Journal to Persistent Storage... -- Subject: Unit systemd-journal-flush.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-journal-flush.service has begun starting up. May 04 16:10:23 my-host systemd[1]: Started Flush Journal to Persistent Storage. -- Subject: Unit systemd-journal-flush.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-journal-flush.service has finished starting up. -- -- The start-up result is done. May 04 16:12:14 my-host polkitd[676]: Registered Authentication Agent for unix-process:5066:17407401 (system bus name : May 04 16:12:14 my-host systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. May 04 16:13:08 my-host nslcd[1289]: [528108] <group/member="postfix"> failed to bind to LDAP server ldap://ldap.vip.sl May 04 16:13:08 my-host nslcd[1289]: [528108] <group/member="postfix"> no available LDAP server found: Can't contact LD May 04 16:13:08 my-host nslcd[1289]: [528108] <group/member="postfix"> no available LDAP server found: Server is unavai May 04 16:13:44 my-host systemd[1]: syslog-ng.service start operation timed out. Terminating. May 04 16:13:44 my-host systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has failed. -- -- The result is failed. May 04 16:13:44 my-host systemd[1]: Unit syslog-ng.service entered failed state. May 04 16:13:44 my-host systemd[1]: syslog-ng.service failed. May 04 16:13:44 my-host polkitd[676]: Unregistered Authentication Agent for unix-process:5066:17407401 (system bus name May 04 16:13:44 my-host systemd[1]: syslog-ng.service holdoff time over, scheduling restart. May 04 16:13:44 my-host systemd[1]: Starting System Logger Daemon... -- Subject: Unit syslog-ng.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit syslog-ng.service has begun starting up. [root@my-host ~]# ps -ef | grep syslog-ng root 5089 1 0 16:13 ? 00:00:00 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid root 5103 32598 0 16:14 pts/1 00:00:00 grep --color=auto syslog-ng [root@my-host ~]# systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: activating (start) since Thu 2017-05-04 16:13:44 PDT; 1min 6s ago Docs: man:syslog-ng(8) Main PID: 5089 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─5089 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid May 04 16:13:44 my-host systemd[1]: Starting System Logger Daemon... [root@my-host ~]#cat /usr/lib/systemd/system/syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) [Service] Type=notify ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid ExecReload=/bin/kill -HUP $MAINPID EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng StandardOutput=journal StandardError=journal Restart=on-failure Environment=SYSTEMD_LOG_LEVEL=debug [Install] WantedBy=multi-user.target -- Thanks, Vijay Amrut.

2 5

PatternDB with CiscoSNMP Module
by Schoonover, Mark E HHHH 05 May '17

05 May '17

Thanks for reading, I'm in the process of creating a log to trap for Cisco devices. I have it working for facilities like this: TRINITY-2-TRINITY_SYSLOG_CRIT with this pattern: @ESTRING::%@@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@ Now there are other facilities with an additional hyphen before the severity: EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' which parses into: Testing message: program='EARL-DFC1-1-EXCESSIVE_PARITY_ERROR' message='Apr 26 14:04:45 hostname 565049: Apr 26 18:04:43.476 UTC: %EARL-DFC1-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM' Match name='.classifier.rule_id', value='09944c71-95eb-4bc0-8575-936931d85715', expected='09944c71-95eb-4bc0-8575-936931d85715' Wrong match name='.cisco.Facility', value='EARL', expected='EARL-DFC1' Wrong match name='.cisco.Severity', value='DFC1', expected='1' Wrong match name='.cisco.MsgName', value='1-EXCESSIVE_PARITY_ERROR', expected='EXCESSIVE_PARITY_ERROR' Match name='.cisco.MsgText', value='EARL 0: Parity error detected in VRAM', expected='EARL 0: Parity error detected in VRAM' I've tried a pattern that would parse EARL and DFC1 into separate variables, then append them into .cisco.Facility - which didn't work. Still that leaves the pattern hardcoded to just two hypens. I've also tried hard coding EARL-DFC1 into the pattern, then use value to set .cisco.Facility correctly. This approach works but being hardcoded, another Facility with embedded hyphens will fail. Ideally I'd like to have a pattern that matches on the last hyphen before the severity and capture that. I could parse using a regexp and setting $1, $2, $3, and $4 to the appropriate variables but I'm concerned about performance issues - I'm thinking this is a last resort solution. Any thoughts on how to proceed? Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================

4 5

unable to start syslog-ng.
by vijay amruth 01 May '17

01 May '17

Hello All, Hope you are all doing great. I am unable to start syslog-ng service Here is some information: *OS Version: Cent OS 7.3* *I compiled it from a tar ball..* [root@xxxxx system]# syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: kvformat,cef,disk-buffer,add-contextual-data,syslogformat,afsocket,affile,afprog,afuser,afamqp,afmongodb,csvparser,confgen,system-source,linux-kmsg-format,basicfuncs,cryptofuncs,dbparser,afstomp,pseudofile,graphite,sdjournal,date Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off *I manually added this file, this didn't come with install...* [root@xxxxx system]# cat syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) [Service] Type=notify Sockets=syslog.socket ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd-ng.pid --fd-limit 50000 ExecReload=/bin/kill -HUP $MAINPID StandardOutput=null Restart=on-failure [Install] WantedBy=multi-user.target Alias=syslog.service [root@sl-sz3-splunk01 system]# pwd /lib/systemd/system *Had parser errors:* [root@xxxxxx ~]# syslog-ng -s Error parsing config, Error compiling template (Unknown template function "format-json") in /usr/local/share/syslog-ng/include/scl/cim/template.conf at line 23, column 32: included from /usr/local/etc/scl.conf line 29, column 1 included from /usr/local/etc/syslog-ng.conf line 8, column 1 template-function "format-cim" "$(format-json --pair @timestamp='${R_ISODATE}' --pair @message='${MSG}' --key .cim.* --shift 5 --key _* --key .* --replace-prefix .=_ --key *.*)\n"; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng *Moved /usr/local/share/syslog-ng/include/scl/cim and no parser errors.* *But still unable to start.* [root@xxxxx system]# systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; disabled; vendor preset: enabled) Active: failed (Result: start-limit) since Mon 2017-05-01 10:46:58 PDT; 7s ago Docs: man:syslog-ng(8) Process: 2170 ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd-ng.pid --fd-limit 50000 (code=exited, status=203/EXEC) Main PID: 2170 (code=exited, status=203/EXEC) May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: syslog-ng.service: main process exited, code=exited, status=203/EXEC May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: Failed to start System Logger Daemon. May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: Unit syslog-ng.service entered failed state. May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: syslog-ng.service failed. May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart. May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: start request repeated too quickly for syslog-ng.service May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: Failed to start System Logger Daemon. May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: Unit syslog-ng.service entered failed state. May 01 10:46:58 sl-sz3-splunk01.slc.ebay.com systemd[1]: syslog-ng.service failed. What I am I missing? Any help is appreciated. Thank you. -- Thanks, Vijay Amrut.

3 3

syslog-ng 3.9.1 erroneously matching regex
by Soumyadip Das Mahapatra 01 May '17

01 May '17

Hi folks, I raised https://github.com/balabit/syslog-ng/issues/1455 Could you guys help me troubleshooting the issue. It looks to be a buggy regex match. Thanks, Soumyadip

2 1

bug in syslog-ng 3.9.1 triggered by bug in ivykis >= 0.40
by Lennert Buytenhek 29 Apr '17

29 Apr '17

Hello! (I subscribed to this mailing list to be able to post, but I have mailing list mail delivery disabled -- please CC me on replies.) A syslog-ng user tried to use syslog-ng with ivykis 0.41 and ran into the following issue: https://github.com/buytenh/ivykis/issues/11 In a nutshell, syslog-ng seems to be registering an ivykis timer (iv_timer) with timeout tv_sec = 0, tv_nsec = 0, and ivykis >= 0.40 added support for timerfd_create, but due to a bug in ivykis, this {0, 0} timeout would get passed into timerfd_settime() verbatim, and timerfd_settime() interprets this zero timeout as a request to disable the timerfd, which would cause timer processing to stop. I will release a new ivykis release with a workaround for this, but I figured that you'd probably want to know about this as well, as I don't think it makes sense for syslog-ng to be registering timers with an absolute timeout of zero. Cheers, Lennert

2 2

Syslog-ng and OSQUERY
by Dwijadas Dey 28 Apr '17

28 Apr '17

Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ? Thanks and regards

4 16

Relay Server Config Help
by wiskbroom＠hotmail.com 27 Apr '17

27 Apr '17

Hello; I am running syslog-NG on a server inside of a DMZ, and on that server I'd like to just forward all messages into my internal syslog-NG server. I feel this is better than having to create a new firewall rule for each new DMZ node. I have a simple syslog-NG config that looks like this: ---------START syslog-ng.conf----------- @version:3.5 @include "scl.conf" @include "/etc/syslog-ng/conf.d/*.conf" options { time-reap(30); mark-freq(10); keep-hostname(yes); chain_hostnames (off); flush_lines (0); time_reopen (10); log_fifo_size (1000); use_dns (yes); use_fqdn (no); create_dirs (no); }; source s_relay { udp(ip(0.0.0.0) port(514) so_rcvbuf(425984)); tcp(ip(0.0.0.0) port(514) max-connections(250) so_rcvbuf(425984) log_iw_size(25000) so_keepalive(yes) log_fetch_limit(100)); syslog(ip(0.0.0.0) transport("tcp") port(1514) max-connections(500) log_iw_size(25000) flags("threaded") log_fetch_limit(100)); }; destination d_syslog_tcp { syslog("10.5.5.10" transport("tcp") port(514)); }; log { source(s_relay); destination(d_syslog_tcp); }; ----------END syslog-NG.conf------------- The problem that I am experiencing is that my messages, once received by my internal syslog-NG server, look like the following: Apr 26 17:31:06 relay-sng-server 126 <30>1 2017-04-26T17:32:01-04:00 relay-client-host-X appserv - - - 606881792 140565409392384 INFO: @2 SessionExchange::ProcessTCPRead Apr 26 17:31:06 relay-sng-server 124 <30>1 2017-04-26T17:32:01-04:00 relay-client-host-X appserv - - - 606881792 140565409392384 DEBUG: OpenSSLHandler::PerformHandshake I am hoping not to muck around too much with my config for my internal syslog-NG servers, and want for the messages above to appear as coming from relay-client-host-X and NOT relay-sng-server. It could potentially be nice to know that the message was first received by relay-sng-server, but... I was considering opening another TCP port on my internal syslog servers, and using that to send from DMZ, then a rewrite, but that all seems far more complicated than necessary. Am I missing something on my syslog-NG conf? Any suggestions greatly appreciated! Thanks all in advance, -Vadim

2 2