March 2014 - syslog-ng - lists.balabit.hu

Program() in destination driver not working for Macros defined in CSV-PARSER
by Justin B 10 Mar '14

10 Mar '14

Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale

3 7

Syslog-ng & Distributed File Systems??
by Jiann-Ming Su 08 Mar '14

08 Mar '14

Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS?

7 20

syslog-ng OSE 3.5.1 is not accepting global options
by Tamayo, Andres 07 Mar '14

07 Mar '14

Hello, I am experiencing problems with syslog-ng OSE 3.5.1 reading the configuration file and accepting its global options. I had problems with 2 global options specified in my syslog-ng.conf file: @version: 3.5 @include "scl.conf" options { flush_lines(0); time_reopen(10); log_fifo_size(1000); use_dns(no); use_fqdn(yes); keep_hostname(no); keep_timestamp(yes); chain_hostnames(no); perm(0640); owner("root"); group("root"); create_dirs(yes); dir_perm(0750); dir_owner("root"); dir_group("root"); }; source s_cache { file("/path/to/some/file.txt", flags(no-parse)); }; destination d_network { syslog("192.168.1.10" port(1901) transport("tcp")); }; log { source(s_cache); destination(d_network); }; The time_reopen and log_fifo_size seem to be forced to zero if I try to specify alternate values for them as shown above. If I do not include them in the configuration file, syslog-ng uses their default values time_reopen=60 and log_fifo_size=10000. Does anybody know what may be causing this problem? The log_fifo_size default value is OK but I need a faster time_reopen window and I cannot find a way to specify it. If I do, syslog-ng forces it to time_reopen=0. I can tell because I get message like this when it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='0' If I remove the time_reopen(10) global option from the syslog-ng.conf file, then I get the following message when syslog-ng it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='60' Please let me know if you have any questions for me. Thank you for your help.

2 5

[Bug 272] New: When syslog-ng server is not available, all syslog-ng clients are frozen
by bugzilla＠bugzilla.balabit.com 07 Mar '14

07 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=272 Summary: When syslog-ng server is not available, all syslog-ng clients are frozen Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: critical Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: carl.chenet-ext(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Os: Ubuntu Precise Syslog-ng version: 3.4.7-1~mhp1.sysvinit Hi, When the syslog-ng server is not available, all syslog-ng clients are frozen. We have to restart syslog-ng by mco because no other access work (ssh, direct console...). I'm including your syslog-ng configuration files. Feel free to ask me whenever information you need. Regards, Carl Chenet -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Problems creating syslog-ng.persit- file
by Tamayo, Andres 07 Mar '14

07 Mar '14

Hello, I am having problems starting up syslog-ng when I specify the location and name for its persist file. Here is the way I am trying to start it: syslog-ng --cfgfile=/etc/syslog-ng.conf --pidfile=/mnt/config/syslog-ng.pid --persist-file=/mnt/config/syslog-ng.persist When I use this command, syslog-ng starts and creates the /mnt/config/syslog-ng.persist file but it adds a '-' (dash) character to the end of the filename, like this: /mnt/config/syslog-ng.persist- Then it executes right to the point where it is about to parse the configuration file (I can see this in the debug logs) and stops. The command return value is 2. I have seen syslog-ng return 0 on success and 1 on failure but I do not know what is the meaning of 2 as the return value. I tried the same filename but a different location (/tmp) and it worked. The filename did not have the appended '-' and syslog-ng returned 0. The location I am trying to use is a flash memory mount that has JFFS2 file system. I wonder if this is part of the problem. I tried to change the name of the persist file to something different like: --persist-file=/mnt/config/something.else and the behavior is the same. The file is created like this: /mnt/config/something.else- and syslog-ng returns 2. Does anybody know why the '-' is appended to the filename or what is the meaning of a 2 return value? Thank you for your help.

3 4

$facility variable not working in filename - Solaris OS
by Feroz Basir 07 Mar '14

07 Mar '14

Hi all, Need some help regarding filename that use $facility variable. For some reason the facility variable was set to 'a' during file naming convention. I don't recall to have 'a' as a facility on remote server. What could be the issue here? This only happen on Solaris syslog-ng server. Others OS seem ok. Config sample: destination d_remoteall {file ("/syslog/$sourceip@$facility@$r_year:$r_month:$r_day:" create_dirs(yes) perm(0644));}; Sample output: 10.10.10.1@a@2014:03:05.gz I'm running syslog-ng version 3.0.2. Thanks. Regards, Feroz Basir

4 8

syslog-ng stats and performance
by Jesse Bowling 07 Mar '14

07 Mar '14

Hello, I'm running into an issue where we're fairly certain that we're dropping log messages somewhere along this path: device -> network -> VMware -> RHEL host -> syslog-ng What I'd like to understand better is what statistics I can gather from syslog-ng itself to help show or rule out drops in the software. I'm using the following general config: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off options { flush_lines (100); time_reopen (2); log_iw_size(100); log_fifo_size (65536); log_msg_size(8192); long_hostnames (off); use_dns(yes); use_fqdn(yes); keep_hostname (no); stats_freq(3600); stats_level(1); dns_cache(yes); keep_timestamp(no); }; When I looked at "syslog-ng-ctl stats" I see these "types" dropped processed stamp stored However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations. Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng? For the RHEL host portion I've tried watching netstat -su and netstat -st but the error counters for those do not seem to indicate that the level of issue we're seeing lies there. The processor for syslog-ng is busy, but averages 75%... Is it foolish to expect VMware to keep up with the level of logs we're taking in? Might virtualization be hiding drops from me? Any help appreciated... Cheers, Jesse -- Jesse Bowling

3 3

TLS-Syslog not working
by stefan.zahnd＠id.unibe.ch 07 Mar '14

07 Mar '14

Hi I hope someone can help me! The syslog-ng in our environment sends syslog messages using tls to our SIEM (Qradar). The following is the configuration of the syslog-ng (ip changed): # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; destination d_qradar_tls {tcp(„1.2.3.4" port(6514) tls( peer-verify(required-untrusted) ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) ); }; destination d_qradar_local { file("/tmp/qradar_local"); }; source s_testlog { file("/tmp/testlog" flags(no-parse)); }; log { source(s_testlog); destination(d_qradar_local); destination(d_qradar_tls); }; When I insert a message into the testlog it is parsed and written into the local destination „d_qradar_local“ but not sent to the remote destination. Syslog-NG in debugging mode (syslog-ng –Fevdt) shows the following: … Syslog connection established; fd='7', server=‚AF_INET(1.2.3.4:6514)', local='AF_INET(0.0.0.0:0)' Incoming log entry; line='test' Initializing destination file writer; template='/tmp/qradar_local', filename='/tmp/qradar_local' Destination timed out, reaping; template='/tmp/qradar_local', filename='/tmp/qradar_local' Closing log transport fd; fd=’15' Using tcpdump to check if some packets are sent to Qradar reveals that only the first insertion of a message into the testlog after a restart of syslog-ng leads to a packet sent to Qradar. Every other insertion has no effect on the remote destination but is always inserted into the local destination (file). Also during the start of syslog-ng two packets are sent to qradar. I’ve also opened a ticket at IBM and awating response. Thank you very much in advance for any help on this! Kind regards, Stefan -- University of Bern IT Services Department

2 1

need help debugging some network received logs that aren't writing to files
by Chris Moody 06 Mar '14

06 Mar '14

Hello. First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it. I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious. I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name. I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch. I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device. Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool.. Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); }; # ===================== # UDP Packet Source source s_udp { udp(); }; # ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); }; # ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); }; # ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; ===== I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything. Any thoughts? Cheers, -Chris

8 15

[Bug 260] New: exit with code 134
by bugzilla＠bugzilla.balabit.com 06 Mar '14

06 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=260 Summary: exit with code 134 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 I just did a reload on version 3.4.4 (with latest patches) and triggered 2013-11-05T21:24:33-08:00 host daemon.crit supervise/syslog-ng[25895]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='134' Is it possible that the return code of syslog-ng is somehow linked to the return code of a child program destination? core file sent to Balazs in private e-mail -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 11