February 2014 - syslog-ng - lists.balabit.hu

[Bug 211] New: dbparser pattern order breaks validation
by bugzilla＠bugzilla.balabit.com 29 May '14

29 May '14

https://bugzilla.balabit.com/show_bug.cgi?id=211 Summary: dbparser pattern order breaks validation Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 If two log lines share a common leading portion such as; xlog: backup pg_xlog/000000010000014700000076 xlog: backup pg_xlog/000000010000014700000076 failed then the order of the patterns in the pattern database xml file is important. For example, if the two patterns used are; xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed and the shorter pattern is first, it will match BOTH of the above lines because it matches the leading portion of both lines. If the patterns are entered with the longer pattern first, then the longer pattern will fail on the shorter message, allowing the match to fall through to the shorter pattern matcher, but will match the longer message. I expect the behavior of the dbparser to match with the pattern containing the MOST total static content that also matches all of the parsing macros. Some method where the order is not important need to be used. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 261] New: DOCBUG: --rekey
by bugzilla＠bugzilla.balabit.com 15 Apr '14

15 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=261 Summary: DOCBUG: --rekey Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 In Example 2.3. (Using the rekey() option) from §2.10.1. (Specifying data types in value-pairs) the documentation claims: "The rekey() option can be used with the format-json template-function as well, using the following syntax: $(format-json --key .cee.* --rekey --shift 4 --add-prefix events.)" However, when using this syntax, the config parser fails with the following error: "Error parsing config, Error compiling template (Error parsing value-pairs: expected an equal sign in key=value pair)" The correct syntax (which works, as suggested by algeron on IRC) would be the following: $(format-json --rekey .cee.* --add-prefix events. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Stats on destinations with macro's ?
by Scot 09 Apr '14

09 Apr '14

Is there a trick to get stats on destinations with macros ? I get stats on my FIFO, local, net work destinations but not on the destinations with macros. Also I don't see any difference in the stats output using any level level 1,2,3. Syslog-ng restarted after each conf change Sent from my iPad

6 11

[Bug 270] New: deadlock with patterndb and format-json ( debug NOT enabled in compilation)
by bugzilla＠bugzilla.balabit.com 12 Mar '14

12 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=270 Summary: deadlock with patterndb and format-json (debug NOT enabled in compilation) Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=90) --> (https://bugzilla.balabit.com/attachment.cgi?id=90) syslog-ng configuration and patterndb file This one was hard to isolate. Using the minimal attached configuration, it's very easy to lock the system down by running the following command (bourne shell): $ i=0; while echo 'sshd: Received disconnect from 1.2.3.4: 11: disconnected by user' > /var/tmp/syslo g-ng.src.fifo; do printf .$i.; i=$(($i+1)); done .0..1..2..3..4..5..6..7..8..9..10..11..12..13..14..15..16..17..18..19..20..21..22..23..24..25..26..27..28 [...] 124..125 [DEADLOCK] After that, syslog-ng is completely locked. command used to run syslog-ng: /sbin/syslog-ng -f /home/faxm0dem/syslog-ng-bug/syslog-ng.conf -R /tmp/syslog-ng.persist -p /tmp/syslog-ng.pid Most importantly, this does *only* happen when --enable-debug is *not* switched during compilation. Moreover, an example pattern in the provided patterndb shows how this can be avoided by renaming the variables. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

Syslog-ng & Distributed File Systems??
by Jiann-Ming Su 08 Mar '14

08 Mar '14

Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS?

7 20

syslog-ng OSE 3.5.1 is not accepting global options
by Tamayo, Andres 07 Mar '14

07 Mar '14

Hello, I am experiencing problems with syslog-ng OSE 3.5.1 reading the configuration file and accepting its global options. I had problems with 2 global options specified in my syslog-ng.conf file: @version: 3.5 @include "scl.conf" options { flush_lines(0); time_reopen(10); log_fifo_size(1000); use_dns(no); use_fqdn(yes); keep_hostname(no); keep_timestamp(yes); chain_hostnames(no); perm(0640); owner("root"); group("root"); create_dirs(yes); dir_perm(0750); dir_owner("root"); dir_group("root"); }; source s_cache { file("/path/to/some/file.txt", flags(no-parse)); }; destination d_network { syslog("192.168.1.10" port(1901) transport("tcp")); }; log { source(s_cache); destination(d_network); }; The time_reopen and log_fifo_size seem to be forced to zero if I try to specify alternate values for them as shown above. If I do not include them in the configuration file, syslog-ng uses their default values time_reopen=60 and log_fifo_size=10000. Does anybody know what may be causing this problem? The log_fifo_size default value is OK but I need a faster time_reopen window and I cannot find a way to specify it. If I do, syslog-ng forces it to time_reopen=0. I can tell because I get message like this when it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='0' If I remove the time_reopen(10) global option from the syslog-ng.conf file, then I get the following message when syslog-ng it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='60' Please let me know if you have any questions for me. Thank you for your help.

2 5

syslog-ng stats and performance
by Jesse Bowling 07 Mar '14

07 Mar '14

Hello, I'm running into an issue where we're fairly certain that we're dropping log messages somewhere along this path: device -> network -> VMware -> RHEL host -> syslog-ng What I'd like to understand better is what statistics I can gather from syslog-ng itself to help show or rule out drops in the software. I'm using the following general config: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off options { flush_lines (100); time_reopen (2); log_iw_size(100); log_fifo_size (65536); log_msg_size(8192); long_hostnames (off); use_dns(yes); use_fqdn(yes); keep_hostname (no); stats_freq(3600); stats_level(1); dns_cache(yes); keep_timestamp(no); }; When I looked at "syslog-ng-ctl stats" I see these "types" dropped processed stamp stored However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations. Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng? For the RHEL host portion I've tried watching netstat -su and netstat -st but the error counters for those do not seem to indicate that the level of issue we're seeing lies there. The processor for syslog-ng is busy, but averages 75%... Is it foolish to expect VMware to keep up with the level of logs we're taking in? Might virtualization be hiding drops from me? Any help appreciated... Cheers, Jesse -- Jesse Bowling

3 3

need help debugging some network received logs that aren't writing to files
by Chris Moody 06 Mar '14

06 Mar '14

Hello. First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it. I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious. I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name. I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch. I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device. Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool.. Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); }; # ===================== # UDP Packet Source source s_udp { udp(); }; # ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); }; # ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); }; # ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; ===== I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything. Any thoughts? Cheers, -Chris

8 15

[Bug 260] New: exit with code 134
by bugzilla＠bugzilla.balabit.com 06 Mar '14

06 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=260 Summary: exit with code 134 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 I just did a reload on version 3.4.4 (with latest patches) and triggered 2013-11-05T21:24:33-08:00 host daemon.crit supervise/syslog-ng[25895]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='134' Is it possible that the return code of syslog-ng is somehow linked to the return code of a child program destination? core file sent to Balazs in private e-mail -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 11

syslog-ng incubator packaged for openSUSE and Fedora
by Peter Czanik 06 Mar '14

06 Mar '14

Hi, syslog-ng incubator had a new release this week. Highlights are: ** Feature: LuaJIT support ** Feature: monitor source ** Feature: Templating support for the Lua destination ** Feature: Lua destinations have easy access to the timestamps ** Feature: The Lua modules now have a deinit hook ** Feature: The Lua modules has access to logging functions ** Feature: The Lua destination can have global variables set ** Feature: graphite output template function ** Feature: An ElasticSearch SCL ** Bugfix: RSS destination does not require newer glib For details check the NEWS file at https://github.com/balabit/syslog-ng-incubator/blob/master/NEWS I did some syslog-ng incubator packaging for openSUSE and Fedora. Functionality is not tested, as I don't have ElasticSearch, graphite, Riemann or lua scripts at hand (yet). As it's untested, there is no guarantee, that it does not eat your server :-) On the other hand testing and feedback is very welcome! openSUSE: https://build.opensuse.org/project/show/home:czanik:syslog-ng35 Fedora: http://copr.fedoraproject.org/coprs/czanik/syslog-ng35/ You should add the repository to your package manager, so all necessary dependencies are automagically resolved. Have a nice weekend! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

3 3