December 2013 - syslog-ng - lists.balabit.hu

[Bug 265] New: syslog-ng does not output the $PID of the process, even when forced
by bugzilla＠bugzilla.balabit.com 23 Dec '13

23 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=265 Summary: syslog-ng does not output the $PID of the process, even when forced Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: support(a)sikkerhed.org Type of the Report: --- Estimated Hours: 0.0 I'm researching syslog-ng, in order to set it up on all my machines, and have them log to a central server on the network. Things are going great, except I can't seem to force syslog-ng to report the $PID of the program that logs, so sometimes I get lines that have missing $PIDS: (anonymized, but based on real log line) 2013-12-05T19:40:44.388099+01:00 web15.example.org pop3d: Connection, ip=[::ffff:x.y.z.w] Whereas I was able to change pureftpd to log with pid: 2013-12-05T19:40:44.639127+01:00 web15.example.org pure-ftpd[872804]: (?(a)x.y.z.w) [INFO] New connection from x.y.z.w Unfortunately, courier-imap does not have such a logging option. I thought that using $PID in the output template would force syslog-ng to log the PID of the reporting process, but this does not seem to be the case. Is it really true that there is no way to force syslog-ng to report the PID? I know that rsyslog has this feature, but I would much rather use syslog-ng :) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 6

[Bug 186] New: resettable stats
by bugzilla＠bugzilla.balabit.com 21 Dec '13

21 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=186 Summary: resettable stats Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: szaki(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 According to https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017457.html It would be a good idea to be able to reset the statistic, when needed, without full restart, or any magic like altering the config a bit and reload. So I suggest the enable_stats_reset(yes) default option, for sources and destinations (everywhere are statcounters defined), and the syslog-ng-ctl reset-stats command - to reset the statistic where the enable_stats_reset(yes) option allows this. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 262] New: patterndb: Computed <values> are not expanded in context-id
by bugzilla＠bugzilla.balabit.com 20 Dec '13

20 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=262 Summary: patterndb: Computed <values> are not expanded in context-id Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=87) --> (https://bugzilla.balabit.com/attachment.cgi?id=87) patterndb ruleset It seems the values are not being expanded as expected in the context-id: -- pdbtool test /tmp/test.pdb -vd Module loaded and initialized successfully; module='syslogformat' Module loaded and initialized successfully; module='basicfuncs' Testing message program='' message='the big "brown" fox' patterndb rule matches; rule_id='A' Advancing patterndb current time because of an incoming message; utc='-1' Correllation context lookup failure, starting a new context; rule='A', context='brown.', context_timeout='0', context_expiration='-1' Match name='.classifier.rule_id', value='A', expected='A' Match name='matched.var', value='brown', expected='brown' -- as you can see, context-id='brown.' instead of 'brown.brown' as expected by expanding '${matched.var}.${computed.var}' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 263] New: patterndb: missing "program" attribute in test_message silently ignored
by bugzilla＠bugzilla.balabit.com 20 Dec '13

20 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=263 Summary: patterndb: missing "program" attribute in test_message silently ignored Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Although the attached pdb validates, the test messages are not run, because "program" attribute is missing. There should be a warning, or something: -- pdbtool --validate test /tmp/test2.pdb -dv Module loaded and initialized successfully; module='syslogformat' Module loaded and initialized successfully; module='basicfuncs' /tmp/test2.pdb validates -- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 266] New: localport option does not bind to a specific port
by bugzilla＠bugzilla.balabit.com 20 Dec '13

20 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=266 Summary: localport option does not bind to a specific port Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zebaz(a)xs4all.nl Type of the Report: bug Estimated Hours: 0.0 The localport option to specify a source port when opening a TCP connection no longer seems to be working. For testing I used the following simplified configurations: @version: 3.5 source src { unix-stream("/dev/log" max-connections(256)); internal(); file("/proc/kmsg"); }; destination logcentral { tcp("logcentral" port(514) localport(1014)); }; log { source(src); destination(logcentral); }; When running syslog-ng -d the relevant output is: Syslog connection established; fd='8', server='AF_INET(<ip removed>:514)', local='AF_INET(0.0.0.0:0)' I tried some variations in the configuration: - source port > 1024 gives the same result - specifying the localip option binds the local socket explicitly to that IP address but still not to a specific port Behaviour has been checked to occur with versions 3.4.2 and 3.4.6 from Gentoo as well as 3.5.2 from the upstream site (compiled from source). Other installations with identical kernel versions run older syslog-ng installations (3.2.x and 3.3.x) with the same settings without a problem. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 11

Please help getting patterndb working.
by Nick 19 Dec '13

19 Dec '13

Hello, By following the documentation examples I think I have recreated a custom parser, but it doesn't work, please can you advise what I have missed? To create my patterndb I have used the example in link [1], to load it and output to a file I have kinda copied the example in link [2]. The issue I have is that my output file is mostly empty, it's like the variables are not being read properly. In case the files do not render properly , please see this gist -> https://gist.github.com/linickx/8002981 I have built a clean fedora 19 box which ships with syslog-ng-3.4.6-1. My /etc/syslog-ng/syslog-ng conf looks like: <PASTE> @version:3.2 # Default options copied from distribution options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); owner("nick"); group("nick"); perm(0640); }; source test_logfile { file("/home/nick/testfile.log"); }; parser test_pattern { db_parser( file("/home/nick/example.xml") ); }; destination test_output { file("/home/nick/output.log" template("${SSH_USERNAME}; ${SSH_CLIENT_ADDRESS}; \n") template_escape(no) ); }; log { source (test_logfile); parser(test_pattern); destination (test_output); }; </PASTE> The /home/nick/example.xml looks like <PASTE> <patterndb version='4' pub_date='2010-10-17'> <ruleset name='ssh' id='123456678'> <pattern>ssh</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern> </patterns> <examples> <example> <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message> <test_values> <test_value name="SSH.AUTH_METHOD">password</test_value> <test_value name="SSH_USERNAME">sampleuser</test_value> <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value> <test_value name="SSH_PORT_NUMBER">42156</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> </PASTE> For /home/nick/testfile.log I have used <PASTE> Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2 Accepted password for user from 10.51.0.27 port 4256 ssh2 </PASTE> after starting syslog-ng I get the following in /home/nick/output.log PASTE ; ; ; ; </PASTE> Any pointers on what I'm doing wrong would be very much appreciated. This is a cross-post from http://communities.balabit.com/balabit/topics/please_help_me_get_patterndb_…, apologies if this offends, the balabit.com website is not clear on where the most active community can be found. Thanks in Advance, Nick [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid…

3 6

quote characters (or other specials) in patterens?
by Nick 19 Dec '13

19 Dec '13

Hello, How should I handle quote characters (or other specials) in patterens? (To help with redering the examples below can be found here: https://gist.github.com/linickx/8038784) This works and will validate as expected (working.xml)... <patterndb version='4' pub_date='2013-12-17'> <ruleset name='bluecoat' id='dd001'> <pattern>bluecoat</pattern> <rules> <rule provider='linickx' id='nbdd001' class='system'> <patterns> <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @</pattern> </patterns> <examples> <example> <test_message program="bluecoat">10:57:56 43 10.8.26.200 - - - OBSERVED "Web Ads/Analytics" http://googleads.g.doubleclick.net/mads/ 200 TCP_CLIENT_REFRESH GET image/png http pagead2.googlesyndication.com 80 / pagead/images/nessie_icon_chevron_white.png - png "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; P76a(K3G5) Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30 (Mobile; afma-sdk-a-v6.2.1)" 10.8.24.5 724 1277 -</t est_message> <test_values> <test_value name="BC_HOUR">10</test_value> <test_value name="BC_MIN">57</test_value> <test_value name="BC_SEC">56</test_value> <test_value name="BC_TIME_TAKEN">43</test_value> <test_value name="BC_CLIENT_ADDRESS">10.8.26.200</test_value> <test_value name="BC_ACTION">OBSERVED</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> Simply by updating <pattern> with a quote for my next match (broken1.xml) ... <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ "</pattern> ... the whole thing bjorks (broken1_output.txt)... [nick@localhost ~]$ pdbtool test --validate nick.xml nick.xml validates Key contains '@' without escaping; key='@"', value='nbdd001' Testing message program='bluecoat' message='10:57:56 43 10.8.26.200 - - - OBSERVED "Web Ads/Analytics" http://googleads.g.doubleclick.net/mads/ 200 TCP_CLIENT_REFRESH GET image/png http pagead2.googlesyndication.com 80 /pagead/images/nessie_icon_chevron_white.png - png "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; P76a(K3G5) Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30 (Mobile; afma-sdk-a-v6.2.1)" 10.8.24.5 724 1277 -' Wrong match name='.classifier.rule_id', value='', expected='nbdd001' Wrong match name='BC_HOUR', value='', expected='10' Wrong match name='BC_MIN', value='', expected='57' Wrong match name='BC_SEC', value='', expected='56' Wrong match name='BC_TIME_TAKEN', value='', expected='43' Wrong match name='BC_CLIENT_ADDRESS', value='', expected='10.8.26.200' Wrong match name='BC_ACTION', value='', expected='OBSERVED' [nick@localhost ~]$ I have tried various escape methods... (escape_traditional.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ \"</pattern> or (escape_at.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ @"</pattern> I have tried various match methods... (test_estring_stopquote.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ "@ESTRING:BC_CATEGORY:"@</pattern> <test_value name="BC_CATEGORY">Web Ads/Analytics</test_value> or (test_estring_incquotes.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ @ESTRING:BC_CATEGORY: @</pattern> <test_value name="BC_CATEGORY">"Web Ads/Analytics"</test_value> or (test_qstring.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ @QSTRING:BC_CATEGORY:"@</pattern> <test_value name="BC_CATEGORY">Web Ads/Analytics</test_value> But no joy! (same error output as above) Any pointers would be appreciated! Testing carried out on: > Fedora release 19 (Schrödinger’s Cat) > syslog-ng-3.4.6-1.fc19.i686 Thanks in Advance, Nick

2 2

[Bug 258] New: syslog-ng 3.4.2 stalls on Ubuntu 12.04 Precise with /var/ log full
by bugzilla＠bugzilla.balabit.com 19 Dec '13

19 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=258 Summary: syslog-ng 3.4.2 stalls on Ubuntu 12.04 Precise with /var/log full Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: carl.chenet-ext(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Hi, We had several cases of frozen KVM virtual machines running Ubuntu 12.04 Precise with syslog-ng 3.4.2 taken from madhouse Debian repository. Issue: the servers are not reachable any more. You can initiate a ssh/console connection but the connection freezes before displaying the banner. Each time the /var/log lvm file system is full. The following errors occur in /var/log/system.log : Oct 30 07:05:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:06:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:07:18 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:08:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:09:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:10:18 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:11:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:12:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:13:19 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:14:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:15:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:16:19 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:17:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Workaround: We have the mcollective client installed on each server so we were able to kill the syslog-ng process by mcollective. when syslog-ng is killed, the system stops freezing immedialty, the ssh connection is back. I'm not sure to understand how syslog-ng freezes the system but it is worth reporting I guess. Regards, Carl Chenet -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 4

Continue on with filter errors
by Lay, James 18 Dec '13

18 Dec '13

Question...is there a way to tell syslog-ng to try and continue on with configuration errors (filter)? Weird question maybe, but I was just wondering. Thank you. James

3 4

[RFC]: syslog-ng and UNIX credentials
by Gergely Nagy 14 Dec '13

14 Dec '13

Hi! We had a short chat with Bazsi earlier today, and he's working on a feature that will allow syslog-ng to pick out UNIX credentials passed through unix sockets (such as /dev/log). This means that we receive the PID, the UID and the GID of the sending program, and can opt to store it someplace. So far, syslog-ng was not doing that, but with the new feature, it becomes possible to store these. The idea at the moment is, is to have a flag for unix-* sources that enables collecting these credentials. If turned off (the default, unless using system(), which would turn it on for /dev/log), nothing changes. If turned on, it would replace the $PID sent over the socket with the one extracted from credentials. It would also add the "${.unix.GID}" and "${.unix.UID}" properties to the log message, along with "${.unix.EXE}" on platforms that support looking up the executable (Linux, for now). We'd like to invite the broader community to share your feelings about this feature, the naming of the properties and how You would like it to work, if you're interested in making use of this functionality. -- |8]

3 4