- syslog-ng - lists.balabit.hu

Please help getting patterndb working.
by Nick 19 Dec '13

19 Dec '13

Hello, By following the documentation examples I think I have recreated a custom parser, but it doesn't work, please can you advise what I have missed? To create my patterndb I have used the example in link [1], to load it and output to a file I have kinda copied the example in link [2]. The issue I have is that my output file is mostly empty, it's like the variables are not being read properly. In case the files do not render properly , please see this gist -> https://gist.github.com/linickx/8002981 I have built a clean fedora 19 box which ships with syslog-ng-3.4.6-1. My /etc/syslog-ng/syslog-ng conf looks like: <PASTE> @version:3.2 # Default options copied from distribution options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); owner("nick"); group("nick"); perm(0640); }; source test_logfile { file("/home/nick/testfile.log"); }; parser test_pattern { db_parser( file("/home/nick/example.xml") ); }; destination test_output { file("/home/nick/output.log" template("${SSH_USERNAME}; ${SSH_CLIENT_ADDRESS}; \n") template_escape(no) ); }; log { source (test_logfile); parser(test_pattern); destination (test_output); }; </PASTE> The /home/nick/example.xml looks like <PASTE> <patterndb version='4' pub_date='2010-10-17'> <ruleset name='ssh' id='123456678'> <pattern>ssh</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern> </patterns> <examples> <example> <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message> <test_values> <test_value name="SSH.AUTH_METHOD">password</test_value> <test_value name="SSH_USERNAME">sampleuser</test_value> <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value> <test_value name="SSH_PORT_NUMBER">42156</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> </PASTE> For /home/nick/testfile.log I have used <PASTE> Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2 Accepted password for user from 10.51.0.27 port 4256 ssh2 </PASTE> after starting syslog-ng I get the following in /home/nick/output.log PASTE ; ; ; ; </PASTE> Any pointers on what I'm doing wrong would be very much appreciated. This is a cross-post from http://communities.balabit.com/balabit/topics/please_help_me_get_patterndb_…, apologies if this offends, the balabit.com website is not clear on where the most active community can be found. Thanks in Advance, Nick [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid…

3 6

quote characters (or other specials) in patterens?
by Nick 19 Dec '13

19 Dec '13

Hello, How should I handle quote characters (or other specials) in patterens? (To help with redering the examples below can be found here: https://gist.github.com/linickx/8038784) This works and will validate as expected (working.xml)... <patterndb version='4' pub_date='2013-12-17'> <ruleset name='bluecoat' id='dd001'> <pattern>bluecoat</pattern> <rules> <rule provider='linickx' id='nbdd001' class='system'> <patterns> <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @</pattern> </patterns> <examples> <example> <test_message program="bluecoat">10:57:56 43 10.8.26.200 - - - OBSERVED "Web Ads/Analytics" http://googleads.g.doubleclick.net/mads/ 200 TCP_CLIENT_REFRESH GET image/png http pagead2.googlesyndication.com 80 / pagead/images/nessie_icon_chevron_white.png - png "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; P76a(K3G5) Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30 (Mobile; afma-sdk-a-v6.2.1)" 10.8.24.5 724 1277 -</t est_message> <test_values> <test_value name="BC_HOUR">10</test_value> <test_value name="BC_MIN">57</test_value> <test_value name="BC_SEC">56</test_value> <test_value name="BC_TIME_TAKEN">43</test_value> <test_value name="BC_CLIENT_ADDRESS">10.8.26.200</test_value> <test_value name="BC_ACTION">OBSERVED</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> Simply by updating <pattern> with a quote for my next match (broken1.xml) ... <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ "</pattern> ... the whole thing bjorks (broken1_output.txt)... [nick@localhost ~]$ pdbtool test --validate nick.xml nick.xml validates Key contains '@' without escaping; key='@"', value='nbdd001' Testing message program='bluecoat' message='10:57:56 43 10.8.26.200 - - - OBSERVED "Web Ads/Analytics" http://googleads.g.doubleclick.net/mads/ 200 TCP_CLIENT_REFRESH GET image/png http pagead2.googlesyndication.com 80 /pagead/images/nessie_icon_chevron_white.png - png "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; P76a(K3G5) Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30 (Mobile; afma-sdk-a-v6.2.1)" 10.8.24.5 724 1277 -' Wrong match name='.classifier.rule_id', value='', expected='nbdd001' Wrong match name='BC_HOUR', value='', expected='10' Wrong match name='BC_MIN', value='', expected='57' Wrong match name='BC_SEC', value='', expected='56' Wrong match name='BC_TIME_TAKEN', value='', expected='43' Wrong match name='BC_CLIENT_ADDRESS', value='', expected='10.8.26.200' Wrong match name='BC_ACTION', value='', expected='OBSERVED' [nick@localhost ~]$ I have tried various escape methods... (escape_traditional.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ \"</pattern> or (escape_at.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ @"</pattern> I have tried various match methods... (test_estring_stopquote.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ "@ESTRING:BC_CATEGORY:"@</pattern> <test_value name="BC_CATEGORY">Web Ads/Analytics</test_value> or (test_estring_incquotes.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ @ESTRING:BC_CATEGORY: @</pattern> <test_value name="BC_CATEGORY">"Web Ads/Analytics"</test_value> or (test_qstring.xml) <pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@ @NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - - @ESTRING:BC_ACTION: @ @QSTRING:BC_CATEGORY:"@</pattern> <test_value name="BC_CATEGORY">Web Ads/Analytics</test_value> But no joy! (same error output as above) Any pointers would be appreciated! Testing carried out on: > Fedora release 19 (Schrödinger’s Cat) > syslog-ng-3.4.6-1.fc19.i686 Thanks in Advance, Nick

2 2

[Bug 258] New: syslog-ng 3.4.2 stalls on Ubuntu 12.04 Precise with /var/ log full
by bugzilla＠bugzilla.balabit.com 19 Dec '13

19 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=258 Summary: syslog-ng 3.4.2 stalls on Ubuntu 12.04 Precise with /var/log full Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: carl.chenet-ext(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Hi, We had several cases of frozen KVM virtual machines running Ubuntu 12.04 Precise with syslog-ng 3.4.2 taken from madhouse Debian repository. Issue: the servers are not reachable any more. You can initiate a ssh/console connection but the connection freezes before displaying the banner. Each time the /var/log lvm file system is full. The following errors occur in /var/log/system.log : Oct 30 07:05:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:06:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:07:18 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:08:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:09:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:10:18 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:11:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:12:18 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:13:19 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:14:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:15:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Oct 30 07:16:19 s-metcld-0002 syslog-ng[8467]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='', first-suppressed-msg='I/O error occurred while writing; fd=\'16\', error=\'No space left on device (28)\'' Oct 30 07:17:19 s-metcld-0002 syslog-ng[8467]: I/O error occurred while writing; fd='16', error='No space left on device (28)' Workaround: We have the mcollective client installed on each server so we were able to kill the syslog-ng process by mcollective. when syslog-ng is killed, the system stops freezing immedialty, the ssh connection is back. I'm not sure to understand how syslog-ng freezes the system but it is worth reporting I guess. Regards, Carl Chenet -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 4

Continue on with filter errors
by Lay, James 18 Dec '13

18 Dec '13

Question...is there a way to tell syslog-ng to try and continue on with configuration errors (filter)? Weird question maybe, but I was just wondering. Thank you. James

3 4

[RFC]: syslog-ng and UNIX credentials
by Gergely Nagy 14 Dec '13

14 Dec '13

Hi! We had a short chat with Bazsi earlier today, and he's working on a feature that will allow syslog-ng to pick out UNIX credentials passed through unix sockets (such as /dev/log). This means that we receive the PID, the UID and the GID of the sending program, and can opt to store it someplace. So far, syslog-ng was not doing that, but with the new feature, it becomes possible to store these. The idea at the moment is, is to have a flag for unix-* sources that enables collecting these credentials. If turned off (the default, unless using system(), which would turn it on for /dev/log), nothing changes. If turned on, it would replace the $PID sent over the socket with the one extracted from credentials. It would also add the "${.unix.GID}" and "${.unix.UID}" properties to the log message, along with "${.unix.EXE}" on platforms that support looking up the executable (Linux, for now). We'd like to invite the broader community to share your feelings about this feature, the naming of the properties and how You would like it to work, if you're interested in making use of this functionality. -- |8]

3 4

Output to file in /proc
by Valentijn Sessink 13 Dec '13

13 Dec '13

Hello list, As most of the functionality of "sshguard", "fail2ban" and others is in syslog-ng, I'm trying to use syslog-ng to add entries to my Linux kernel based firewall (iptables). I'm almost ready to implement one, but I have a question about undeliverable log lines: what happens if a certain file() destination blocks? Will syslog-ng retry to deliver the same message, or will it lose the message? What I'm doing is using the "ipt_recent" module; with this module, one can check if a certain IP address was seen before. For example, the following rule will block you if you were seen 5 or more times ("hitcount 5") in the "violators" list last minute ("seconds 60): iptables -A INPUT -m recent \ --rcheck --hitcount 5 --seconds 60 --name violators -j DROP The fun part is, that you can add IP adresses by stuffing them into a file in /proc: echo '+127.0.0.1' > /proc/net/xt_recent/violators ... will add 127.0.0.1 to the "violators" with the current time. However: echo 'whatever' > /proc/net/xt_recent/violators ... gives an error: Invalid argument (22) As far as I can see, syslog-ng will not try again to deliver the same message; but is this by design? I.e. can I trust syslog-ng to not "block" because of a single malformed IP address? Because then, an implementation of sshguard could be simply swapped with a simple: destination df_ipt_recent { file ("/proc/net/xt_recent/blocker" template("+${usracct.device}\n") ); }; filter f_ipt_recent { tags("secevt") and match("REJECT" value("secevt.verdict")); }; log { source (s_all); parser(pattern_db); filter(f_addtofirewall);destination(df_addtofirewall);}; ... which would be cool! Best regards, Valentijn

6 26

syslog-ng 3.5.1 - question about flags(final)...
by Johnson, Chris (HP TippingPoint Roseville) 10 Dec '13

10 Dec '13

Hello all, I'm in the process of upgrading from version 3.3.9 to 3.5.1 and have a question about how the 'flags(final);' is working in 3.5.1. In 3.3.9, I use the following structure of imbedded log statements: ################################################################################ # Service ipsec # filter f_ipsec_pgm{program("IPSEC-*" type("glob")) or program("IKE-*" type("glob")) or program("CHARON-*" type("glob")) or program("charon-*" type("glob")); }; filter f_ipsec_lvl_01{level(warning..emerg)}; filter f_ipsec_lvl_02{level(info..emerg)}; log { source(s_local); filter(f_ipsec_pgm); log { filter(f_ipsec_lvl_01); destination(d_logID_11); }; log { filter(f_ipsec_lvl_02); rewrite(r_quote_newlines); destination(d_logID_13); }; flags(final); }; In this case log messages of the 'correct' program would further be filtered on their severity level. * Info level messages would only be sent to 'd_logID_13' * Warning level messages (and above) would be sent to BOTH 'd_logID_11' and 'd_logID_13'. * Debug level messages would be discarded. Under 3.5.1, the 'filtering ' stops after it matches once: * Warning messages (and above) are only sent to 'd_logID_11' and NOT 'd_logID_13'. * Info messages are still (correctly) being sent only to 'd_logID_13'. If I remove (or comment out) the 'flags(final);' statement, messages are filtered correctly (i.e. the way I *want* them to be filtered :)) except that they also are being processed by all the following log statements and are being caught in my final filter of 'program("*" type("glob"))'. NOTE: yes, I know that I could use 'flags(fallback)' in my final filter, but that would still have every message processed by every log filter, and I would like to avoid that. So, what would be the correct way to set up my log statement to re-create the 3.3 behavior? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

3 7

[review request] 3.6queue/f/stats-reset
by Balazs Scheidler 10 Dec '13

10 Dec '13

Hi, I'm about to integrate a stats-counter related change and prior to doing that I'd like to solicit feedback. There are both user-visible changes and developer visible changes, feedback on both is sought. Code related comments are preferred on github, functionality comments are best sent via email in response to this one. Here's the branch: https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset Thanks in advance. User-visible changes: ===================== syslog-ng is keeping track on a number of statistics counters that can be used to track what is happening. The detail level of these counters is controlled by the stats-level() global option, which defaults to zero. syslog-ng reports these counters periodically every stats-freq() seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on the command line. At stats-level(3) a number of "dynamic" counters are enabled, which track the number of messages on a per $HOST, per $PROGRAM and per $HOST_FROM basis. This means that you can follow how many messages a given $PROGRAM is generating in a period. The issue with dynamic counters is that they can consume a lot of memory if the rate of new $PRORGRAM/$HOST values is high. Previously the cleanup happened in a limited way at configuration reload (e.g. SIGHUP) time, which is not really useful for controlling memory usage by this feature. The current patch series introduces a stats-lifetime() option and syslog-ng starts inspecting counters every once in a while and prune those that are updated earlier than stats-lifetime(). Not-so visible changes ====================== I have fixed a few bugs and renamed a function to make it easier to understand. Details are in the git history. Balazs Scheidler (7): logsource: use stats_instant_inc_dynamic_counter() for HOST counters logsource: track time for PROGRAM counters mainloop: remove old orphanage based stats cleanup stats: add stats_prune_old_counters() stats-timer: implement stats_lifetime option cfg: add stats-lifetime() global option stats: rename stats_instance_inc_dynamic_counter() to stats_register_and_increment_dynamic_counter() lib/cfg-grammar.y | 16 ++++++++------- lib/cfg-parser.c | 1 + lib/cfg.c | 3 ++- lib/cfg.h | 1 + lib/logsource.c | 15 +++----------- lib/mainloop.c | 1 - lib/stats-timer.c | 61 +++++++++++++++++++++++++++++++++++++++---------------- lib/stats-timer.h | 2 +- lib/stats.c | 52 +++++++++++++++++++++++++++++++++++++++-------- lib/stats.h | 7 +++---- 10 files changed, 108 insertions(+), 51 deletions(-) -- Bazsi

6 12

[review request] please check
by Balazs Scheidler 10 Dec '13

10 Dec '13

Hi, As a number of things depend on the logtransport.c split, I'd like to merge that separately. https://github.com/balabit/syslog-ng/pull/36 The branch itself is really just shuffling around files, basically the monolitic logtransport.c that contained 4 classes were split into separate files under the lib/transport subdirectory. Thanks in advance. Bazsi

2 1

[git] 3.6queue prefix in branches changed to 3.6
by Balazs Scheidler 07 Dec '13

07 Dec '13

Hi, For consistency, I've renamed all "3.6queue" prefixes in branch names to 3.6. This makes them possible to push all "pending" branches with a single push call and is consistent with the branches aimed at different major releases. I might remove the prefix altogether in the future (e.g. using no prefix for development branches destined at the latest master), but as I'm working a number of things in parallel for 3.6, I prefer to keep them as one bunch for now. -- Bazsi

1 0