November 2011 - syslog-ng - lists.balabit.hu

Program stop/start bug in 3.3.x
by Martin Holste 22 Dec '11

22 Dec '11

I'm getting a weird bug with the program destination in which on shutdown it sends TERM and then sends a start: <snip> Sending destination program a TERM signal; cmdline='perl /usr/local/elsa/node/elsa.pl -c /etc/elsa_node.conf', child_pid='21421' Starting destination program; cmdline='perl /usr/local/elsa/node/elsa.pl -c /etc/elsa_node.conf' </snip> 3.2.4 is fine.

2 1

patterndb & not describing the entire log message
by Christophe Brocas 22 Dec '11

22 Dec '11

Hello As I said before in other message, I currently work with patterndb to write a pattern file describing Windows log messages related to file system. These messages are sent to Syslog-NG through Snare software. Example : Aug 12 15:08:48 10.0.0.1 MSWinEventLog 1 Security 541 ven. août 12 15:08:46 201 4656 Microsoft-Windows-Security-Auditing D00000001\accout1 N/A Success Audit hostname.d00000001.subdom1.dom1.fr Système de fichiers Un handle vers un objet a été demandé. Sujet : ID de sécurité : S-1-5-21-514083418-1667312537-733111063-1179 Nom du compte : account1 Domaine du compte : D00000001 ID d’ouverture de session : 0x8e90858 Objet : Serveur de l’objet : Security Type d’objet : File Nom de l’objet : E:\xxx\yyyy\filename.doc.doc ID du handle : 0x2b0 Informations sur le processus : ID du processus : 0xeb0 Nom du processus : C:\Windows\System32\inetsrv\w3wp.exe Informations sur la demande d’accès : ID de la transaction : {00000000-0000-0000-0000-000000000000} Accès : DELETE SYNCHRONIZE ReadAttributes Masque d’accès : 0x110080 Privilèges utilisés pour les vérifications d’accès : - Nombre de SID restreints : 0 540 Question : That kind of messages are extremely long and some areas seem empty in my example but may be not in the future. Is there a way with pattern to not describe the entire message but only the relevant part for us (the equivalent of wildcard in regexp) ? Thanks Christophe ***************************************************** "Le contenu de ce courriel et ses éventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

4 7

syslog-ng 3.3.1 quits at reload
by Michael Hocke 21 Dec '11

21 Dec '11

-----BEGIN PGP SIGNED MESSAGE----- Hi there, I am having a problem with syslog-ng 3.3.1. Once in a while syslog-ng quits with a QUIT signal after it was asked to reload the configuration through a HUP signal. I have two instances of it running in separate Solaris 10 zones. Both are extremely busy and both crashed once within the last two weeks. By looking at the code in lib/mainloop.c I found the place where syslog-ng quits. It is within main_loop_reload_config_apply caused by a corrupt old configuration. Here is the backtrace with dbx: - ------< SNIP >------- (dbx) where current thread: t@1 =>[1] _kill(0x347f, 0x3, 0x0, 0xfef557d2, 0x80ef41c, 0xfed6217c), at 0xfeb7af85 [2] main_loop_reload_config_apply(0x8047920, 0xfefad000, 0x8047908, 0xfef707f9, 0x80ef368, 0xfef92934, 0x0, 0x0, 0xfef706c9), at 0xfef70595 [3] main_loop_io_worker_sync_call(0x0, 0x8047920, 0x0, 0x0, 0x0, 0x0), at 0xfef70742 [4] iv_signal_event(0xfefad000, 0x8047960, 0x400, 0x796c746e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfef8d958 [5] iv_event_raw_got_event(0xfefad018, 0x8047db0, 0xfe9, 0xfefaa468, 0x2, 0x1), at 0xfef8d54a [6] iv_main(0xfefad180, 0xfef9291e, 0x804c2e8, 0x0, 0x804c7b0, 0x8047e1c), at 0xfef8ac82 [7] main_loop_run(0x8049cd7, 0x8047e44, 0x8047e48, 0x8047e2c, 0x8047e30, 0x8049390), at 0xfef71259 [8] main(0x1, 0x8047e60, 0x8047e70), at 0x80490fa - ------< SNIP >------- I used to run 3.0.6 before without a problem. The configuration of the server that crashed last night is pretty vanilla: - ------< SNIP >------- @version: 3.3 @include "scl.conf" options { time_reopen (10); chain_hostnames (no); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); perm (0644); }; # syslog-ng internal logging source s_local { internal (); }; # local syslogging only source s_sys { system (); }; destination d_sysmsg { file ("/dev/sysmsg"); }; destination d_local { file ("/var/log/syslog-ng.log"); }; destination d_messages { file ("/var/adm/messages"); }; destination d_mail { file ("/var/log/mail.log"); }; destination d_daemon { file ("/var/log/daemon.log"); }; destination d_auth { file ("/var/log/auth.log"); }; destination d_cron { file ("/var/log/cron.log"); }; destination d_user { file ("/var/log/user.log"); }; destination d_lpr { file ("/var/log/lpr.log"); }; destination d_news { file ("/var/log/news.log"); }; destination d_uucp { file ("/var/log/uucp.log"); }; destination d_local0 { file ("/var/log/ipfilter.log"); }; destination d_local1 { file ("/var/log/tcpwrapper.log"); }; destination d_local2 { file ("/var/log/tripwire.log"); }; destination d_local3 { file ("/var/log/local3.log"); }; destination d_local4 { file ("/var/log/local4.log"); }; destination d_local5 { file ("/var/log/local5.log"); }; destination d_local6 { file ("/var/log/local6.log"); }; destination d_local7 { file ("/var/log/local7.log"); }; destination d_snmptraps { file ("/usr/local/var/log/snmptrapd.${R_YEAR}${R_MONTH}${R_DAY}${R_HOUR}.log" group ("snmptrap") perm (416)); }; # this syslog config send a copy of every message to a Splunk server destination d_archive { udp ("x.x.x.x"); }; filter f_sysmsg { level (notice) and facility (kern); }; filter f_messages { facility (kern); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_cron { facility (cron); }; filter f_user { facility (user); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; log { source (s_local); destination (d_local); }; log { source (s_sys); filter (f_sysmsg); destination (d_sysmsg); destination (d_archive); }; log { source (s_sys); filter (f_messages); destination (d_messages); destination (d_archive); }; log { source (s_sys); filter (f_mail); destination (d_mail); destination (d_archive); }; log { source (s_sys); filter (f_daemon); destination (d_daemon); destination (d_archive); }; log { source (s_sys); filter (f_auth); destination (d_auth); destination (d_archive); }; log { source (s_sys); filter (f_cron); destination (d_cron); destination (d_archive); }; log { source (s_sys); filter (f_user); destination (d_snmptraps); }; log { source (s_sys); filter (f_lpr); destination (d_lpr); destination (d_archive); }; log { source (s_sys); filter (f_news); destination (d_news); destination (d_archive); }; log { source (s_sys); filter (f_uucp); destination (d_uucp); destination (d_archive); }; log { source (s_sys); filter (f_user); destination (d_snmptraps); }; log { source (s_sys); filter (f_lpr); destination (d_lpr); destination (d_archive); }; log { source (s_sys); filter (f_news); destination (d_news); destination (d_archive); }; log { source (s_sys); filter (f_uucp); destination (d_uucp); destination (d_archive); }; log { source (s_sys); filter (f_local0); destination (d_local0); destination (d_archive); }; log { source (s_sys); filter (f_local1); destination (d_local1); destination (d_archive); }; log { source (s_sys); filter (f_local2); destination (d_local2); destination (d_archive); }; log { source (s_sys); filter (f_local3); destination (d_local3); destination (d_archive); }; log { source (s_sys); filter (f_local4); destination (d_local4); destination (d_archive); }; log { source (s_sys); filter (f_local5); destination (d_local5); destination (d_archive); }; log { source (s_sys); filter (f_local6); destination (d_local6); destination (d_archive); }; log { source (s_sys); filter (f_local7); destination (d_local7); destination (d_archive); }; - ------< SNIP >------- Anybody else having this problem? Any hope we can nail this down? - - Michael -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.0.3 (Build 1) Charset: us-ascii wsBVAwUBTrmaT5bfnpCg64TVAQE3JAf+KCFVzUL/MGWTgMqRLowtYRDhT8NtThnL p4bRqDqggXjJR8eQhY8dRKyi7zSN+T59WnGunCA2WD98hX5+bUWtyR3yVs5GH4KK XplVrpoPzJNOQFnPrjnrg0VGahO83XMO0nIcS5HdgId8hsGgKKknkWosDTFl97J+ IPxQeqoVuj92DT9MSm00z6SLrX9XtD4dK762zcQ2Tsvz9j9KW21jiWo9sUor6Lpe Dh4Mle0V/dGJOBi7j7UiQOSQCRikl20zTZtId7wsFqsDVGSFlOKTvW+l3XLok0KU ufVnI46N8mrnUWFqDU5SQ952gelIsIAdj6UUwKioZQMafeZvvRLwFA== =9NxF -----END PGP SIGNATURE-----

3 7

patterndb - user defined parsers
by Evan Rempel 19 Dec '11

19 Dec '11

It would be useful to permit users to define parsers in the patterndb. For example, in our environment, by policy we user a special set and order of characters of our administrators log into hosts and administer them. It would be useful to define a parser of @SYSADMIN@ that would match only our sysadmin accounts. We could then use this parser in the patterndb to take some action such as sending a message to the administrators about the event. Another example would be to create parser for @LOCALIP@ that matches my organizaions IP space. That way a set of rules can be defined using @LOCALIP@ for some kind of alerting, and then any organization could redifine the @LOCALIP@ and use all of the goodness that some third party had created for monitoring logs like an intrusion protection system. Current parsers can be described as QSTRING - match opening char - while not closing char, keep looking ESTRING - while not end string, keep looking NUMBER - while digit keep looking So it seems that general parsers could be constructed with two styles of matching, and then concatenating the together. 1. While in set of characters [some list of characters] 2. While not in set of characters [some list of characters] I would call these INSET to match 1 or more of a set of characters, unless a #-# were specified, then a minimum to a maximum would be required. OUTSET to match 1 or more of anything except the characters, unless a #-# were specified, then a minimum to a maximum would be required. (perhaps a count of + or * could be used to specify 1 or more and 0 or more respectively) and then limit the count of such occurrences so that you could build the @IPv4@ parser as @INSET::123456789*1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@ and @NUMBER@ would be @INSET::123456789:1@@INSET::0123456789@ @FLOAT@ would be @INSET::0123456789.@ Then a user could make <parser name="THOUSAND">@INSET::,:0-1@@INSET::0123456789:3@</parser> <parser name="MONEY">$@INSET::123456789:1-3@@THOUSAND:::*@.@INSET::0123456789:2@ This is kind of like inventing regular expressions :-( I'm not sure how well this fits into the radix tree matching structure, but I wanted to start this discussion. Given the MONEY example, I think it is obvious that there needs to be a way to specify repeating groups of "something" Let the discussion begin!

4 9

Feature Request - patterndb match set
by Evan Rempel 18 Dec '11

18 Dec '11

I have come across some odd lines that really can't be matched/parsed by the patterndb 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number. I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair. Failing that I would propose that a @SET@ parser. @SET:name:character set@ This will match a sequence of characters that contain any of, and only those characters listed by "character set" This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled. Comments? Evan

5 6

[PATCH] [dbparser] SET parser
by Balint Kovacs 18 Dec '11

18 Dec '11

Hi, based on Evan's idea, please find a patch attached, that implements the SET parser that matches any number of characters specified as its parameters. Balint commit ad5da5958104a1dcf5e2492370cbb7037815c3ae Author: Balint Kovacs <blint(a)balabit.hu> Date: Sun Nov 27 18:40:28 2011 +0100 [dbparser] Implemented SET parser The SET parser matches any characters specified in its parameter list without further checks. This is a generic solution for the need of matching and arbitrary number of whitespaces in messages with padding for fields Reported-By: Evan Rempel <erempel(a)uvic.ca> Signed-Off-By: Balint Kovacs <blint(a)balabit.hu> diff --git a/modules/dbparser/radix.c b/modules/dbparser/radix.c index 6afa9a4..8c43709 100644 --- a/modules/dbparser/radix.c +++ b/modules/dbparser/radix.c @@ -116,6 +116,21 @@ r_parser_anystring(guint8 *str, gint *len, const gchar *param, gpointer state, R } gboolean +r_parser_set(guint8 *str, gint *len, const gchar *param, gpointer state, RParserMatch *match) +{ + *len = 0; + + while (param && strchr(param, str[*len])) + (*len)++; + + if (*len > 0) + { + return TRUE; + } + return FALSE; +} + +gboolean r_parser_ipv4(guint8 *str, gint *len, const gchar *param, gpointer state, RParserMatch *match) { gint dots = 0; @@ -379,6 +394,21 @@ r_new_pnode(guint8 *key) parser_node->parse = r_parser_anystring; parser_node->type = RPT_ANYSTRING; } + else if (strcmp(params[0], "SET") == 0) + { + if (params_len == 3) + { + parser_node->parse = r_parser_estring_c; + parser_node->type = RPT_ESTRING; + } + else + { + g_free(parser_node); + msg_error("Missing SET parser parameters", + evt_tag_str("type", params[0]), NULL); + parser_node = NULL; + } + } else if (g_str_has_prefix(params[0], "QSTRING")) { if (params_len == 3) diff --git a/modules/dbparser/radix.h b/modules/dbparser/radix.h index 9082419..9756dfe 100644 --- a/modules/dbparser/radix.h +++ b/modules/dbparser/radix.h @@ -40,7 +40,8 @@ enum RPT_ANYSTRING, RPT_IPV6, RPT_IP, - RPT_FLOAT + RPT_FLOAT, + RPT_SET }; typedef struct _RParserMatch @@ -124,6 +125,8 @@ r_parser_type_name(guint8 type) return "IP"; case RPT_FLOAT: return "FLOAT"; + case RPT_SET: + return "SET"; default: return "UNKNOWN"; }

2 2

[Bug 149] New: patterndb action message does not do variable substitution
by bugzilla＠bugzilla.balabit.com 17 Dec '11

17 Dec '11

https://bugzilla.balabit.com/show_bug.cgi?id=149 Summary: patterndb action message does not do variable substitution Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using a simple message action in a patterndb rule the message can not be populated with any variables. I am not using any correlation, I am just performing the message action every time I match the rule pattern. <rule id="463e34dd-f946-4f25-9cd2-9c00e25eba62" class="cluster" provider="UVic"> <patterns> <pattern>@ESTRING:: @@ESTRING:: @@NUMBER::@ @NUMBER@:@NUMBER@:@NUMBER@.@NUMBER@ @NUMBER@: VERBS RDMA library libibverbs.so failed to load: libibverbs.so: cannot open shared object file: No such file or directory.</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">$.classifier.rule_id</value> <value name="PROGRAM">patterndb</value> </values> </message> </action> </actions> </rule> results in the verbatim log line of 2011-11-28T10:12:21-08:00 int(a)catamount.comp.uvic.ca kern.emerg patterndb: $.classifier.rule_id The log line should be 2011-11-28T10:12:21-08:00 int(a)catamount.comp.uvic.ca kern.emerg patterndb: 463e34dd-f946-4f25-9cd2-9c00e25eba62 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 15

[Bug 146] New: pdbtool match does not display tags
by bugzilla＠bugzilla.balabit.com 09 Dec '11

09 Dec '11

https://bugzilla.balabit.com/show_bug.cgi?id=146 Summary: pdbtool match does not display tags Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: bug Estimated Hours: 0.0 When using the pdbtool match command the tags of a message are not displayed. % pdbtool match -M "Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350>" -p mmfs.xml --program=mmfs MESSAGE=Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> PROGRAM=mmfs .classifier.class=security .classifier.rule_id=463e34dd-f946-4f25-9cd2-9c00e25eba4c .net.destination.address=172.20.102.38 .net.destination.name=hermes0080 .mmfs.node=c0n350 application=mmfs -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Log to syslog file, filter from fifo
by Lay, James 09 Dec '11

09 Dec '11

Topic says it. I think I have a pretty simple layout: log { source(s_local); filter(f_firewall); destination(d_file); destination(fifo); }; Can I change this to: log { source(s_local); destination(d_file); filter(f_firewall); destination(fifo); }; ? Will this log to file, then filter, then go to fifo? Thanks and hope this isn't a silly question...I read this from the admin guide: Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags. So I'm hoping that is the case. Thank you. James

6 8

syslog-ng 3.3.3 rewrite question regarding cisco IOS Messages
by Thomas Wollner 02 Dec '11

02 Dec '11

Hello List, I try to rewrite cisco IOS syslog messages with timestamps in the MESSAGE field. I want to remove the timestamp from the message and set the program to the so called mnemonic of the message.. I found the following example on the list: ... filter f_rewrite_cisco_program { match('%([^:]: ([^\n]+)' value("MESSAGE") type("pcre") flags("store-matches" "nobackref")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program))); }; log { source(s_all); rewrite(r_cisco_program); destination(d_mydestination); }; But that does not work. I tried a lot of different rewrite syntaxes, none of them work for me. If I just employ the filter f_rewrite_cisco_program I` am able to filter out the cisco messages. Sample log line (written with template $R_ISODATE $HOST $MSG): 2011-11-30T18:23:50+01:00 192.168.1.1 217122: Nov 30 17:23:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down I`m using syslog-ng 3.3.3 debian package from madhouse. How I can rewrite my messages to filter the timestamp in the message field? any ideas? Any help is higly welcome, thanks in advance, Tom ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

4 18