December 2011 - syslog-ng - lists.balabit.hu

patterndb - user defined parsers
by Evan Rempel 19 Dec '11

19 Dec '11

It would be useful to permit users to define parsers in the patterndb. For example, in our environment, by policy we user a special set and order of characters of our administrators log into hosts and administer them. It would be useful to define a parser of @SYSADMIN@ that would match only our sysadmin accounts. We could then use this parser in the patterndb to take some action such as sending a message to the administrators about the event. Another example would be to create parser for @LOCALIP@ that matches my organizaions IP space. That way a set of rules can be defined using @LOCALIP@ for some kind of alerting, and then any organization could redifine the @LOCALIP@ and use all of the goodness that some third party had created for monitoring logs like an intrusion protection system. Current parsers can be described as QSTRING - match opening char - while not closing char, keep looking ESTRING - while not end string, keep looking NUMBER - while digit keep looking So it seems that general parsers could be constructed with two styles of matching, and then concatenating the together. 1. While in set of characters [some list of characters] 2. While not in set of characters [some list of characters] I would call these INSET to match 1 or more of a set of characters, unless a #-# were specified, then a minimum to a maximum would be required. OUTSET to match 1 or more of anything except the characters, unless a #-# were specified, then a minimum to a maximum would be required. (perhaps a count of + or * could be used to specify 1 or more and 0 or more respectively) and then limit the count of such occurrences so that you could build the @IPv4@ parser as @INSET::123456789*1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@ and @NUMBER@ would be @INSET::123456789:1@@INSET::0123456789@ @FLOAT@ would be @INSET::0123456789.@ Then a user could make <parser name="THOUSAND">@INSET::,:0-1@@INSET::0123456789:3@</parser> <parser name="MONEY">$@INSET::123456789:1-3@@THOUSAND:::*@.@INSET::0123456789:2@ This is kind of like inventing regular expressions :-( I'm not sure how well this fits into the radix tree matching structure, but I wanted to start this discussion. Given the MONEY example, I think it is obvious that there needs to be a way to specify repeating groups of "something" Let the discussion begin!

4 9

Re: [syslog-ng] Log only one host
by tokie＠tiscali.it 19 Dec '11

19 Dec '11

Hi, I wrong to write. filter is before to destination, but don't work!! Can you help me?? I would like log every host in one different file. Tks Tokie Message: 4 Date: Tue, 13 Dec 2011 20:47:21 +0100 From: Szalay Attila <sasa(a)balabit.hu> Subject: Re: [syslog-ng] Log only one host To: "tokie(a)tiscali.it" <tokie(a)tiscali.it>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1323805641.9268.6.camel(a)mochrul.balabit> Content-Type: text/plain; charset="UTF-8" Hi All! On Tue, 2011-12-13 at 15:39 +0100, tokie(a)tiscali.it wrote: > log { > source(udp); > > destination (d_fw); > filter(f_fw); > }; The log path is a sequence. If you first write the logs into the destination the filter will do nothing. You should change the destination and the filter statement. -- Attila Szalay Support (L3) Team Leader e-mail: attila.szalay(a)balabit.com phone: +36 1 398 6707 BalaBit IT Security www.balabit.com H-1117 Budapest, Aliz street 2. This Communication is Confidential. We only send and receive email on the basis of the term set out at http://www. balabit.com/disclaimer/. ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit. hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 80, Issue 19 ***************************************** E' nata indoona: chiama, videochiama e messaggia Gratis. Scarica indoona per iPhone, Android e PC: http://www.indoona.com/

1 0

Feature Request - patterndb match set
by Evan Rempel 18 Dec '11

18 Dec '11

I have come across some odd lines that really can't be matched/parsed by the patterndb 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number. I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair. Failing that I would propose that a @SET@ parser. @SET:name:character set@ This will match a sequence of characters that contain any of, and only those characters listed by "character set" This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled. Comments? Evan

5 6

[PATCH] [dbparser] SET parser
by Balint Kovacs 18 Dec '11

18 Dec '11

Hi, based on Evan's idea, please find a patch attached, that implements the SET parser that matches any number of characters specified as its parameters. Balint commit ad5da5958104a1dcf5e2492370cbb7037815c3ae Author: Balint Kovacs <blint(a)balabit.hu> Date: Sun Nov 27 18:40:28 2011 +0100 [dbparser] Implemented SET parser The SET parser matches any characters specified in its parameter list without further checks. This is a generic solution for the need of matching and arbitrary number of whitespaces in messages with padding for fields Reported-By: Evan Rempel <erempel(a)uvic.ca> Signed-Off-By: Balint Kovacs <blint(a)balabit.hu> diff --git a/modules/dbparser/radix.c b/modules/dbparser/radix.c index 6afa9a4..8c43709 100644 --- a/modules/dbparser/radix.c +++ b/modules/dbparser/radix.c @@ -116,6 +116,21 @@ r_parser_anystring(guint8 *str, gint *len, const gchar *param, gpointer state, R } gboolean +r_parser_set(guint8 *str, gint *len, const gchar *param, gpointer state, RParserMatch *match) +{ + *len = 0; + + while (param && strchr(param, str[*len])) + (*len)++; + + if (*len > 0) + { + return TRUE; + } + return FALSE; +} + +gboolean r_parser_ipv4(guint8 *str, gint *len, const gchar *param, gpointer state, RParserMatch *match) { gint dots = 0; @@ -379,6 +394,21 @@ r_new_pnode(guint8 *key) parser_node->parse = r_parser_anystring; parser_node->type = RPT_ANYSTRING; } + else if (strcmp(params[0], "SET") == 0) + { + if (params_len == 3) + { + parser_node->parse = r_parser_estring_c; + parser_node->type = RPT_ESTRING; + } + else + { + g_free(parser_node); + msg_error("Missing SET parser parameters", + evt_tag_str("type", params[0]), NULL); + parser_node = NULL; + } + } else if (g_str_has_prefix(params[0], "QSTRING")) { if (params_len == 3) diff --git a/modules/dbparser/radix.h b/modules/dbparser/radix.h index 9082419..9756dfe 100644 --- a/modules/dbparser/radix.h +++ b/modules/dbparser/radix.h @@ -40,7 +40,8 @@ enum RPT_ANYSTRING, RPT_IPV6, RPT_IP, - RPT_FLOAT + RPT_FLOAT, + RPT_SET }; typedef struct _RParserMatch @@ -124,6 +125,8 @@ r_parser_type_name(guint8 type) return "IP"; case RPT_FLOAT: return "FLOAT"; + case RPT_SET: + return "SET"; default: return "UNKNOWN"; }

2 2

[Bug 149] New: patterndb action message does not do variable substitution
by bugzilla＠bugzilla.balabit.com 17 Dec '11

17 Dec '11

https://bugzilla.balabit.com/show_bug.cgi?id=149 Summary: patterndb action message does not do variable substitution Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using a simple message action in a patterndb rule the message can not be populated with any variables. I am not using any correlation, I am just performing the message action every time I match the rule pattern. <rule id="463e34dd-f946-4f25-9cd2-9c00e25eba62" class="cluster" provider="UVic"> <patterns> <pattern>@ESTRING:: @@ESTRING:: @@NUMBER::@ @NUMBER@:@NUMBER@:@NUMBER@.@NUMBER@ @NUMBER@: VERBS RDMA library libibverbs.so failed to load: libibverbs.so: cannot open shared object file: No such file or directory.</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">$.classifier.rule_id</value> <value name="PROGRAM">patterndb</value> </values> </message> </action> </actions> </rule> results in the verbatim log line of 2011-11-28T10:12:21-08:00 int(a)catamount.comp.uvic.ca kern.emerg patterndb: $.classifier.rule_id The log line should be 2011-11-28T10:12:21-08:00 int(a)catamount.comp.uvic.ca kern.emerg patterndb: 463e34dd-f946-4f25-9cd2-9c00e25eba62 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 15

Syslog daemon restart fails on AIX 7.1 in central server
by Rakesh Rajasekharan 16 Dec '11

16 Dec '11

Hi, I am faced with a rather strange issue with syslog 3.0.7 version. The daemon does not start at all when I try to run it in the central server mode. I though do not get any error messages . But, I noticed that after a reboot of the machine. Everything works fine. Can someone please suggest of any possible areas to check. Thanks, Rakesh

2 2

web interface
by Bely Silva Junior 15 Dec '11

15 Dec '11

Hi, I see phpsyslog-ng was deprecated, another option what would be a choice for this ? logzilla is a good idea? -- Bely Silva Junior

4 5

Problem instert data MSSQL
by Bely Silva Junior 14 Dec '11

14 Dec '11

hi i have problem when i try insert data in mssql server, when i try run script, just 2 or 3 lines insert ok, but , insert stop . I use pipe, my confs. destination d_mssql { pipe("/var/mssql.syslog-ng.pipe" template( "INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg ) VALUES ( '$HOST','$FACILITY','$PRIORITY','$LEVEL','$TAG','$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC','$PROGRAM','$MSG' );\n" ) template-escape(yes)); template-escape(yes)); }; log { source(s_all); destination(d_mssql); }; my script: if [ -e /var/mssql.syslog-ng.pipe ]; then while [ -e /var/mssql.syslog-ng.pipe ] do #isql -v MSSQL "syslogng" "h0hYgbDB6Q" < /var/mssql.syslog-ng.pipe > /dev/null isql -v MSSQL "syslogng" "h0hYgbDB6Q" < /var/mssql.syslog-ng.pipe done else mkfifo /var/mssql.syslog-ng.pipe fi when i try run script: ./syslog2mssql.sh +---------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +---------------------------------------+ SQL> SQLRowCount returns 1 SQL> +------------+ | | +------------+ +------------+ SQLRowCount returns 0 SQL> SQLRowCount returns -1 SQL> SQLRowCount returns 1 SQL> [ISQL]ERROR: Could not SQLPrepare SQL> [ISQL]ERROR: Could not SQLPrepare./syslog2mssql.sh +---------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +---------------------------------------+ SQL> SQLRowCount returns 1 SQL> +------------+ | | +------------+ +------------+ SQLRowCount returns 0 SQL> SQLRowCount returns -1 SQL> SQLRowCount returns 1 SQL> [ISQL]ERROR: Could not SQLPrepare SQL> [ISQL]ERROR: Could not SQLPrepare someone have idea for this ? -- Bely Silva Junior

2 1

syslog-ng article in BSDmag, not just for BSD users
by Peter Czanik 14 Dec '11

14 Dec '11

Hello, Robert Fekete, our technical writer wrote a very nice article for BSDmag about syslog-ng with the title: "Extracting Useful Information From Log Messages". The following topics are addressed: • The message model behind syslog messages • The importance of metadata • How to process, recognize, and correlate log messages using syslog-ng If you want just this single article, you can download it from http://www.balabit.com/support/documentation/BSD_12_2011%20-%20BalaBit_sing… Other way I'd recommend to download the complete December issue of BSDmag from http://bsdmag.org/magazine/1781-rolling-your-own-kernel Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

GUI Web Interface for Syslog NG
by Kaushal Shriyan 14 Dec '11

14 Dec '11

Hi I happen to see that http://code.google.com/p/php-syslog-ng/ Php-syslog-ng is now known as LogZilla <http://www.logzilla.pro/>. Same owner, better code :-). But to my surprise,LogZilla is available for 30 days evaluation period. is there a free version available or a equivalent GUI Web Interface for Syslog NG Regards Kaushal

4 3