- syslog-ng - lists.balabit.hu

Invalid frame header messages
by Mr. Jan Walter 21 Dec '11

21 Dec '11

Hi everyone, I compiled version 3.3.3 and tried getting a client and server to talk based on the documentation on the balabit site, specifically the documentation for OSE version 3.3. The server config file: -- source remotelogs { syslog( port(2072) transport(tcp) ); } ; destination r_log { file("/var/log/remotelogs/all"); }; log{ source(remotelogs); destination(r_log); } ; -- And the client sending: -- # remote logging test log # @version: 3.3 source s_kern { file("/proc/kmsg"); }; source s_local { unix-stream("/dev/log"); internal(); }; destination d_file {file("/var/log/messages"); }; destination d_secure { file("/var/log/secure");}; # set the filters filter f_messages { not facility(auth, authpriv);}; filter f_authpriv { facility(auth, authpriv); }; destination loghost { tcp(10.195.105.214 port(2072) ); }; log { source(s_kern) ; source(s_local) ; filter(f_messages) ; destination(loghost) ; } ; -- There are some other log directives there for local files, I'll post them if people want to see them So when this runs, I get on the server: Dec 7 11:18:05 xxxxxxxxxx syslog-ng[4188]: Syslog connection closed; fd='21', client='AF_INET(10.203.54.9:54837)', local='AF_INET(0.0.0.0:2072)' And the client: Dec 7 11:19:04 xxxxxxxxxxxx syslog-ng[1735]: Syslog connection established; fd='20', server='AF_INET(10.195.105.214:2072)', local='AF_INET(0.0.0.0:0)' I read the email thread from 2009, and tried the vanilla configs - any ideas? Thanks, Jan

3 4

syslog-ng 3.3.3 pipe source does not parse host
by Evan Rempel 21 Dec '11

21 Dec '11

Try as I might, I can not get a pipe source to maintain the hostname. Syslog-ng includes the hostname as the $MSGHDR (or the $PROGRAM). my config # ----------------------------------- source patterndb { pipe("/var/log/syslog.pipes/patterndb" log_fetch_limit(500) log_iw_size(100000) ); }; source int { internal(); }; template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); }; destination d_var_patterndb { file("/var/log/patterndb.$R_YEAR$R_MONTH$R_DAY.000000" owner("root") group("syslogs") perm(0640) template(t_standardfile)); }; log { source(patterndb); destination(d_var_patterndb); }; # ----------------------------------- data I write tothe pipe # ----------------------------------- <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- The output file # ----------------------------------- 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- Am I missing something silly with the 3.3 config syntax (Up until now I have only used up to 3.0) Evan.

2 1

[Bug 153] New: default year when no year is present should be current year
by bugzilla＠bugzilla.balabit.com 21 Dec '11

21 Dec '11

https://bugzilla.balabit.com/show_bug.cgi?id=153 Summary: default year when no year is present should be current year Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When parsing a source that has a month and day but no year, the S_YEAR macro does not default to R_YEAR. It seems to be defaulting to R_YEAR+1 What is the intention when there is no year in the source? To test, use historicatl syslog or rsyslog to create a syslog file, then use that file as a source (I was doing a "cat filename > /smoe/pipe" and using a pipe source) and write a file that has an ISODate in it. The year is 2012. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng 3.3.3 and multiple incarnations of the same parser
by Michael Hocke 21 Dec '11

21 Dec '11

-----BEGIN PGP SIGNED MESSAGE----- Hi guys, I ran into another problem with 3.3.3. It seems that I can use a cvs-parser only in one and only one log path. If I reuse it in multiple log paths the columns of the cvs parser are empty. I downgraded to 3.0.9 and that resolved my problem. Here is an example config: - --------< SNIP >---------- source s_remote { udp () }; parser p_hostname_site { csv-parser (columns ("REMOTEHOST.SITE", "REMOTEHOST.NAME") delimiters ("-") template ("${HOST}")); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; destination d_cacl { file ("/usr/local/var/log/current/${REMOTEHOST.SITE:-other}/cisco-acl.${S_YEAR}${S_MONTH}${S_DAY}${S_HOUR}.log" template ("$S_DATE ${REMOTEHOST.NAME:-none} $MSGHDR$MSG\n") }; destination d_caps { file ("/usr/local/var/log/current/${REMOTEHOST.SITE:-other}/cisco-aps.${S_YEAR}${S_MONTH}${S_DAY}${S_HOUR}.log" template ("$S_DATE ${REMOTEHOST.NAME:-none} $MSGHDR$MSG\n") }; log (source (s_remote); filter (f_local0); parser (p_hostname_site); destination (d_cacl); }; log (source (s_remote); filter (f_local1); parser (p_hostname_site); destination (d_caps); }; - --------< SNIP >---------- The incoming logs for local0 would look like this: Dec 2 16:00:00 florence-VPN-FLORENCE %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 florence-VPN-FLORENCE %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 Under 3.3.3 it would produce the following logs: /usr/local/var/log/current/other/cisco-acl.2011120216.log: Dec 2 16:00:00 none %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 none %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 The same configuration with 3.0.9 produces the following: /usr/local/var/log/current/florence/cisco-acl.2011120216.log: Dec 2 16:00:00 VPN-FLORENCE %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 VPN-FLORENCE %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 If I remove the parser from one of the log paths 3.3.3 would spit out the same output as 3.0.9 but of course, the second log path would need its own parser to make it work. I am unable to check syslog-ng 3.1 or 3.2. - - Michael -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.0.3 (Build 1) Charset: us-ascii wsBVAwUBTtjvwJbfnpCg64TVAQFmvwf9EZ3wShhBxIXOGXWiSJguF07HCgw99Wxu amUrvAAeI9nZSj4wvpQCZNLrpPJ2Q7/DCSKTunzopHNJVwMQQvFSwzaAcezNNt4Y ogmTAx+TeyMc4rYuknpG9Khz6WXFTM8rH4imjNziDSxSwkqZrCQDnmYM4B2pxfqo BNZbcRKifbo9mts0FbeD9hpBiYNxt+k9pXPtqUD193ihUWcnNF4NG3hgt/1RXB7E 7v79A6swf30PXSLpdJn8fATptKVrfzzvXlW0ncNolSUyKK7XaY4kgvjsnFba++CS eTil+xkINNnuUz2FDYfEDziTYFeezO/lhSrhE8BiDaRt1Gx10XfbDQ== =2Idy -----END PGP SIGNATURE-----

4 5

[PATCH] afprog: Terminate the child process on deinit.
by Gergely Nagy 21 Dec '11

21 Dec '11

When reloading the configuration, terminate the child process of a program destination, for the following reasons: it will be restarted anyway by afprog_dd_reopen(), so there's no harm done. But if dash is used as /bin/sh, simply closing the stdin of the process will not make it stop. That, in turn, results in us starting the same program over and over again on each SIGHUP. So instead of doing this, send a SIGTERM to the child on deinit, too. Reported-by: Thomas Wollner <tw(a)wollner-net.de> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afprog/afprog.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/modules/afprog/afprog.c b/modules/afprog/afprog.c index 6ce74b0..264e85b 100644 --- a/modules/afprog/afprog.c +++ b/modules/afprog/afprog.c @@ -328,6 +328,16 @@ afprogram_dd_deinit(LogPipe *s) { AFProgramDestDriver *self = (AFProgramDestDriver *) s; + if (self->pid != -1) + { + msg_verbose("Sending destination program a TERM signal", + evt_tag_str("cmdline", self->cmdline->str), + evt_tag_int("child_pid", self->pid), + NULL); + kill(self->pid, SIGTERM); + self->pid = -1; + } + if (self->writer) log_pipe_deinit(self->writer); -- 1.7.7.3

2 1

patterndb - user defined parsers
by Evan Rempel 19 Dec '11

19 Dec '11

It would be useful to permit users to define parsers in the patterndb. For example, in our environment, by policy we user a special set and order of characters of our administrators log into hosts and administer them. It would be useful to define a parser of @SYSADMIN@ that would match only our sysadmin accounts. We could then use this parser in the patterndb to take some action such as sending a message to the administrators about the event. Another example would be to create parser for @LOCALIP@ that matches my organizaions IP space. That way a set of rules can be defined using @LOCALIP@ for some kind of alerting, and then any organization could redifine the @LOCALIP@ and use all of the goodness that some third party had created for monitoring logs like an intrusion protection system. Current parsers can be described as QSTRING - match opening char - while not closing char, keep looking ESTRING - while not end string, keep looking NUMBER - while digit keep looking So it seems that general parsers could be constructed with two styles of matching, and then concatenating the together. 1. While in set of characters [some list of characters] 2. While not in set of characters [some list of characters] I would call these INSET to match 1 or more of a set of characters, unless a #-# were specified, then a minimum to a maximum would be required. OUTSET to match 1 or more of anything except the characters, unless a #-# were specified, then a minimum to a maximum would be required. (perhaps a count of + or * could be used to specify 1 or more and 0 or more respectively) and then limit the count of such occurrences so that you could build the @IPv4@ parser as @INSET::123456789*1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@ and @NUMBER@ would be @INSET::123456789:1@@INSET::0123456789@ @FLOAT@ would be @INSET::0123456789.@ Then a user could make <parser name="THOUSAND">@INSET::,:0-1@@INSET::0123456789:3@</parser> <parser name="MONEY">$@INSET::123456789:1-3@@THOUSAND:::*@.@INSET::0123456789:2@ This is kind of like inventing regular expressions :-( I'm not sure how well this fits into the radix tree matching structure, but I wanted to start this discussion. Given the MONEY example, I think it is obvious that there needs to be a way to specify repeating groups of "something" Let the discussion begin!

4 9

Re: [syslog-ng] Log only one host
by tokie＠tiscali.it 19 Dec '11

19 Dec '11

Hi, I wrong to write. filter is before to destination, but don't work!! Can you help me?? I would like log every host in one different file. Tks Tokie Message: 4 Date: Tue, 13 Dec 2011 20:47:21 +0100 From: Szalay Attila <sasa(a)balabit.hu> Subject: Re: [syslog-ng] Log only one host To: "tokie(a)tiscali.it" <tokie(a)tiscali.it>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1323805641.9268.6.camel(a)mochrul.balabit> Content-Type: text/plain; charset="UTF-8" Hi All! On Tue, 2011-12-13 at 15:39 +0100, tokie(a)tiscali.it wrote: > log { > source(udp); > > destination (d_fw); > filter(f_fw); > }; The log path is a sequence. If you first write the logs into the destination the filter will do nothing. You should change the destination and the filter statement. -- Attila Szalay Support (L3) Team Leader e-mail: attila.szalay(a)balabit.com phone: +36 1 398 6707 BalaBit IT Security www.balabit.com H-1117 Budapest, Aliz street 2. This Communication is Confidential. We only send and receive email on the basis of the term set out at http://www. balabit.com/disclaimer/. ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit. hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 80, Issue 19 ***************************************** E' nata indoona: chiama, videochiama e messaggia Gratis. Scarica indoona per iPhone, Android e PC: http://www.indoona.com/

1 0

Feature Request - patterndb match set
by Evan Rempel 18 Dec '11

18 Dec '11

I have come across some odd lines that really can't be matched/parsed by the patterndb 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number. I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair. Failing that I would propose that a @SET@ parser. @SET:name:character set@ This will match a sequence of characters that contain any of, and only those characters listed by "character set" This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled. Comments? Evan

5 6

[PATCH] [dbparser] SET parser
by Balint Kovacs 18 Dec '11

18 Dec '11

Hi, based on Evan's idea, please find a patch attached, that implements the SET parser that matches any number of characters specified as its parameters. Balint commit ad5da5958104a1dcf5e2492370cbb7037815c3ae Author: Balint Kovacs <blint(a)balabit.hu> Date: Sun Nov 27 18:40:28 2011 +0100 [dbparser] Implemented SET parser The SET parser matches any characters specified in its parameter list without further checks. This is a generic solution for the need of matching and arbitrary number of whitespaces in messages with padding for fields Reported-By: Evan Rempel <erempel(a)uvic.ca> Signed-Off-By: Balint Kovacs <blint(a)balabit.hu> diff --git a/modules/dbparser/radix.c b/modules/dbparser/radix.c index 6afa9a4..8c43709 100644 --- a/modules/dbparser/radix.c +++ b/modules/dbparser/radix.c @@ -116,6 +116,21 @@ r_parser_anystring(guint8 *str, gint *len, const gchar *param, gpointer state, R } gboolean +r_parser_set(guint8 *str, gint *len, const gchar *param, gpointer state, RParserMatch *match) +{ + *len = 0; + + while (param && strchr(param, str[*len])) + (*len)++; + + if (*len > 0) + { + return TRUE; + } + return FALSE; +} + +gboolean r_parser_ipv4(guint8 *str, gint *len, const gchar *param, gpointer state, RParserMatch *match) { gint dots = 0; @@ -379,6 +394,21 @@ r_new_pnode(guint8 *key) parser_node->parse = r_parser_anystring; parser_node->type = RPT_ANYSTRING; } + else if (strcmp(params[0], "SET") == 0) + { + if (params_len == 3) + { + parser_node->parse = r_parser_estring_c; + parser_node->type = RPT_ESTRING; + } + else + { + g_free(parser_node); + msg_error("Missing SET parser parameters", + evt_tag_str("type", params[0]), NULL); + parser_node = NULL; + } + } else if (g_str_has_prefix(params[0], "QSTRING")) { if (params_len == 3) diff --git a/modules/dbparser/radix.h b/modules/dbparser/radix.h index 9082419..9756dfe 100644 --- a/modules/dbparser/radix.h +++ b/modules/dbparser/radix.h @@ -40,7 +40,8 @@ enum RPT_ANYSTRING, RPT_IPV6, RPT_IP, - RPT_FLOAT + RPT_FLOAT, + RPT_SET }; typedef struct _RParserMatch @@ -124,6 +125,8 @@ r_parser_type_name(guint8 type) return "IP"; case RPT_FLOAT: return "FLOAT"; + case RPT_SET: + return "SET"; default: return "UNKNOWN"; }

2 2

[Bug 149] New: patterndb action message does not do variable substitution
by bugzilla＠bugzilla.balabit.com 17 Dec '11

17 Dec '11

https://bugzilla.balabit.com/show_bug.cgi?id=149 Summary: patterndb action message does not do variable substitution Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using a simple message action in a patterndb rule the message can not be populated with any variables. I am not using any correlation, I am just performing the message action every time I match the rule pattern. <rule id="463e34dd-f946-4f25-9cd2-9c00e25eba62" class="cluster" provider="UVic"> <patterns> <pattern>@ESTRING:: @@ESTRING:: @@NUMBER::@ @NUMBER@:@NUMBER@:@NUMBER@.@NUMBER@ @NUMBER@: VERBS RDMA library libibverbs.so failed to load: libibverbs.so: cannot open shared object file: No such file or directory.</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">$.classifier.rule_id</value> <value name="PROGRAM">patterndb</value> </values> </message> </action> </actions> </rule> results in the verbatim log line of 2011-11-28T10:12:21-08:00 int(a)catamount.comp.uvic.ca kern.emerg patterndb: $.classifier.rule_id The log line should be 2011-11-28T10:12:21-08:00 int(a)catamount.comp.uvic.ca kern.emerg patterndb: 463e34dd-f946-4f25-9cd2-9c00e25eba62 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 15