- syslog-ng - lists.balabit.hu

Can I put all the parsed fields into a destination template?
by VÁMOS Balázs 21 Dec '11

21 Dec '11

Hi, I would like to forward the parsed fields to LOGalyze in an unstructured way. Is it possible to create a serialized object (JSON or something like that) after parsing message with patterndb? -- VÁMOS Balázs LOGalyze

4 4

syslog-ng 3.3.1 quits at reload
by Michael Hocke 21 Dec '11

21 Dec '11

-----BEGIN PGP SIGNED MESSAGE----- Hi there, I am having a problem with syslog-ng 3.3.1. Once in a while syslog-ng quits with a QUIT signal after it was asked to reload the configuration through a HUP signal. I have two instances of it running in separate Solaris 10 zones. Both are extremely busy and both crashed once within the last two weeks. By looking at the code in lib/mainloop.c I found the place where syslog-ng quits. It is within main_loop_reload_config_apply caused by a corrupt old configuration. Here is the backtrace with dbx: - ------< SNIP >------- (dbx) where current thread: t@1 =>[1] _kill(0x347f, 0x3, 0x0, 0xfef557d2, 0x80ef41c, 0xfed6217c), at 0xfeb7af85 [2] main_loop_reload_config_apply(0x8047920, 0xfefad000, 0x8047908, 0xfef707f9, 0x80ef368, 0xfef92934, 0x0, 0x0, 0xfef706c9), at 0xfef70595 [3] main_loop_io_worker_sync_call(0x0, 0x8047920, 0x0, 0x0, 0x0, 0x0), at 0xfef70742 [4] iv_signal_event(0xfefad000, 0x8047960, 0x400, 0x796c746e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfef8d958 [5] iv_event_raw_got_event(0xfefad018, 0x8047db0, 0xfe9, 0xfefaa468, 0x2, 0x1), at 0xfef8d54a [6] iv_main(0xfefad180, 0xfef9291e, 0x804c2e8, 0x0, 0x804c7b0, 0x8047e1c), at 0xfef8ac82 [7] main_loop_run(0x8049cd7, 0x8047e44, 0x8047e48, 0x8047e2c, 0x8047e30, 0x8049390), at 0xfef71259 [8] main(0x1, 0x8047e60, 0x8047e70), at 0x80490fa - ------< SNIP >------- I used to run 3.0.6 before without a problem. The configuration of the server that crashed last night is pretty vanilla: - ------< SNIP >------- @version: 3.3 @include "scl.conf" options { time_reopen (10); chain_hostnames (no); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); perm (0644); }; # syslog-ng internal logging source s_local { internal (); }; # local syslogging only source s_sys { system (); }; destination d_sysmsg { file ("/dev/sysmsg"); }; destination d_local { file ("/var/log/syslog-ng.log"); }; destination d_messages { file ("/var/adm/messages"); }; destination d_mail { file ("/var/log/mail.log"); }; destination d_daemon { file ("/var/log/daemon.log"); }; destination d_auth { file ("/var/log/auth.log"); }; destination d_cron { file ("/var/log/cron.log"); }; destination d_user { file ("/var/log/user.log"); }; destination d_lpr { file ("/var/log/lpr.log"); }; destination d_news { file ("/var/log/news.log"); }; destination d_uucp { file ("/var/log/uucp.log"); }; destination d_local0 { file ("/var/log/ipfilter.log"); }; destination d_local1 { file ("/var/log/tcpwrapper.log"); }; destination d_local2 { file ("/var/log/tripwire.log"); }; destination d_local3 { file ("/var/log/local3.log"); }; destination d_local4 { file ("/var/log/local4.log"); }; destination d_local5 { file ("/var/log/local5.log"); }; destination d_local6 { file ("/var/log/local6.log"); }; destination d_local7 { file ("/var/log/local7.log"); }; destination d_snmptraps { file ("/usr/local/var/log/snmptrapd.${R_YEAR}${R_MONTH}${R_DAY}${R_HOUR}.log" group ("snmptrap") perm (416)); }; # this syslog config send a copy of every message to a Splunk server destination d_archive { udp ("x.x.x.x"); }; filter f_sysmsg { level (notice) and facility (kern); }; filter f_messages { facility (kern); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_cron { facility (cron); }; filter f_user { facility (user); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; log { source (s_local); destination (d_local); }; log { source (s_sys); filter (f_sysmsg); destination (d_sysmsg); destination (d_archive); }; log { source (s_sys); filter (f_messages); destination (d_messages); destination (d_archive); }; log { source (s_sys); filter (f_mail); destination (d_mail); destination (d_archive); }; log { source (s_sys); filter (f_daemon); destination (d_daemon); destination (d_archive); }; log { source (s_sys); filter (f_auth); destination (d_auth); destination (d_archive); }; log { source (s_sys); filter (f_cron); destination (d_cron); destination (d_archive); }; log { source (s_sys); filter (f_user); destination (d_snmptraps); }; log { source (s_sys); filter (f_lpr); destination (d_lpr); destination (d_archive); }; log { source (s_sys); filter (f_news); destination (d_news); destination (d_archive); }; log { source (s_sys); filter (f_uucp); destination (d_uucp); destination (d_archive); }; log { source (s_sys); filter (f_user); destination (d_snmptraps); }; log { source (s_sys); filter (f_lpr); destination (d_lpr); destination (d_archive); }; log { source (s_sys); filter (f_news); destination (d_news); destination (d_archive); }; log { source (s_sys); filter (f_uucp); destination (d_uucp); destination (d_archive); }; log { source (s_sys); filter (f_local0); destination (d_local0); destination (d_archive); }; log { source (s_sys); filter (f_local1); destination (d_local1); destination (d_archive); }; log { source (s_sys); filter (f_local2); destination (d_local2); destination (d_archive); }; log { source (s_sys); filter (f_local3); destination (d_local3); destination (d_archive); }; log { source (s_sys); filter (f_local4); destination (d_local4); destination (d_archive); }; log { source (s_sys); filter (f_local5); destination (d_local5); destination (d_archive); }; log { source (s_sys); filter (f_local6); destination (d_local6); destination (d_archive); }; log { source (s_sys); filter (f_local7); destination (d_local7); destination (d_archive); }; - ------< SNIP >------- Anybody else having this problem? Any hope we can nail this down? - - Michael -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.0.3 (Build 1) Charset: us-ascii wsBVAwUBTrmaT5bfnpCg64TVAQE3JAf+KCFVzUL/MGWTgMqRLowtYRDhT8NtThnL p4bRqDqggXjJR8eQhY8dRKyi7zSN+T59WnGunCA2WD98hX5+bUWtyR3yVs5GH4KK XplVrpoPzJNOQFnPrjnrg0VGahO83XMO0nIcS5HdgId8hsGgKKknkWosDTFl97J+ IPxQeqoVuj92DT9MSm00z6SLrX9XtD4dK762zcQ2Tsvz9j9KW21jiWo9sUor6Lpe Dh4Mle0V/dGJOBi7j7UiQOSQCRikl20zTZtId7wsFqsDVGSFlOKTvW+l3XLok0KU ufVnI46N8mrnUWFqDU5SQ952gelIsIAdj6UUwKioZQMafeZvvRLwFA== =9NxF -----END PGP SIGNATURE-----

3 7

Syslog-ng Message Statistics
by vincent＠ragosta.net 21 Dec '11

21 Dec '11

Some of our older systems are utilizing syslog-ng 1.6.12 and I am interested in gathering log counts for each unique source that is reporting. I don't believe there is any mechanism in this version of syslog-ng for performing such analysis, correct? Thanks.

2 1

syslog-ng remote MSSQL databases
by Bely Silva Junior 21 Dec '11

21 Dec '11

dear Srs I need configure syslog-ng insert data on remote MSSQL database, it's possible ? my conf: destination d_mssql { sql(type(mssql) host("ipaddres") port("1433") username("syslogng") password("password") database("syslogng") table("logs") columns("datetime varchar(16)","host varchar(32)" ,"program varchar(32)","pid varchar(8)","message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes() ); }; I had error when restart syslog-ng say : syntax erro -- Bely Silva Junior

2 1

Invalid frame header messages
by Mr. Jan Walter 21 Dec '11

21 Dec '11

Hi everyone, I compiled version 3.3.3 and tried getting a client and server to talk based on the documentation on the balabit site, specifically the documentation for OSE version 3.3. The server config file: -- source remotelogs { syslog( port(2072) transport(tcp) ); } ; destination r_log { file("/var/log/remotelogs/all"); }; log{ source(remotelogs); destination(r_log); } ; -- And the client sending: -- # remote logging test log # @version: 3.3 source s_kern { file("/proc/kmsg"); }; source s_local { unix-stream("/dev/log"); internal(); }; destination d_file {file("/var/log/messages"); }; destination d_secure { file("/var/log/secure");}; # set the filters filter f_messages { not facility(auth, authpriv);}; filter f_authpriv { facility(auth, authpriv); }; destination loghost { tcp(10.195.105.214 port(2072) ); }; log { source(s_kern) ; source(s_local) ; filter(f_messages) ; destination(loghost) ; } ; -- There are some other log directives there for local files, I'll post them if people want to see them So when this runs, I get on the server: Dec 7 11:18:05 xxxxxxxxxx syslog-ng[4188]: Syslog connection closed; fd='21', client='AF_INET(10.203.54.9:54837)', local='AF_INET(0.0.0.0:2072)' And the client: Dec 7 11:19:04 xxxxxxxxxxxx syslog-ng[1735]: Syslog connection established; fd='20', server='AF_INET(10.195.105.214:2072)', local='AF_INET(0.0.0.0:0)' I read the email thread from 2009, and tried the vanilla configs - any ideas? Thanks, Jan

3 4

syslog-ng 3.3.3 pipe source does not parse host
by Evan Rempel 21 Dec '11

21 Dec '11

Try as I might, I can not get a pipe source to maintain the hostname. Syslog-ng includes the hostname as the $MSGHDR (or the $PROGRAM). my config # ----------------------------------- source patterndb { pipe("/var/log/syslog.pipes/patterndb" log_fetch_limit(500) log_iw_size(100000) ); }; source int { internal(); }; template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); }; destination d_var_patterndb { file("/var/log/patterndb.$R_YEAR$R_MONTH$R_DAY.000000" owner("root") group("syslogs") perm(0640) template(t_standardfile)); }; log { source(patterndb); destination(d_var_patterndb); }; # ----------------------------------- data I write tothe pipe # ----------------------------------- <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- The output file # ----------------------------------- 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- Am I missing something silly with the 3.3 config syntax (Up until now I have only used up to 3.0) Evan.

2 1

[Bug 153] New: default year when no year is present should be current year
by bugzilla＠bugzilla.balabit.com 21 Dec '11

21 Dec '11

https://bugzilla.balabit.com/show_bug.cgi?id=153 Summary: default year when no year is present should be current year Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When parsing a source that has a month and day but no year, the S_YEAR macro does not default to R_YEAR. It seems to be defaulting to R_YEAR+1 What is the intention when there is no year in the source? To test, use historicatl syslog or rsyslog to create a syslog file, then use that file as a source (I was doing a "cat filename > /smoe/pipe" and using a pipe source) and write a file that has an ISODate in it. The year is 2012. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng 3.3.3 and multiple incarnations of the same parser
by Michael Hocke 21 Dec '11

21 Dec '11

-----BEGIN PGP SIGNED MESSAGE----- Hi guys, I ran into another problem with 3.3.3. It seems that I can use a cvs-parser only in one and only one log path. If I reuse it in multiple log paths the columns of the cvs parser are empty. I downgraded to 3.0.9 and that resolved my problem. Here is an example config: - --------< SNIP >---------- source s_remote { udp () }; parser p_hostname_site { csv-parser (columns ("REMOTEHOST.SITE", "REMOTEHOST.NAME") delimiters ("-") template ("${HOST}")); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; destination d_cacl { file ("/usr/local/var/log/current/${REMOTEHOST.SITE:-other}/cisco-acl.${S_YEAR}${S_MONTH}${S_DAY}${S_HOUR}.log" template ("$S_DATE ${REMOTEHOST.NAME:-none} $MSGHDR$MSG\n") }; destination d_caps { file ("/usr/local/var/log/current/${REMOTEHOST.SITE:-other}/cisco-aps.${S_YEAR}${S_MONTH}${S_DAY}${S_HOUR}.log" template ("$S_DATE ${REMOTEHOST.NAME:-none} $MSGHDR$MSG\n") }; log (source (s_remote); filter (f_local0); parser (p_hostname_site); destination (d_cacl); }; log (source (s_remote); filter (f_local1); parser (p_hostname_site); destination (d_caps); }; - --------< SNIP >---------- The incoming logs for local0 would look like this: Dec 2 16:00:00 florence-VPN-FLORENCE %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 florence-VPN-FLORENCE %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 Under 3.3.3 it would produce the following logs: /usr/local/var/log/current/other/cisco-acl.2011120216.log: Dec 2 16:00:00 none %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 none %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 The same configuration with 3.0.9 produces the following: /usr/local/var/log/current/florence/cisco-acl.2011120216.log: Dec 2 16:00:00 VPN-FLORENCE %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 VPN-FLORENCE %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 If I remove the parser from one of the log paths 3.3.3 would spit out the same output as 3.0.9 but of course, the second log path would need its own parser to make it work. I am unable to check syslog-ng 3.1 or 3.2. - - Michael -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.0.3 (Build 1) Charset: us-ascii wsBVAwUBTtjvwJbfnpCg64TVAQFmvwf9EZ3wShhBxIXOGXWiSJguF07HCgw99Wxu amUrvAAeI9nZSj4wvpQCZNLrpPJ2Q7/DCSKTunzopHNJVwMQQvFSwzaAcezNNt4Y ogmTAx+TeyMc4rYuknpG9Khz6WXFTM8rH4imjNziDSxSwkqZrCQDnmYM4B2pxfqo BNZbcRKifbo9mts0FbeD9hpBiYNxt+k9pXPtqUD193ihUWcnNF4NG3hgt/1RXB7E 7v79A6swf30PXSLpdJn8fATptKVrfzzvXlW0ncNolSUyKK7XaY4kgvjsnFba++CS eTil+xkINNnuUz2FDYfEDziTYFeezO/lhSrhE8BiDaRt1Gx10XfbDQ== =2Idy -----END PGP SIGNATURE-----

4 5

[PATCH] afprog: Terminate the child process on deinit.
by Gergely Nagy 21 Dec '11

21 Dec '11

When reloading the configuration, terminate the child process of a program destination, for the following reasons: it will be restarted anyway by afprog_dd_reopen(), so there's no harm done. But if dash is used as /bin/sh, simply closing the stdin of the process will not make it stop. That, in turn, results in us starting the same program over and over again on each SIGHUP. So instead of doing this, send a SIGTERM to the child on deinit, too. Reported-by: Thomas Wollner <tw(a)wollner-net.de> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afprog/afprog.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/modules/afprog/afprog.c b/modules/afprog/afprog.c index 6ce74b0..264e85b 100644 --- a/modules/afprog/afprog.c +++ b/modules/afprog/afprog.c @@ -328,6 +328,16 @@ afprogram_dd_deinit(LogPipe *s) { AFProgramDestDriver *self = (AFProgramDestDriver *) s; + if (self->pid != -1) + { + msg_verbose("Sending destination program a TERM signal", + evt_tag_str("cmdline", self->cmdline->str), + evt_tag_int("child_pid", self->pid), + NULL); + kill(self->pid, SIGTERM); + self->pid = -1; + } + if (self->writer) log_pipe_deinit(self->writer); -- 1.7.7.3

2 1

patterndb - user defined parsers
by Evan Rempel 19 Dec '11

19 Dec '11

It would be useful to permit users to define parsers in the patterndb. For example, in our environment, by policy we user a special set and order of characters of our administrators log into hosts and administer them. It would be useful to define a parser of @SYSADMIN@ that would match only our sysadmin accounts. We could then use this parser in the patterndb to take some action such as sending a message to the administrators about the event. Another example would be to create parser for @LOCALIP@ that matches my organizaions IP space. That way a set of rules can be defined using @LOCALIP@ for some kind of alerting, and then any organization could redifine the @LOCALIP@ and use all of the goodness that some third party had created for monitoring logs like an intrusion protection system. Current parsers can be described as QSTRING - match opening char - while not closing char, keep looking ESTRING - while not end string, keep looking NUMBER - while digit keep looking So it seems that general parsers could be constructed with two styles of matching, and then concatenating the together. 1. While in set of characters [some list of characters] 2. While not in set of characters [some list of characters] I would call these INSET to match 1 or more of a set of characters, unless a #-# were specified, then a minimum to a maximum would be required. OUTSET to match 1 or more of anything except the characters, unless a #-# were specified, then a minimum to a maximum would be required. (perhaps a count of + or * could be used to specify 1 or more and 0 or more respectively) and then limit the count of such occurrences so that you could build the @IPv4@ parser as @INSET::123456789*1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@.@INSET::123456789:1@@INSET::0123456789:0-2@ and @NUMBER@ would be @INSET::123456789:1@@INSET::0123456789@ @FLOAT@ would be @INSET::0123456789.@ Then a user could make <parser name="THOUSAND">@INSET::,:0-1@@INSET::0123456789:3@</parser> <parser name="MONEY">$@INSET::123456789:1-3@@THOUSAND:::*@.@INSET::0123456789:2@ This is kind of like inventing regular expressions :-( I'm not sure how well this fits into the radix tree matching structure, but I wanted to start this discussion. Given the MONEY example, I think it is obvious that there needs to be a way to specify repeating groups of "something" Let the discussion begin!

4 9