September 2011 - syslog-ng - lists.balabit.hu

Spoof-source option on Cygwin
by pedro castro 27 Sep '11

27 Sep '11

Hi we have Windows 2003 and Windows 2008 with cygwin. We receive events from several devices on syslog-ng server and redirect them to other systems. I want to use spoof-source option but i see that i need to compile syslog-ng from source. I have tried it but it fails in make. Does anyone have any idea? Regards

2 4

Re: [syslog-ng] syslog-ng Digest, Vol 77, Issue 34
by Luis Pugoy 27 Sep '11

27 Sep '11

> > Message: 1 > Date: Mon, 26 Sep 2011 18:07:00 +0800 > From: Luis Pugoy <lpugoy(a)insynchq.com> > Subject: Re: [syslog-ng] How to send tracebacks through syslog-ng? > (Jakub Jankowski) > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <CA+WGxEhGFX7V9C+71BgcU1TZsb21+FbzX5GdCZ=ZLPEDJ4fYFg(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > On Sun, Sep 25, 2011 at 1:57 AM, Luis Pugoy <lpugoy(a)insynchq.com> wrote: > > > Message: 4 > >> Date: Sat, 24 Sep 2011 09:31:34 +0200 (CEST) > >> From: Jakub Jankowski <shasta(a)toxcorp.com> > >> Subject: Re: [syslog-ng] How to send tracebacks through syslog-ng? > >> To: Syslog-ng users' and developers' mailing list > >> <syslog-ng(a)lists.balabit.hu> > >> Message-ID: <alpine.LNX.2.00.1109240928240.15891(a)bluzg.slackware.pl> > >> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > >> > >> On 2011-09-24, Luis Pugoy wrote: > >> > >> > rewrite r_rewrite_subst { subst("a*", "?", value("MESSAGE") > >> type("pcre") > >> > flags("utf8" "global")); }; > >> > > >> > I thought that it would replace 0 or more occurrences of 'a' with a > >> question > >> > mark. However, when the input is 'a', the output becomes 'a?'. It just > >> > appends a question mark at the end. 'aa' also becomes 'aa?'. What is > >> weirder > >> > still is that even if the input does not contain 'a', the question > mark > >> is > >> > still appended. So '1' becomes '1?'. > >> > >> Well, does "1" match /a*/ ? :) > >> It doesn't make any difference if that's /a*/, /b*/ or /.*/ - this kind > of > >> regexp matches everything, even an empty string. > >> > >> HTH > >> > >> -- > >> Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ > >> GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D > > > > > > Hello Jakub. > > > > Thanks for the reply. You're right in that "1" matches /a*/, but wouldn't > > the resulting replacement be "?" instead of "1?" ? Also, if I just use > /a/ > > and use "a" as input, it does not seem to match. The text is still > outputted > > as "a". > > > > I don't know the policy regarding bumping threads here, but I'd like to > bump > this. Can anyone confirm the results I'm getting? Is there also something > wrong with my understanding of pcre? I'm sorry if this is not allowed. > I was able to solve my initial problem. The "\p{Zl}" must be changed to "\\p}{Zl}".

1 0

[PATCH] correlation: explicit context timeout action
by Balint Kovacs 23 Sep '11

23 Sep '11

Hi, I'm trying to use patterndb correlation to find matching firewall connection startup and ending log messages and emit a consolidated message. I found that with high-volume load, the memory usage of syslog-ng climbs rapidly and it would be beneficial to be able to kick already ended events from the memory. Please find a patch below (against 3.4) that does this, I thought the easy place for the user would be the <action> part of the ending rule, this way it can even be specified when the context should be ended. Balint From: Balint Kovacs <blint(a)balabit.hu> Date: Wed, 21 Sep 2011 11:46:30 +0200 Subject: [PATCH] correlation: add action to explicitly end context If there is a message that can be associated with the end of a context, the below action can explicitly end it, reducing the memory footprint. Example: <action> <end-context /> </action> Signed-off-by: Balint Kovacs <blint(a)balabit.hu> --- modules/dbparser/patterndb-int.h | 3 ++- modules/dbparser/patterndb.c | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletions(-) diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..eff77e6 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -115,7 +115,8 @@ enum enum { RAC_NONE, - RAC_MESSAGE + RAC_MESSAGE, + RAC_EXPIRE }; /* a rule may contain one or more actions to be performed */ diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 343ef6d..72e97e0 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -577,6 +577,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con emit(genmsg, TRUE, emit_data); log_msg_unref(genmsg); break; + case RAC_EXPIRE: + if (context) + { + msg_debug("Expiring patterndb correllation context as directed by action", + evt_tag_str("last_rule", context->rule->rule_id), + evt_tag_long("remaining_context_count", g_hash_table_size(context->db->state)), + NULL); + g_hash_table_remove(context->db->state, &context->key); + timer_wheel_del_timer(db->timer_wheel, context->timer); + pdb_context_unref(context); + } + break; default: g_assert_not_reached(); break; @@ -921,6 +933,16 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name state->current_action->content_type = RAC_MESSAGE; state->current_message = &state->current_action->content.message; } + else if (strcmp(element_name, "end-context") == 0) + { + if (!state->in_action) + { + *error = g_error_new(1, 0, "Unexpected <end-context> element, it must be inside an action"); + return; + } + state->current_action->content_type = RAC_EXPIRE; + state->current_message = &state->current_action->content.message; + } } void -- 1.7.0.4

1 2

Re: [syslog-ng] Is it possible to replace a string with a new line using rewrite?
by Luis Pugoy 22 Sep '11

22 Sep '11

> > > Message: 4 > Date: Wed, 21 Sep 2011 21:20:44 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Is it possible to replace a string with a new > line using rewrite? > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87pqitr9bn.fsf(a)luthien.mhp> > Content-Type: text/plain > > Luis Pugoy <lpugoy(a)insynchq.com> writes: > > > I am trying to replace a string with a newline using rewrite. Is it > > possible? Thanks. > > rewrite r_nl { > subst("<insert regexp here>", > "\n", value("MSG")); > }; > > Or, if you want to replace the whole MSG with a newline: > > rewrite r_nlmsg { set("\n", value("MSG")); }; > > -- > |8] > > Thanks. I was using single quotes ('\n') so it wasn't working for me before. I didn't realize that there's a distinction between single and double quotes in syslog-ng.

1 0

Is it possible to replace a string with a new line using rewrite?
by Luis Pugoy 21 Sep '11

21 Sep '11

Hello, I am trying to replace a string with a newline using rewrite. Is it possible? Thanks.

2 1

syslog-ng 3.3.0beta2 chroot
by Dave Haywood 21 Sep '11

21 Sep '11

Hi, I have compiled syslog-ng 3.3.0beta2 under RHEL AS 4.8 (don't ask) and am having a possible chroot() issue. syslog-ng is running chroot as a non-root user. All the build libraries etc (there were a lot to get this to run under RHEL 4.8) are under /usr/local. I already run syslog-ng 2.0.9 under RHEL AS 4.9. I am seeing differences between the 2.0.9 and 3.3.0b2 releases. The differences with 3.3.0b2 are: 1) daemon seems to switch user *before* binding the network interfaces. This means I can't bind to TCP/UDP ports < 1024. 2) daemon chroot()s before loading the syslog-ng.conf file. This means I have to mount the config dir under the chroot() dir. So, my questions are: 1) Is the behaviour I describe in 3.3.30b2 intentional or could this be a problem with my environment? 2) Why has the behaviour changed in this way? 3) If the daemon were to: read the config file bind the network interfaces as root *before* dropping privelege and doing the chroot() couldn't a lot of the chroot() hastle (below) be avoided? I have 3.3.0b2 running chroot() but only by: mount -o bind /lib /chrootdir/lib mount -o bind /usr/local /chrootdir/usr/local mount -o bind /dev /chrootdir/dev changing the syslog-ng.conf TCP/UDP listen ports to be > 1024 Regards, Dave.

2 1

Messages starting with tab not processed cleanly
by Jon Nathan 21 Sep '11

21 Sep '11

We're running into an issue where some syslog messages on our centralized syslog-ng server lack originating hostnames. These messages happen to be formatted funny - the message part starts with a tab (\0x09) character. Solaris 10 x86 everywhere, Syslog-ng 3.1.4, Eventlog 0.2.12. Client syslog: Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.warning] WARNING: /pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0 (sd2): Sep 15 20:30:21 h.example.com Error for Command: read Error Level: Fatal Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] Requested Block: 16065 Error Block: 16065 Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] Vendor: SEAGATE Serial Number: 074093HVSX Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] Sense Key: Aborted Command Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] ASC: 0x8 (LUN communication time-out), ASCQ: 0x1, FRU: 0x81 This shows up in my centralized syslog-ng server as: Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.warning] WARNING: /pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0 (sd2): Sep 15 20:30:21 Error for Command: read Error Level: Fatal Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] Requested Block: 16065 Error Block: 16065 Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] Vendor: SEAGATE Serial Number: 074093HVSX Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] Sense Key: Aborted Command Sep 15 20:30:21 h.example.com scsi: [ID 107833 kern.notice] ASC: 0x8 (LUN communication time-out), ASCQ: 0x1, FRU: 0x81 Note that the 2nd line doesn't have a hostname. This is the message coming from the host (tcpdump -n -v -s 0 "port 514"): 20:30:21.454902 IP (tos 0x0, ttl 255, id 43670, offset 0, flags [DF], proto UDP (17), length 141) 10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 113 Facility kernel (0), Severity warning (4) Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.warning] WARNING: /pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0 (sd2): 20:30:21.454941 IP (tos 0x0, ttl 255, id 43671, offset 0, flags [DF], proto UDP (17), length 109) 10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 81 Facility kernel (0), Severity warning (4) Msg: Sep 15 20:30:21 \0x09Error for Command: read Error Level: Fatal 20:30:21.454966 IP (tos 0x0, ttl 255, id 43672, offset 0, flags [DF], proto UDP (17), length 139) 10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 111 Facility kernel (0), Severity notice (5) Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09Requested Block: 16065 Error Block: 16065 20:30:21.454986 IP (tos 0x0, ttl 255, id 43673, offset 0, flags [DF], proto UDP (17), length 148) 10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 120 Facility kernel (0), Severity notice (5) Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09Vendor: SEAGATE Serial Number: 074093HVSX 20:30:21.455010 IP (tos 0x0, ttl 255, id 43674, offset 0, flags [DF], proto UDP (17), length 104) 10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 76 Facility kernel (0), Severity notice (5) Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09Sense Key: Aborted Command 20:30:21.455031 IP (tos 0x0, ttl 255, id 43675, offset 0, flags [DF], proto UDP (17), length 137) 10.1.1.1.59299 > 10.1.1.2.514: SYSLOG, length: 109 Facility kernel (0), Severity notice (5) Msg: Sep 15 20:30:21 scsi: [ID 107833 kern.notice] \0x09ASC: 0x8 (LUN communication time-out), ASCQ: 0x1, FRU: 0x81 (Hostnames/IPs sanitized) The second message starts with a tab (\0x09) after the timestamp. This seems to throw off syslog-ng such that it won't append a hostname to the message. We do a lot of parsing that relies on the hostname being present in all messages. Any idea why this seems to break syslog-ng, or how to fix it? Our syslog-ng.conf is relatively generic. Here are the relevant parts: filter f_syslog { level(info..emerg); }; destination d_syslog { file("/syslog/logs/$YEAR/$MONTH/$DAY/syslog" flags(no-multi-line)); }; log{ source(s_sys); source(s_remote); filter(f_syslog); destination(d_syslog); }; (Sorry if this hits the list twice, I don't think my first try worked.) Thanks, -Jon

4 3

buffer logs from initramfs until syslog-ng starts
by Patrick H. 21 Sep '11

21 Sep '11

So I'm trying to find a way to buffer logs from extremely early in the boot process (from when the initramfs is still running), and then dump them to syslog-ng once it starts. Has anyone done anything similar? The only real idea that comes to mind is to use busybox's syslogd and have it use a circular buffer, then configure syslog-ng with a program source that reads the buffer and then terminates the the daemon (something like 'logread ; pkill -x syslogd &>/dev/null'), and set follow_freq to 0. Whether this will work or not, I dont know as I've never played with program sources. Also a problem I see with this is that the busybox syslogd would need to be killed before syslog-ng tries to open /dev/log and /proc/kmsg, but I'm not sure how to go about that. I do seem to recall some other utility I've ran across in my travels that is designed for this exact purpose, in that it buffers syslog messages during boot, then dumps its buffer once the main syslog daemon has started and terminates, but I cant remember what its called (and google is failing me). So does anyone have any good ideas for solving this situation?

4 4

Re: [syslog-ng] Syslog-NG error msg : out of memory
by Henry Xu 21 Sep '11

21 Sep '11

Any one can update on this? Thanks. From: Henry Xu [mailto:bxu@italkbb.com] Sent: Wednesday, September 07, 2011 13:03 To: 'syslog-ng(a)lists.balabit.hu' Subject: Syslog-NG error msg : out of memory Hi All, Today I got monitoring alarms about Syslog-NG. I checked log and found the following errors: Sep 6 21:35:35 host1 1:manager[db] [353-1] ERROR: out of memory Sep 6 21:35:35 host1 1:manager[db] [353-2] DETAIL: Failed on request of size 131071998. Sep 6 21:35:35 host1 1:manager[db] [354-1] ERROR: out of memory Sep 6 21:35:35 host1 1:manager[db] [354-2] DETAIL: Failed on request of size 131071998. Sep 6 21:35:35 host1 1:manager[db] [355-1] ERROR: out of memory Sep 6 21:35:35 host1 1:manager[db] [355-2] DETAIL: Failed on request of size 131071998. Sep 6 21:35:35 host1 1:manager[db] [356-1] ERROR: out of memory Sep 6 21:35:35 host1 1:manager[db] [356-2] DETAIL: Failed on request of size 131071998. Sep 6 21:35:35 host1 1:manager[db] [357-1] ERROR: out of memory Sep 6 21:35:35 host1 1:manager[db] [357-2] DETAIL: Failed on request of size 131071998. Is this related to the memory leak? Does someone have the same experience and whether have a way to avoid this? Thanks, Henry

3 2

program filter regex
by Matt Zagrabelny 21 Sep '11

21 Sep '11

Hi, I've looked through the OSE reference PDF and have a question about the program filter regex. I've done some experimentation and I would have thought the following to work: I'm trying to match puppet programs in syslog-ng (version 3.1) log entry: Sep 2 14:36:25 hornet puppet-agent[15008]: Finished catalog run in 0.75 seconds This filter works: filter f_puppet { program("puppet.*"); }; But this does not: filter f_puppet { program("puppet"); }; For things like grep, perl, and ruby the latter regex ("puppet") would have matched the program string "puppet-agent". Syslog-ng does not seem to behave the same way. Is there a way to get the grep/perl behavior? If not, it seems like it doesn't follow the conventions of other free software. Would anyone consider it a wishlist bug? Thanks for the info or advice! -matt zagrabelny

2 1