December 2011 - syslog-ng - lists.balabit.hu

Syslog-ng is not sending messages line by line
by Alptugay 06 Dec '11

06 Dec '11

Hello all, I have a python script which prints 30 lines. And my syslog-ng is listening this script and writing it's messages to a stream. But the application which is listening to the stream recieves all the lines in one line. To be more clear my source program is basically like this: while i<30: print "cccc dddd" i=i+1 But my application listening to the stream gets a line like this: [[u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:1 1 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 c ccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd' ], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13 >Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 1 1:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 d dos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>De c 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:4 7:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'], [u'<13>Dec 6 11:47:11 ddos1 cccc dddd'] .... etc But it should recieve one line from the stream for one print command of the source program. Btw I am using freeBSD _____________________________________________________________________ Register now! Use this rare opportunity! http://partners.labris.eu

2 1

Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu 10.10
by Luis Pugoy 05 Dec '11

05 Dec '11

> > Message: 3 > Date: Mon, 05 Dec 2011 10:10:40 +0100 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > 10.10 > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <87aa77l6y7.fsf(a)algernon.balabit> > Content-Type: text/plain > > Luis Pugoy <lpugoy(a)insynchq.com> writes: > > > Does anyone have any insight into this? The segfault is happening more > > often now. :( > > I'd suggest either upgrading to 3.3 from my repo[1], or if an upgrade is > not feasible, then if you could compile 3.1 with debug symbols, get it > to drop a core, and do a backtrace, that would help tremendously. > > [1]: http://asylum.madhouse-project.org/projects/debian/ > > To do the latter, the following steps should work: > > $ apt-get source syslog-ng > $ sudo apt-get build-dep syslog-ng > $ cd syslog-ng-* > $ DEB_BUILD_OPTIONS="debug nostrip" dpkg-buildpackage -us -uc -rfakeroot > $ sudo dpkg -i ../syslog-ng_*.deb > > Then add --enable-core to SYSLOGNG_OPTS in /etc/default/syslog-ng, and > when it segfaults, it'll make a core in (as far as I remember) /. Once > that happened, do something like this: > > $ gdb -c /core /usr/bin/syslog-ng > (gdb) thread apply all bt full > > This should give us a few hints. > > -- > |8] > I installed a newer version of the PCRE library. That seems to have worked for now. If it happens again I will be sure to follow your instructions, thanks.

1 0

syslog-ng 3.1.2 segmentation fault in Ubuntu 10.10
by Luis Pugoy 05 Dec '11

05 Dec '11

Hello, Our syslog-ng server is lately raising a segfault every day or so. The error is of the form "segfault at 7f8677df0338 ip 00007f8676fa8e9d sp 00007fff1d2ec280 error 4 in libpcre.so.3.12.1[7f8676f96000+30000]." Does anyone know what this means? Thanks.

3 3

Question on syslog-ng filtering and performance - host vs. netmask
by jrhendri＠maine.rr.com 03 Dec '11

03 Dec '11

Hi, I am using syslog-ng to do log collection and "routing" to different destinations based on the origin of the log. I used to (still do in some cases) filter like this: filter f_one_list { host("10.0.0.1") or host("10.0.2.1") or ... host("10.3.4.5"); } Since (too) many log sources break the RFC and put all kinds of strangeness in the "header", I started using netmask instead: filter f_another_list { netmask("10.0.0.1/32") or netmask("10.0.2.1/32") or ... netmask("10.3.4.5/32"); } My question is this - Is there a measurable performance difference from one to another? "intuitively" it would seem doing the netmask check should be faster than parsing the syslog header, but I thought I'd ask here. (currently I have some servers processing 4000+ messages per second with no real problem, but the CPUs are clearly busy. BTW - I am using 3.2.4 at the moment (looking at moving to the multi-threaded newer versions in the future) Thanks! Jim Hendrick

3 2

syslog-ng log server stops listening on network
by Paul Muther 02 Dec '11

02 Dec '11

Hello all, I have a syslog-ng log server running 3.2.4 listening on a TCP port 4514. We are not a high volume environment (server processes on average between 10 and 20 MPS). Twice now the server has just stopped accepting traffic on the TCP port causing all the clients to store messages. There seems to be no explanation for the event and the local service is running and processing all local logs for the server no problem. I feel like this might be an IP port conflict but I just want to make sure I'm not missing anything. Thanks much, Paul M

2 1

syslog-ng 3.3.3 rewrite question regarding cisco IOS Messages
by Thomas Wollner 02 Dec '11

02 Dec '11

Hello List, I try to rewrite cisco IOS syslog messages with timestamps in the MESSAGE field. I want to remove the timestamp from the message and set the program to the so called mnemonic of the message.. I found the following example on the list: ... filter f_rewrite_cisco_program { match('%([^:]: ([^\n]+)' value("MESSAGE") type("pcre") flags("store-matches" "nobackref")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program))); }; log { source(s_all); rewrite(r_cisco_program); destination(d_mydestination); }; But that does not work. I tried a lot of different rewrite syntaxes, none of them work for me. If I just employ the filter f_rewrite_cisco_program I` am able to filter out the cisco messages. Sample log line (written with template $R_ISODATE $HOST $MSG): 2011-11-30T18:23:50+01:00 192.168.1.1 217122: Nov 30 17:23:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down I`m using syslog-ng 3.3.3 debian package from madhouse. How I can rewrite my messages to filter the timestamp in the message field? any ideas? Any help is higly welcome, thanks in advance, Tom ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

4 18

patterndb skewing timestamps
by Clements, Frank 01 Dec '11

01 Dec '11

I've been working a bit over the last few weeks with patterndb - specifically correlating messages. I just noticed that when log messages are correlated, patterndb is skewing the timestamps from the first message (possibly others in between as well) by a few seconds. While, this may not seem like a big problem it results in logs that do not align property in history. A little on the configuration. I have two patterns to match on receiving the second I generate a new message with the format: logHost=${HOST} mapStart=${S_UNIXTIME}@2 mapStop=${S_UNIXTIME} protocol=${PROGRAM} insideAddr=${.dict.insideAddr}@1 insidePort=${.dict.insidePort}@1 outsideAddr=${.dict.outsideAddr}@1 outsidePort=${.dict.outsidePort}@1 destAddr=${.dict.destAddr}@2 destPort=${.dict.destPort}@2 My original log lines are as follows: Dec 1 08:39:41 AX2600 UC: e0a8636e:a16c->5f2c65b3:a16c to 8f2c77ca:a1 Dec 1 08:48:06 AX2600 UF: e0a8636e:a16c->5f2c65b3:a16c The generated log line is: logHost=RHOSTNAME mapStart=1322747067 mapStop=1322747286 protocol=U insideAddr=e0a8636e insidePort=a16c outsideAddr=5f2c65b3 outsidePort=a16c destAddr=80a8650c destPort=3f According to the docs S_ represents the log message time - so that means mapStart should line up, but it does not: 1322747067 == Thu Dec 01 2011 08:44:27. The stop message does not suffer from this issue. Anyone have a workaround for this? I tried using R_UNIXTIME@2 instead, but that is also skewed. -- Frank Clements

1 0

An interesting assert in 3.3.1
by Gergely Nagy 01 Dec '11

01 Dec '11

A while ago I noticed that my syslog-ng 3.3 segfaults every time I boot up my desktop machine at home (but then it gets restarted by the supervisor), and once its booted up, I can't reproduce the problem anymore. I assume it has something to do with the system being in a reasonably early state, or the load being particularly high. While I didn't have time to debug it further, I'd like to record the current state of investigation here: Core was generated by `/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --enable-core'. Program terminated with signal 6, Aborted. #0 0x00007fd927cc6405 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. in ../nptl/sysdeps/unix/sysv/linux/raise.c (gdb) bt #0 0x00007fd927cc6405 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007fd927cc9680 in *__GI_abort () at abort.c:92 #2 0x00007fd928ae6e31 in g_assertion_message (domain=<optimized out>, file=<optimized out>, line=<optimized out>, func=0x7fd929609dc0 "log_writer_work_perform", message=0x7fd920007a60 "assertion failed: ((self->super.flags & PIF_INITIALIZED) != 0)") at /tmp/buildd/glib2.0-2.28.6/./glib/gtestutils.c:1358 #3 0x00007fd928ae73d0 in g_assertion_message_expr (domain=0x0, file=0x7fd929609acb "../../../lib/logwriter.c", line=128, func=0x7fd929609dc0 "log_writer_work_perform", expr=<optimized out>) at /tmp/buildd/glib2.0-2.28.6/./glib/gtestutils.c:1369 #4 0x00007fd9295d3570 in log_writer_work_perform (s=0x1d68080) at ../../../lib/logwriter.c:128 #5 0x00007fd9295d3897 in log_writer_io_flush_output (s=0x1d68080) at ../../../lib/logwriter.c:198 #6 0x00007fd9295ff986 in iv_run_active_list (active=0x7fff3da27700) at ../../../../../lib/ivykis/lib/iv_main.c:226 #7 0x00007fd9295ffb0d in iv_main () at ../../../../../lib/ivykis/lib/iv_main.c:269 #8 0x00007fd9295d8082 in main_loop_run () at ../../../lib/mainloop.c:722 #9 0x0000000000401a22 in main (argc=1, argv=0x7fff3da27878) at ../../../syslog-ng/main.c:260 (gdb) The particular writer is a program destination (sshguard), and at that point, it's probably not running yet. (For the record, I don't have threaded(yes) in my config at the moment). I'll look into this further, as time permits. -- |8]

1 1

ELSA Google Group
by Martin Holste 01 Dec '11

01 Dec '11

Just a quick note for those of you using or interested in ELSA, I created a Google Group for it at https://groups.google.com/group/enterprise-log-search-and-archive . Still feel free to email me directly for help, but I created the group so that the questions and answers might be visible to others searching on Google for help. Thanks, Martin

1 0