- syslog-ng - lists.balabit.hu

syslog-ng 3.3.3 rewrite question regarding cisco IOS Messages
by Thomas Wollner 02 Dec '11

02 Dec '11

Hello List, I try to rewrite cisco IOS syslog messages with timestamps in the MESSAGE field. I want to remove the timestamp from the message and set the program to the so called mnemonic of the message.. I found the following example on the list: ... filter f_rewrite_cisco_program { match('%([^:]: ([^\n]+)' value("MESSAGE") type("pcre") flags("store-matches" "nobackref")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program))); }; log { source(s_all); rewrite(r_cisco_program); destination(d_mydestination); }; But that does not work. I tried a lot of different rewrite syntaxes, none of them work for me. If I just employ the filter f_rewrite_cisco_program I` am able to filter out the cisco messages. Sample log line (written with template $R_ISODATE $HOST $MSG): 2011-11-30T18:23:50+01:00 192.168.1.1 217122: Nov 30 17:23:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down I`m using syslog-ng 3.3.3 debian package from madhouse. How I can rewrite my messages to filter the timestamp in the message field? any ideas? Any help is higly welcome, thanks in advance, Tom ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

4 18

patterndb skewing timestamps
by Clements, Frank 01 Dec '11

01 Dec '11

I've been working a bit over the last few weeks with patterndb - specifically correlating messages. I just noticed that when log messages are correlated, patterndb is skewing the timestamps from the first message (possibly others in between as well) by a few seconds. While, this may not seem like a big problem it results in logs that do not align property in history. A little on the configuration. I have two patterns to match on receiving the second I generate a new message with the format: logHost=${HOST} mapStart=${S_UNIXTIME}@2 mapStop=${S_UNIXTIME} protocol=${PROGRAM} insideAddr=${.dict.insideAddr}@1 insidePort=${.dict.insidePort}@1 outsideAddr=${.dict.outsideAddr}@1 outsidePort=${.dict.outsidePort}@1 destAddr=${.dict.destAddr}@2 destPort=${.dict.destPort}@2 My original log lines are as follows: Dec 1 08:39:41 AX2600 UC: e0a8636e:a16c->5f2c65b3:a16c to 8f2c77ca:a1 Dec 1 08:48:06 AX2600 UF: e0a8636e:a16c->5f2c65b3:a16c The generated log line is: logHost=RHOSTNAME mapStart=1322747067 mapStop=1322747286 protocol=U insideAddr=e0a8636e insidePort=a16c outsideAddr=5f2c65b3 outsidePort=a16c destAddr=80a8650c destPort=3f According to the docs S_ represents the log message time - so that means mapStart should line up, but it does not: 1322747067 == Thu Dec 01 2011 08:44:27. The stop message does not suffer from this issue. Anyone have a workaround for this? I tried using R_UNIXTIME@2 instead, but that is also skewed. -- Frank Clements

1 0

An interesting assert in 3.3.1
by Gergely Nagy 01 Dec '11

01 Dec '11

A while ago I noticed that my syslog-ng 3.3 segfaults every time I boot up my desktop machine at home (but then it gets restarted by the supervisor), and once its booted up, I can't reproduce the problem anymore. I assume it has something to do with the system being in a reasonably early state, or the load being particularly high. While I didn't have time to debug it further, I'd like to record the current state of investigation here: Core was generated by `/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --enable-core'. Program terminated with signal 6, Aborted. #0 0x00007fd927cc6405 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. in ../nptl/sysdeps/unix/sysv/linux/raise.c (gdb) bt #0 0x00007fd927cc6405 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007fd927cc9680 in *__GI_abort () at abort.c:92 #2 0x00007fd928ae6e31 in g_assertion_message (domain=<optimized out>, file=<optimized out>, line=<optimized out>, func=0x7fd929609dc0 "log_writer_work_perform", message=0x7fd920007a60 "assertion failed: ((self->super.flags & PIF_INITIALIZED) != 0)") at /tmp/buildd/glib2.0-2.28.6/./glib/gtestutils.c:1358 #3 0x00007fd928ae73d0 in g_assertion_message_expr (domain=0x0, file=0x7fd929609acb "../../../lib/logwriter.c", line=128, func=0x7fd929609dc0 "log_writer_work_perform", expr=<optimized out>) at /tmp/buildd/glib2.0-2.28.6/./glib/gtestutils.c:1369 #4 0x00007fd9295d3570 in log_writer_work_perform (s=0x1d68080) at ../../../lib/logwriter.c:128 #5 0x00007fd9295d3897 in log_writer_io_flush_output (s=0x1d68080) at ../../../lib/logwriter.c:198 #6 0x00007fd9295ff986 in iv_run_active_list (active=0x7fff3da27700) at ../../../../../lib/ivykis/lib/iv_main.c:226 #7 0x00007fd9295ffb0d in iv_main () at ../../../../../lib/ivykis/lib/iv_main.c:269 #8 0x00007fd9295d8082 in main_loop_run () at ../../../lib/mainloop.c:722 #9 0x0000000000401a22 in main (argc=1, argv=0x7fff3da27878) at ../../../syslog-ng/main.c:260 (gdb) The particular writer is a program destination (sshguard), and at that point, it's probably not running yet. (For the record, I don't have threaded(yes) in my config at the moment). I'll look into this further, as time permits. -- |8]

1 1

ELSA Google Group
by Martin Holste 01 Dec '11

01 Dec '11

Just a quick note for those of you using or interested in ELSA, I created a Google Group for it at https://groups.google.com/group/enterprise-log-search-and-archive . Still feel free to email me directly for help, but I created the group so that the questions and answers might be visible to others searching on Google for help. Thanks, Martin

1 0

Bug in rewrite logic, 3.3.3
by Dave Rawks 30 Nov '11

30 Nov '11

I've got two different log paths defined that have the same rewrite included in them. The second log path always causes the rewrite to overwrite the incorrect macro value. > ######################## > # Destinations > ######################## > > destination dn_json { > tcp( "logstash" > port(10514) > so_keepalive(yes) > template("$(format_json --pair \@timestamp=\"$ISODATE\" message=\"$MESSAGE\" priority=\"$PRIORITY\" severity=\"$SEVERITY\" facility=\"$FACILITY\" program=\"$PROGRAM\" pid=\"$PID\" date=\"$R_DATE\" host=\"$HOST\" )\n") > ); > }; > > ######################## > #Rewrites > ######################## > > rewrite r_no_pid{ > set("0", value("PID") condition("$PID" =="")); > }; > > ######################## > # Log paths > # order matters if you use "flags(final);" to mark the end of processing in a > # "log" statement. ORDER MATTERS! > ####################### > > > Send all log events on to logstash > log { > source(s_remote); > rewrite(r_no_pid); > destination(dn_json); > }; > > log { > source(s_netloghost); > rewrite(r_no_pid); > destination(df_netloghost); > destination(dn_opennms); > destination(dn_json); > }; If the above configuration is used the events that pass through the second log path have their MESSAGE macro overwritten as "0" instead of their PID macro as is defined in the r_no_pid rewrite rule. The events that pass through the first logpath behave as expected. If I comment out the first logpath then the second behaves as expected. -Dave

2 1

distro URLs for syslog-ng
by Peter Czanik 29 Nov '11

29 Nov '11

Hello, Please take a look at http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… It has now information for many Linux distributions, where up-to-date syslog-ng package information is available, unofficial repositories with latest and/or backported syslog-ng, bugreporting, etc. Let me know, if you have info or links to additional distributions or repositories! Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 147] New: missing share/xsd/patterndb-4.xsd schema definition
by bugzilla＠bugzilla.balabit.com 29 Nov '11

29 Nov '11

https://bugzilla.balabit.com/show_bug.cgi?id=147 Summary: missing share/xsd/patterndb-4.xsd schema definition Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Is OSE 3.3.3 I downloaded the source and installed, but only go schema definitions for database formats 1,2 and 3. The manual for Syslog-ng 3.3.3 has a secion on "What's new in the syslog-ng pattern database format V4" so I can only assume that the version 4 schema should be present. Can you provide one? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 148] New: .classifier. unknown not set when patterndb unable to classify message
by bugzilla＠bugzilla.balabit.com 29 Nov '11

29 Nov '11

https://bugzilla.balabit.com/show_bug.cgi?id=148 Summary: .classifier.unknown not set when patterndb unable to classify message Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 filter f_tag_unknown { tags(".classifier.unknown"); }; Using this filter results in no messages going to a destination, but using filter f_tag_unknown { not tags(".classifier.system"); }; logs all of the unclassified messages. The problem is that I am going to have a lot more than just .classifier.system in my pattern database, so this is not a viable workaround. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 145] New: Comilation on Solaris 10 with gcc 3.4.3 fails
by bugzilla＠bugzilla.balabit.com 29 Nov '11

29 Nov '11

https://bugzilla.balabit.com/show_bug.cgi?id=145 Summary: Comilation on Solaris 10 with gcc 3.4.3 fails Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: andyh(a)nominet.org.uk Type of the Report: bug Estimated Hours: 0.0 Hi, Trying to compile 3.3.3 from source on Solaris 10 with gcc 3.4.4 and get this: libtool: compile: /usr/sfw/bin/gcc -DHAVE_CONFIG_H -I. -I.. -D_GNU_SOURCE -I../lib/include -I../lib/include -I../lib -I../modules/include -Wall -m64 -mcpu=v9 -D_REENTRANT -D_POSIX_C_SOURCE=199506L -D_POSIX_PTHREAD_SEMANTICS -D_XPG4_2 -MT iv_event.lo -MD -MP -MF .deps/iv_event.Tpo -c iv_event.c -fPIC -DPIC -o iv_event.o iv_event.c: In function `iv_event_run_pending_events': iv_event.c:74: error: unrecognizable insn: (insn 80 79 81 2 (set (reg:DI 139) (lo_sum:DI (reg:DI 139) (unspec:DI [ (symbol_ref:DI ("__tls") [flags 0xa] <var_decl fefbb480 __tls>) ] 0))) -1 (nil) (nil)) iv_event.c:74: internal compiler error: in extract_insn, at recog.c:2083 Please submit a full bug report, with preprocessed source if appropriate. See <URL:http://gcc.gnu.org/bugs.html> for instructions. It appears (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21613) that there may be a workaround, but I'm not a developer and can't understand hwo to apply the workaround -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

Problems with custom patterndb
by Clements, Frank 29 Nov '11

29 Nov '11

Hello Everyone! I've been trying to get some custom patterns put together to do some log correlation and I'm having one hell of a time getting a working pattern. I think I need a second, third, forth set of eyes on this ... Any help is appreciated! Message: Nov 25 12:02:27 GENERATED NAT-UDP-C: 192.168.107.132:12260 -> 207.44.101.104:12260 to 48.70.67.223:940 Pattern: @IPvANY:.dict.insideAddr@:@NUMBER:.dict.insidePort@ -> @IPvANY:.dict.outsideAddr@:@NUMBER:.dict.outsidePort@ to @IPvANY:.dict.destAddr@:@NUMBER:.dict.destPort@ I've looked at a few examples from the community patterns, but nothing in this stands out as being "wrong". Thanks - Frank W Clements

2 4