December 2009 - syslog-ng - lists.balabit.hu

syslog-ng 3.1 administrator guide draft
by Robert Fekete 09 Dec '09

09 Dec '09

Dear syslog-ng users, A draft version of the 3.1 administrator guide is available at http://people.balabit.hu/frobert/syslog-ng-v3.1admin-guide-en-1ed-draft.pdf Its mainly interesting if you are giving pattern databases a try: Section 8.6.2.3. Creating pattern databases now describes the latest V3 format, and in Appendix 1 you will find a man page for pdbtool. Also, some experimental/unsupported/unchecked/for-testing-only pattern databases are now available at http://www.balabit.com/downloads/files/patterndb-snapshot/, but Marci will write in more detail about that. Have fun! Robert

1 0

syslog-ng OSE 3.0.5 released
by Balazs Scheidler 09 Dec '09

09 Dec '09

Dear syslog-ng users, I've uploaded syslog-ng OSE 3.0.5 to our webservers recently which has some quite important bugfixes. I'd recommend any serious deployment of syslog-ng 3.x to upgrade to this release. The results of the last four months were accumulated into this release. Some statistics for those who like them: 37 files changed, 885 insertions(+), 468 deletions(-) In this timespan, altogether 36 patches were produced by the following people: Anthony Lineham (1) Balazs Scheidler (25) Marton Illes (2) Tamas Pal (3) Tevesz Andras (4) Viktor Tusa (1) Some of the bugreporters and testers also deserve credit, who were: Tamas Monos Sandor Geller Imre Lazar Arkadiusz Miśkiewicz Elan Ruusamäe Thank you for your contribution. Also please note that a new feature release of syslog-ng was published in recent weeks, numbered version 3.1beta1. That beta version has improvements in various areas you might want to check out. We'll only include bugfixes in syslog-ng 3.0.x in the future. The list of changes: 3.0.5 Thu, 03 Dec 2009 18:02:56 +0100 Bugfixes: * Fixed a possible hang that caused syslog-ng to completely stop processing messages, in case SQL destination is enabled in configure. The bug does not need a working SQL destination. * Fixed a possible 100% CPU usage in case an udp destination went down. * Fixed host_override/program_override options that were broken in 3.0.4. * Fixed the SQL destination to avoid dropping the output queue in case a HUP was received. * Fixed a possible segmentation fault when parsing syntactically incorrect patterndb files. * Fixed ESTRING parser to include the ending quote character in the result. * Report an error in case there's no terminating brace in template macro references, e.g. no closing brace in ${macro} format. * Fixed FreeBSD init script to add $prefix/sbin to the PATH environment variable. This problem caused syslog-ng not to start up on boot. * Fixed FreeBSD /dev/klog message reading, follow_freq() needs to default to zero in this case (just like Linux /proc/kmsg). * Fixed a possible segfault in SIGHUP processing if an included file contains a syntax error. * RFC5424 over UDP erroneously required frame length, just like TCP, this is now fixed. * Fixed HP-UX 11iv2 compilation issue related to pread/pwrite compile errors. * The flow-control flag was erroneously enabled if any of the log statements contained either "final" or "fallback". * Fixed rewrite processing, in some cases a rewritten value could be propagated to wrong branches in the log processing tree. Other changes: * Added support for Cisco ASA timestamps. * Added version checks for PCRE in the configure script. * Integrate with system LSB init scripts a bit better, with 3.0.5 the init script status reports should nicely integrate with the host system. Happy logging, -- Bazsi

1 0

Lock up problems when using unix-dgram("/dev/log") type on
by Lance Laursen 09 Dec '09

09 Dec '09

Hello, I'm having problems with machines eventually hanging on all processes that write to /dev/log when using unix-dgram("/dev/log") with Syslog-NG 3.0.4. The servers run fine for a while and hum along as expected. Unfortunately the success does not last, with various programs completely hanging after an undetermined time. Having an existing root shell as this happens allows me to kill syslog-ng, freeing up all locks. Repro'ing this is...well, annoying. I have 300+ servers running this build of syslog-ng fine, all using unix-stream(). The 4 servers that are locking up are the only ones I have running unix-dgram(). Completely fresh ubuntu 8.04 installs with syslog-ng 3.0.4, identical to all other boxes aside from the one syslog-ng option. I've got strace output that is hanging after programs try to write to /dev/log as well. I'm currently doing a repro by running "while true ; do logger -p local0.info ...longest_message_possible... ; sleep 1s ; done" in non-exact science and have managed to pile things up after just over 120 messages, or two minutes. I can still hop around as root, but all programs that try to write to /dev/log pile up. The pile up seems to be log-size/throughput based, not time-based after some rudimentary tests - though it could be something random that is triggering it while my crappy tests are running. My next test plans to have small log messages in very rapid succession. I'm running: # uname -a Linux tny0032 2.6.24-24-generic #1 SMP Tue Jul 7 19:10:36 UTC 2009 x86_64 GNU/Linux # cat /etc/debian_version lenny/sid (ubuntu 8.04) Here's my source definition: # all known message sources source s_all { internal(); unix-dgram("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; Here's some strace output that locks after trying to write to /dev/log: # strace su - lance ... ... stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0 socket(PF_FILE, SOCK_DGRAM, 0) = 3 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 connect(3, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0 sendto(3, "<85>Dec 4 02:12:43 su[7086]: pa"..., 148, MSG_NOSIGNAL, NULL, 0 # strace logger -p local0.info lalala produces the same lock-point as above. I thought dgram should be connectionless? I'm not sure how syslog-ng could be locking up resources. Has anyone seen this before? I will continue looking for a better repro case, if anyone has any ideas though shout. I am using unix-dgram solely because it does not break to a new log entry on NewLines. I was encountering a problem where, using unix-stream, lighttpd's multi-line log output was getting broken up into multiple syslog lines. This would have been fine, except when the new line is broken out and made into new log entries, the $hostname and $program fields get stripped out, leaving me with just $date $msg. This basically negated the ability to filter and relay logs effectively. I can elaborate further here if requested, but making unix-stream behave the same as unix-dgram with regards to multi-line log messages would solve all my problems. Thanks, -Lance

3 3

Re: [syslog-ng] Macro expansion in program destination
by chris packham 07 Dec '09

07 Dec '09

> No, this is *not* what syslog-ng does. Syslog-ng does *not* pass the macro value > for $S_DAY do the command line, and thus the 30 does *not* appear at all. I think a barrier to implementing this kind of functionality is that the program is started long before any log messages are received so there is nothing intelligent that the program macro could be expanded to. Starting the program when the first log message is received is a bad idea because 1) there may be an overhead in starting any given program which may mean log messages are lost and 2) if there is anything wrong with your program you won't know until the log message is received (instead of getting some indication when syslog-ng is started/sighup'd) But... I just had a thought on how this could be implemented (in a rather hack-ish manner) assuming the macro expansion _could_ be applied to the program destination (which I realize is not the case right now). One handy behaviour of syslog-ng is that if the program() exits it is restarted (I'm going on memory here so I reserve the right to be completely wrong). Normally program destinations _shouldn't_ exit, but if that program were to exit when syslog-ng comes to restart the program could perform the macro expansion again. I'd still favour and alternative (like parsing in the destination program) but if its an itch you feel like scratching grab the code and go for it. Also just another random thought for a Monday (to be taken with symptoms of caffeine withdrawal in mind). You could perform some trickery with named pipes (see mkfifo(1)). Lets say you wanted program x to receive log messages with some knowledge of the day without needing to do any parsing. You could setup 31 named pipes called /tmp/pipe-1 /tmp/pipe-2.... /tmp/pipe-31 then you could ask syslog-ng to write to a file destination d_pipe ("/tmp/pipe-$S_DAY") external to all this you could start 31 instances of program x each reading from one of the named pipes.

2 1

pipe causing lockup?
by Tim Rupp 07 Dec '09

07 Dec '09

Hi list, I'm hoping someone can explain this to me because it's taken several days of strace'ing and debugging for me to draw this conclusion. I have a source that right now is receiving a relay of syslog from another syslog-ng server, version 1.6.11. The new syslog-ng server is 3.0.4 using the RHEL 5 amd64 rpm. The new server has the following source to receive from the old server source s_net { udp( ip(0.0.0.0) port(514) ); }; I was seeing similar problems as described by Mike in this discussion https://lists.balabit.hu/pipermail/syslog-ng/2009-September/013371.html I strace'd the syslog-ng process, would wait for it to die, and then attempted to see if there were any patterns. In each strace syslog-ng appeared to hand right after processing a write to a fifo; the relevant strace output is below. 25461 write(340, "Dec 3 22:32:32 cmsstor52 sshd[28272]: Authorized to root, krb5 principal host/cmssrv28.fnal.gov(a)FNAL.GOV (krb5_kuserok)\n", 121) = 121 25461 poll([{fd=39, events=POLLOUT}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd= 9, events=POLLIN}], 7, 29155) = 1 ([{fd=9, revents=POLLIN}]) 25461 recvfrom(9, "<14>Dec 3 22:32:30 fndhcp1 dhcp_checkd[1679]: [ID 702911 user.info] -- DHCP Watchdog reports: All DHCP registration processes are runni ng --\n", 8192, 0, {sa_family=AF_INET, sin_port=htons(46911), sin_addr=inet_addr("131.225.110.121")}, [16]) = 144 25461 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0 25461 poll([{fd=39, events=POLLOUT}], 1, 0) = 0 (Timeout) 25461 lseek(14, 0, SEEK_END) = 1212423178 25461 write(14, "Dec 3 22:32:30 fndhcp1 dhcp_checkd[1679]: [ID 702911 user.info] -- DHCP Watchdog reports: All DHCP registration processes are running --\ n", 140) = 140 25461 lseek(38, 0, SEEK_END) = 641655 25461 write(38, "Dec 3 22:32:30 fndhcp1 dhcp_checkd[1679]: [ID 702911 user.info] -- DHCP Watchdog reports: All DHCP registration processes are running --\ n", 140) = 140 25461 lseek(40, 0, SEEK_END) = 641655 25461 write(40, "Dec 3 22:32:30 fndhcp1 dhcp_checkd[1679]: [ID 702911 user.info] -- DHCP Watchdog reports: All DHCP registration processes are running --\ n", 140) = 140 25461 lseek(41, 0, SEEK_END) = 641655 25461 write(41, "Dec 3 22:32:30 fndhcp1 dhcp_checkd[1679]: [ID 702911 user.info] -- DHCP Watchdog reports: All DHCP registration processes are running --\ n", 140) = 140 fd 9 is the UDP socket. Right after those messages, it would do a bunch of close() calls ... 25461 close(203) = 0 25461 close(150) = 0 25461 close(565) = 0 25461 close(637) = 0 25461 close(517) = 0 ... followed by a poll that, this time, didn't include the UDP socket 25461 poll([{fd=39, events=POLLOUT}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=8, events=POLLIN}], 6, 0) = 0 (Timeout) 25461 poll([{fd=39, events=POLLOUT}], 1, 0) = 0 (Timeout) 25461 lseek(11, 0, SEEK_END) = 107767627 25461 write(11, "Dec 3 22:33:01 clogged syslog-ng[25461]: Reaping unused destination files; template='/logging/syslog-ng/$HOST/messages'\n", 121) = 121 25461 lseek(12, 0, SEEK_END) = 105497311 I don't have flow-control enabled in my config. After several days of head scratching and reading, I narrowed it down to having something to do with my DHCP destinations/filters/logs I have a pipe() destination that points to a fifo destination d_cstdb { pipe("/opt/dhcp_syslog_cst_db"); }; a filter filter f_dhcp { facility(local4,user) and host("fndhcp1*"); }; and the corresponding log statement log { source(s_net); filter(f_dhcp); destination(d_cstdb); }; I've concluded that the problem is caused by the pipe/fifo. On the old server I have something reading from the fifo. On the new server I hadn't stood that process up yet. syslog-ng would start just fine, would even write several entries to that fifo, and then would reach a point where things would hang. It would stop reading from the UDP socket, and just log things like destinations expiring. I looked online at the documentation for FIFOs and linux and noted that 3.0.4 source appears to specify that fifo's be opened non-blocking. So forgive the silly question, but if the fifo is opened non-blocking, why is syslog-ng appearing to block when writing to a fifo that isn't being read from? Also, why would the UDP socket file descriptor be removed from the poll list? An lsof of the syslog-ng process clearly showed that it was still bound to *:syslog Is this is bug or normal operation? Maybe I don't understand linux's behavior in regards to fifos. Thanks for any clarification, Tim

2 4

Question about verisign certs
by Jimmy McDonald 06 Dec '09

06 Dec '09

I have syslog-ng installed and configured for mutual authentication. When the client was using a self signed cert it worked. The client now has a verisign cert. Imported the CA and an intermediate cert and the public cert so the chain is complete but the error on authentication says: invalid CA certificate, depth=2 I have tried putting the pub cert in cert.d with the ca and intermediate in ca.d. I also tried putting the intermediate in cert.d I made a hash for the ca and the intermediate. I'm not really sure what else to try. Any help would be greatly appreciated. Thanks, Jimmy Sent from my iPhone... So expect typos.

2 1

Expected performance
by Christopher Bland 04 Dec '09

04 Dec '09

Hey guys, I know it's hard to compare apples to apples when you start talking about performance and hardware but I would appreciate some feedback. I am currently have a 32Bit HP Proliant DL380 with 2 2.8Ghz cpus and 4G of memory running Fedora 10. At present I have 150 hosts generating between 10-15G worth of logs per day. I dump all of my logs to a Mysql database so that I can use php-syslog-ng. The OS is build on a 1.2T raid 5 disk array. To cut down on I/O I have the database writing to a ext3 filesystem while the rest of the box uses LVM. I have mounted the database partition with noatime and implemented all of the mysqltuner suggestions. My box is performing slow like molasses. My config looks like this: options { sync (0); time_reopen (10); log_fifo_size (1000); chain_hostnames(no); long_hostnames (off); keep_hostname(no); use_dns (yes); dns_cache(yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); }; source s_everything { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); udp(); }; destination d_database { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination d_asa { file("/var/log/syslog-ng/network/asa/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" owner(root) group(staff) perm(0650) dir_perm(0750) create_dirs(yes)); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; filter f_asa { host("asain-temp1"); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; log { source(s_everything); filter(f_asa); destination(d_asa); }; log { source(s_everything); destination(d_database); }; my.cnf looks like this: [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql old_passwords=1 skip-innodb skip-bdb skip-name-resolve table_cache = 128 tmp_table_size = 256M max_heap_table_size = 256M query_cache_size = 128M query_cache_limit = 4M read_rnd_buffer_size = 1M thread_cache_size = 8 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [ndbd] connect-string="nodeid=2;host=localhost:1186" [ndb_mgm] connect-string="host=localhost:1186" My load looks like this while inserting data: Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie Cpu(s): 20.4%us, 2.3%sy, 0.1%ni, 75.9%id, 0.7%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3634412k total, 3507692k used, 126720k free, 20224k buffers Swap: 8388600k total, 64k used, 8388536k free, 3246292k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6868 mysql 20 0 165m 20m 5088 S 91.9 0.6 1488:41 mysqld 7679 root 20 0 4140 1932 948 S 15.6 0.1 158:18.24 syslog-ng 7705 root 20 0 8456 1756 1384 S 5.9 0.0 79:18.84 mysql 21296 root 20 0 2556 996 740 R 2.0 0.0 0:00.01 top 1 root 20 0 2012 780 568 S 0.0 0.0 0:01.62 init 2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/0 4 root 15 -5 0 0 0 S 0.0 0.0 0:17.46 ksoftirqd/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 6 root RT -5 0 0 0 S 0.0 0.0 0:00.18 migration/1 7 root 15 -5 0 0 0 S 0.0 0.0 0:26.76 ksoftirqd/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 9 root RT -5 0 0 0 S 0.0 0.0 0:00.13 migration/2 10 root 15 -5 0 0 0 S 0.0 0.0 0:22.02 ksoftirqd/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 12 root RT -5 0 0 0 S 0.0 0.0 0:00.11 migration/3 13 root 15 -5 0 0 0 S 0.0 0.0 0:11.79 ksoftirqd/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 15 root 15 -5 0 0 0 S 0.0 0.0 0:04.69 events/0 16 root 15 -5 0 0 0 S 0.0 0.0 0:02.86 events/1 17 root 15 -5 0 0 0 S 0.0 0.0 0:02.80 events/2 18 root 15 -5 0 0 0 S 0.0 0.0 0:02.91 events/3 19 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 95 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0 96 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1 97 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/2 98 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/3 100 root 15 -5 0 0 0 S 0.0 0.0 0:00.20 kblockd/0 101 root 15 -5 0 0 0 S 0.0 0.0 0:00.14 kblockd/1 102 root 15 -5 0 0 0 S 0.0 0.0 0:00.33 kblockd/2 103 root 15 -5 0 0 0 S 0.0 0.0 0:12.47 kblockd/3 105 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid 106 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify The box gets almost unusable when I do a query to retrieve data from the database? Again, I would appreciate any thoughts or suggestions . -Chris Bland

7 8

[patch] syslog-ng 3.0.4 init script doesn't remove stale PID file on startup
by Gordon McAllister 04 Dec '09

04 Dec '09

Hello, I'm running the syslog-ng 3.0.4 RPM on CentOS 5.4 and came across a problem where syslog-ng would not start properly after an unclean shutdown, since the old PID file was still in place. Attached is a patch to the init script supplied with the RPM that attempts to correct this. Regards, ---Gordon

3 3

rewriteable fields
by chris packham 03 Dec '09

03 Dec '09

Hi List, Just looking for a bit of clarification on exactly what fields I can rewrite in a message. Basically I've got an annoying debug message coming out of a program and while I could filter it off to a separate file, what I really want to do is drop its level to debug (from warning) and let my existing filters take care of it. The docs say "Setting a field can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers" so modifying LEVEL is not listed but its not explicitly excluded either. I tried "rewrite r_norm {set("1", value("LEVEL")); };" but that didn't work. However "rewrite r_norm {set("FOO", value("MESSAGE")); };" and "rewrite r_norm {set("BAR", value("PROGRAM")); };" work as expected. Are there limitations as to which parts of a message I can rewrite? Are there some fuller examples hanging around? Thanks, Chris

3 2

syslog-ng OSE 3.1beta1 released
by Balazs Scheidler 03 Dec '09

03 Dec '09

Dear syslog-ng users, I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways: 1) of course it has new features, see below for the most interesting bits 2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement. For those who hurry, here's a link for the source code: https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1be… And here are the binaries for Linux/FreeBSD systems: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… Select the Downloads tab, and in the Version selector select 3.1beta1. What is new in syslog-ng OSE 3.1 -------------------------------- * Support for patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT. Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog. Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. Patterndb in general and the v1 format database is described in the syslog-ng manual at: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD The changes in the patterndb format as they evolved were described in Marton Illes's blog at http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html But see the other related posts as well. Old patterndb databases can be converted to the new format by putting them in a directory and using the pdbtool utility using the command: $ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml -D /opt/syslog-ng/etc/patterns.d Assuming the installation prefix of syslog-ng is /opt/syslog-ng Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location: https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/ You can convert these db files using pdbtool as described above. Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post. * Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message. See the manpage and Marci's post: http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html * Support for message tags: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering. http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html * Added support for rewriting structured data. Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message. Now all these became possible by using the same syntax that didn't work earlier, e.g. rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); }; * Macros and name-value pairs got a little tighter integration, in filters where syslog-ng 3.0 was limited to only use name-value pairs, with 3.1 you can also use macros. The following now works: match("<regexp>" value("R_DATE")); syslog-ng is now warning you in case you are using '$' prefix in the value syntax. * Enhanced dynamic name-value performance by a factor of three. The summary says it all, the performance dynamic name-value pairs that the various parsers produce got faster, thus the performance penalty of structuring the incoming messages got smaller. * Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These are patterndb parsers to make it easier to describe log messages. * Added non-standard and non-portable facility codes (range 10-15), decouple syslog-ng facility name information from the system used to compile syslog-ng on. Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. Any feedback, success/failure reports is more than appreciated. -- Bazsi

2 1