- syslog-ng - lists.balabit.hu

[patch] fix linking issue in 3.2beta1
by Guillaume Rousse 18 Oct '10

18 Oct '10

The following patch fixes some linking issue when using -Wl,--no-undefined flag. -- BOFH excuse #126: it has Intel Inside

2 1

Performance analysis (was Re: UDP packet loss with syslog-ng)
by Lars Kellogg-Stedman 18 Oct '10

18 Oct '10

I've run a series of tests against our log server, using loggen-generated logging rates of 100, 1000, 2000, 4000, 8000, 16000, and 32000 messages/sec, and measured the results for each rate using socket buffer sizes ranging from 128KB to 16M. The results show, essentially, what buffer size you need to meet a target rate of message logging. So I'm happy now, and I'm getting the sort of rates I expect to be able to support. I figured other folks might like the data. I've put the raw data online as well as a graph of the results: http://drop.io/syslog_ng The graph shows so_rcvbuf() size along the X axis, and packet loss along the Y axis.

3 3

Bazsi's blog: Syslog-ng correllation
by Balazs Scheidler 18 Oct '10

18 Oct '10

Syslog-ng correllation I think we’ve reached an important milestone with syslog-ng: log message correllation was added to db-parser(). As you probably know dbparser and its sister project patterndb is able to transform unstructured syslog messages into a normalized format: the human readable string content becomes a set of name-value pairs. The problem is that in a lot of cases messages miss one or two details that would really be needed to understand them and this information usually comes in a followup message. For example: one message in postfix logs contain the sender address and while the recipient information comes in the next message. It is trivial to understand that in reality most cases you want the information in sender,recipient pairs. Another example is sshd, where the authentication failure comes in one and the exact reason for the failure comes in the next. Currently what you can do with syslog-ng is to put the separate messages into two SQL tables and join them at query time. This gets ugly quite fast: increased storage needs, the hassle with managing two tables instead of one and not to mention the increase of the time needed to query the database. Sometimes the sole reason for creating SQL tables in this case is to perform the correllation, otherwise you’d be happier with a CSV file. And that’s what became possible now with the latest git commit of syslog-ng 3.2. The idea is simple: when a patterndb rule matches, you can tell syslog-ng to remember that message by adding it to a correllation state. This state is identified with information extracted from the message making it a unique session identifier. When the next line comes in you can reference the information stored earlier. Basically the correllation state is a list of log messages associated with a session id. To add a new message to this state, you need a store rule: <rule id=”…”> <patterns> <pattern>foo session: @STRING:sessionid@, param: @STRING:param@</pattern> </patterns> <store id=”$sessionid” timeout=”60″/> </rule> The id attribute of the store element specifies a template containing any syslog-ng name-value pairs, probably extracted from the current message itself. When the final information comes in you can use the join attribute of the values tag: <rule id=”…”> <patterns> <pattern>bar session: @STRING:sessionid@</pattern> </patterns> <values join=”$sessionid”> <value name=”param”>${param}@1</value> </values> </rule> here the join attribute specifies the session to look up (which must match in the two messages), and if there’s a match all messages stored in the correllation state becomes available when evaluating the name-value pairs associated with the current message. The key here is the new syntax in the template string “@1″ appended to a name-value pair reference. After the “@” character, you can reference a message in the correllation state by specifying the index backward from the current message. This way @0 is the current message, @1 is the one prior to the current one, @2 is before that and so on. There are more complex ways to use/query the contents of the correllation state, but those will appear in a followup post. Stay tuned!

2 8

UDP packet loss with syslog-ng
by Lars Kellogg-Stedman 17 Oct '10

17 Oct '10

Hello all, I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of memory. Using stock kernel settings (e.g., without adjusting net.core.rmem_default), we're not able to handle much more than 100 messages/second (generated from a remote host using the "loggen" tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000 msg/sec, we see closer to 60% packet loss. Our configuration looks approximately like this (template definitions elided for brevity): options { time_reap(30); mark_freq(10); keep_hostname(yes); use_fqdn(yes); dns_cache(2000); dns_cache_expire(86400); }; source s_network { udp(); tcp(port(514)); }; destination d_syslog { file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages" template(t_daily_log) create_dirs(yes) ); file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY" template(t_host_log) create_dirs(yes) ); }; log { source(s_network); destination(d_syslog); }; I didn't think these message rates were terribly high, so I was surprised at the loss. We've confirmed that the loss is entirely between the kernel and the application -- using wireshark, we've verified that all of the packets are arriving at the host, and using this: awk '{print}' /inet/udp/514/0/0 > out Our packet loss is < 1%. If I raise the rmem settings like this: net.core.rmem_default = 512000 net.core.rmem_max = 1024000 Then it looks like I can support messages rates around 1000 msgs/sec. If I try with 2000 msgs/sec, the loss rates jumps up again (to around 30%). Do these numbers make sense? This is an unloaded server. The only log traffic hitting this system is from my loggen runs. The filesystem is ext3 on top of a hardware RAID5 array. I've tried fiddling with some of the syslog-ng global options (e.g., flush_lines(), log_fetch_limit()), but without having much impact on performance. I would appreciate any help you can send our way. Thanks! -- Lars

6 26

syslog-ng OSE 3.2beta1 has been released (source + binaries)
by Balazs Scheidler 16 Oct '10

16 Oct '10

Dear syslog-ng users, I'm happy to announce that an important milestone towards the new syslog-ng release has been reached. syslog-ng OSE 3.2beta1 has been uploaded to our website. It contains a lot of important new functionality compared to 3.2alpha2 and also some care has been taken to shake out release-critical bugs. As of know I don't know about any Release Critical issues, but I do now that a number of such bugs have been fixed since alpha2. So if you had issues with the alpha release and didn't report the issues, it still is probable that this release fixes them. I think this beta is good enough for everyone to try out, so if you like the new features, I'd appreciate if you could report with either negative or positive feedback to the syslog-ng mailing list. If you haven't tracked 3.2 development so far, I'd recommend to read the 3.2alpha2 and 3.2beta1 announcements for a sumary of new features coming in 3.2. You can find these here: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2alp… http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2bet… This time in addition to the source code, I also release syslog-ng binaries for Linux/i386 and Linux/amd64. The binaries are not thouroughly tested and will probably never be. The canonical release format for syslog-ng is the source code. Anyway, if you feel there are problems with the installer/binaries, feedback is welcome, we're willing to fix them. Here's the NEWS entry for the 3.2beta1 release: 3.2beta1 Mon, 11 Oct 2010 12:25:07 +0200 Changes and new features destined to the syslog-ng 3.2 release are complete, and starting with this release, only bugfixes and minor changes are possible. There's only one exception to this: the correllation framework in db-parser() is still considered experimental and is recommended for early adopters only. This beta has gone through some testing and initial blocker problems were fixed before the release. Right now I'm not aware of any serious issues, but as always, testing is appreciated. New features since 3.2alpha2: * Added support for message correllation in db-parser. See the relevant blog posts for more information: http://bazsi.blogs.balabit.com/2010/10/syslog-ng-correllation-updated/ http://bazsi.blogs.balabit.com/2010/09/syslog-ng-correllation/ * Added "pdbtool patternize", which implements automatic patterndb generation from a sample log file. http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ * Added template functions framework and some initial functions: http://bazsi.blogs.balabit.com/2010/09/introducing-template-functions/ The new functions are: $(echo), $(grep) and $(if) * Added support to process native syslog.conf file using the syslogconf SCL plugin. http://bazsi.blogs.balabit.com/2010/09/syslog-ng-now-supports-the-syslog-co… * Added support for comparison operators in filter expressions, e.g. it is now possible to use "$FACILITY_NUM" < "5". String and numeric operators are also provided, the same way as in perl. * syslog-ng now automatically detects if an incoming message is in RFC3164 or RFC5424 format. This means that the syslog driver can be used to process both. * Added pdbtool validation support, using the "pdbtool test --validate". Requires an installed xmllint program. * pdbtool is now able to merge patterndb XML files recursively in order to make it easy to use the results of the patterndb project. * db-parser() automatically assigns class-specific tags to messages, this means that a message classified "system" will get a ".classifier.system" tag in addition to storing the class in a name-value pair named ${.classifier.class} * It is now possible to use multiple program name patterns for a single ruleset in patterndb. * Added $(ipv4-to-int) template function to convert an IP address to its numeric representation. Bugfixes since 3.2alpha2: * Fixed a possible infinite loop in "pdbtool test" in case program/message was missing from the sample message. * SQL: revert don't require the current CVS version of libdbi * Don't report "this config file version is too old" multiple times. * Underscore and dash are assumed to be equivalent in plugin names. * Various memory leaks were plugged. Removed functions: * Removed the use_time_recvd() global and per-destination option, deprecated since 3.0. Can be substituted with $R_ prefix in macro names. Other changes: * Restructured the source tree in order to make compilations of independent plugins easier and faster. Modules go to modules/ subdirectory, the core lives under lib/ and the main executables go into syslog-ng/ * SCL paths are determined relative to ${datadir} instead of ${prefix} to make distribution packaging easier. * Pass -avoid-version when linking modules. * syslog-ng now requires bison 2.4, this is also checked by the configure script. Credits: I'd like to thank the following people who made this release possible. People in no particular order: * Péter Gyöngyösi for his patternize work * Márton Illés for pdbtool test * Róbert Fekete for his feedback on various functions/ideas * Csaba Major for listening to my stupid ideas :) * Martin Grauel for giving a talk on syslog-ng 3.2 and for his number of useful comments * Bálint Kovács for trying patternize, giving me bugreports and for his patternize related ideas * Péter Czanik for asking distributor maintainers to package syslog-ng 3.2 and to return feedback on the build process * Matthew Hall, Martin Holste, Patrick H. for his patterndb feedback and their support on the mailing that gave me time to work on syslog-ng instead of doing syslog-ng related mailing Hopefully I haven't forgotten anyone. I'd like to open the 3.3 branch of syslog-ng now and I'd appreciate if distributions would start adopting the 3.2 package. It comes with big build changes, so maintainers please expect some more time with this release. I'm trying to fix bugs fast as they appear to get a stable 3.2 by the end of this year. Happy logging! -- Bazsi

1 0

anushri kannu wants to chat
by anushri kannu 16 Oct '10

16 Oct '10

----------------------------------------------------------------------- anushri kannu wants to stay in better touch using some of Google's coolest new products. If you already have Gmail or Google Talk, visit: http://mail.google.com/mail/b-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… You'll need to click this link to be able to chat with anushri kannu. To get Gmail - a free email account from Google with over 2,800 megabytes of storage - and chat with anushri kannu, visit: http://mail.google.com/mail/a-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… Gmail offers: - Instant messaging right inside Gmail - Powerful spam protection - Built-in search for finding your messages and a helpful way of organizing emails into "conversations" - No pop-up ads or untargeted banners - just text ads and related information that are relevant to the content of your messages All this, and its yours for free. But wait, there's more! By opening a Gmail account, you also get access to Google Talk, Google's instant messaging service: http://www.google.com/talk/ Google Talk offers: - Web-based chat that you can use anywhere, without a download - A contact list that's synchronized with your Gmail account - Free, high quality PC-to-PC voice calls when you download the Google Talk client We're working hard to add new features and make improvements, so we might also ask for your comments and suggestions periodically. We appreciate your help in making our products even better! Thanks, The Google Team To learn more about Gmail and Google Talk, visit: http://mail.google.com/mail/help/about.html http://www.google.com/talk/about.html (If clicking the URLs in this message does not work, copy and paste them into the address bar of your browser).

4 4

pattern naming problems
by Peter Czanik 16 Oct '10

16 Oct '10

Hello, I ran into an interesting situation. Yesterday I created patterns for uw-imapd. Today I looked at some log files from courier imapd and found, that this imap implementation also uses "imapd" for $PROGRAM (and "imapd-ssl" for port 993 connections). "imapd" and "imapd-ssl" messages looked the same. So, here is a list of questions: - how to handle at file/ruleset/etc level when two applications have the same $PROGRAM - how to handle, when the same application uses different $PROGRAM in different situations Also, I took a look at postfix logs, and they look rather ugly: Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: connect from czp.localnet[192.168.2.179] Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: 8434B41C30: client=czp.localnet[192.168.2.179], sasl_method=PLAIN, sasl_username=czanik Oct 13 21:35:29 ubuntu postfix/cleanup[3946]: 8434B41C30: message-id=<4CB609F2.8(a)blabla.com> Oct 13 21:35:29 ubuntu postfix/qmgr[3570]: 8434B41C30: from=<czanik(a)blabla.com>, size=619, nrcpt=1 (queue active) Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: disconnect from czp.localnet[192.168.2.179] Oct 13 21:35:50 ubuntu postfix/smtp[3947]: connect to targetmachine[1.2.3.4]:25: Connection timed out Oct 13 21:35:50 ubuntu postfix/smtp[3947]: 8434B41C30: to=<czanik@targetmachine>, relay=none, delay=21, delays=0.02/0.01/21/0, dsn=4.4.1, status=deferred (connect to targetmachine[1.2.3.4]:25: Connection timed out) This is an SMTP authentication, and then the e-mail is tried to be delivered to targetmachine. There are many different names as $PROGRAM, $PID also has many different values. But "8434B41C30" could easily be used as session identifier for all of this. The question is the same: how should pattern name be handled? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 4

repeated lines
by Peter Czanik 16 Oct '10

16 Oct '10

Hello, I'm just preparing patterns for uw-imapd. Problem: Oct 12 16:54:12 ubuntu imapd[10555]: connect from 192.168.2.179 (192.168.2.179) Oct 12 16:54:12 ubuntu imapd[10555]: imap service init from 192.168.2.179 Oct 12 16:54:17 ubuntu imapd[10555]: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] Oct 12 16:54:20 ubuntu imapd[10555]: AUTHENTICATE PLAIN failure host=czp.localnet [192.168.2.179] Oct 12 16:54:20 ubuntu imapd[10555]: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] Oct 12 16:54:48 ubuntu imapd[10555]: Unexpected client disconnect, while reading line user=asdf host=czp.localnet [192.168.2.179] As you can see, the line containing most information about the login failure is repeated twice: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] How can I make sure, that only one name value pair is generated from this event? Bazsi: we talked about the importance of time on Monday: messages for a single event arrived in an 8 seconds time span... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

Diagnosing patterndb load errors
by Matthew Hall 15 Oct '10

15 Oct '10

I am getting the following sequence of load errors for my patterndbs. Duplicates eliminated, timestamps zeroed out, host info anonymized. Has anybody else gotten these and learned how to fix them? I'm having a hard time figuring out what 'type', 'name', and 'value' mean, to track them down in the XML files. Thanks, Matthew Hall. Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Name-value pairs cannot have a zero-length name; value='' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='Neighbor Down: Dead timer expired', value='19' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='36' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='39' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='40' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='41' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='44' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=', domain = <not specified>', value='8' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='12' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='13' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='20' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='21' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='22' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='23' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='24' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='27' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='29' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='30' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='31' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='33' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='35' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='4', name='None', value='38' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='vpn-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='load-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='firewall-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='group', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='. Please check authentication server radius/ldap/tacacs logs.', value='50' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='wireless-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Termination requested via signal, terminating; Jan 01 00:00:00 s_internal@host syslog-ng[23005]: syslog-ng shutting down; version='3.1.1' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='wireless-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='vpn-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='load-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='firewall-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: syslog-ng starting up; version='3.1.1'

3 3

Bazsi's blog: syslog-ng correllation updated
by Balazs Scheidler 15 Oct '10

15 Oct '10

syslog-ng correllation updated I’m trying to push syslog-ng 3.2beta1 out on the door, but as I was writing the NEWS entry I had to realize that the latest state of the patterndb correllation functions are undocumented so far. So here goes a blog post which tries to summarize how it works, so that I can include it in the NEWS entry :) My previous post on the topic used a syntax that included explicit “store” and “join” elements, but I’ve decided to drop those, as they stood in the way for some more juicy functionality. What remained is that the correllation is still focused around a “correllation state”, or as I’ve called internally a “context”. A context consists of a series of log messages related to each other in some way. As new messages come in, they may be associated with a context (e.g. added to the context). Also, when an incoming message is identified it can trigger some actions to be performed. And these actions can use all the information that was stored previously in the context. Let’s see how this work out with concrete examples: each rule in the patterndb has a “context-id” attribute telling db-parser() which context the given message should be associated with. This example covers an SSH login message: <rule id=”…” context-id=”$HOST:$PROGRAM:$PID” context-timeout=”86400″ context-scope=”global”> <patterns> <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern> </patterns> … </rule> Since multiple rules can reference the same context, multiple different kind of messages may be added into the same context as a result. E.g. the logout event looks like this: <rule context-id=”$HOST:$PROGRAM:$PID” context-timeout=”0″ context-scope=”global”> <patterns> <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:usracct.username:@</pattern> </patterns> … </rule> As you can see a “session” is identified using the triplet ($HOST, $PROGRAM, $PID) and these two rules correllate the login/logout events into the same context, which means that you can create a derived event that contains information from both of them. Please note that it is fairly common that messages only need to be correllated if they originate from the same host: e.g. this SSH login message needs only to be correllated to its logout counterpart if they both originate from the same host. In the previous example this was achieved using explicit macros in the context-id attribute, however since this is quite often the case, this was worked into a function of its own right: each rule can have a context-scope attribute: <rule id=”…” context-scope=”process” context-id=”ssh-login” context-timeout=”5″> </rule> The context-scope tells syslog-ng which messages need to be considered when looking for correllations: * process: only consider messages that have matching $HOST, $PROGRAM and $PID values * program: only consider messages that have matching $HOST and $PROGRAM values * host: only consider messages that have matching $HOST values * global: any kind of message is fine The default is to use “process”, which means that if it is true that the same process is emitting all the messages that you want to correllate, then you don’t need to use a variable part in your context-id attribute. But it is also important to know that it is way faster to specify the scope this way than it’d be to add all relevant macros to your context-id attribute. So far so good, we have all the functions that we used to have with the previous versions of the functionality. But I mentioned something about “actions” to be performed. Until now a patterndb rule basically only identified the incoming message, possibly associated tags and name-value pairs, but didn’t perform anything else. This is being changed: one or more actions can be associated with a patterndb rule in order to make it possible to react to more complex situations. Here’s an example action: <action rate=”1/86400″> <message> <values> <value name=”MESSAGE”>a patterndb rule matched</value> </values> </message> </action> Right now the only real response to a message is to generate another message, but this allows us to do a couple of powerful transformations, especially with the following options that you can specify for an action tag: * condition: specifies a syslog-ng filter expression that needs to be matched in order to really perform the action. It is evaluated on the current message that matched the rule. * rate: <num>/<period> specifies how much messages are to be generated (num), in the specified time period (period). Excess messages are dropped. For example: “1/60″ allows 1 message / minute. Rates apply to the given scope for the given rule/action. E.g. context-scope=”host”, rate=”1/60″ means that one message gets generated for _each host_ per minute. * trigger: specifies when to execute the action, there are two possible triggers right now: * match: execute immediately once the rule matches * timeout: execute when the correllation timer expires I’d like to highlight two things: * it is possible to react to the expiration of a correllation timer (e.g. trigger=”timeout”) * it is possible to generate a message only in case a given condition is met (e.g. “$PID” == “”) Right now new messages are posted to the internal() driver. This is not the way I wanted it to be, but doing my original plan would require an enormous refactorization of the code, and it is too late for that too happen. My original idea was to let the db-parser() emit multiple messages, but since the current state of affairs in syslog-ng assumes that only sources generate messages, that needs a lot of work. But hey, we need something to do for syslog-ng 3.3, right?

5 10