October 2010 - syslog-ng - lists.balabit.hu

pattern matching on xxx#
by Bill Anderson 18 Oct '10

18 Oct '10

I have hostnames of the format xxxx# such as host1, hostb1, hostc1. I need to split that into two fields such as (host,1). Unfortunately, since @@ escapes the @ and STRING and it's followers ALSO match digits, I've not found the obvious means to get that out. Conceptually something like @LETTER:host.name@@NUMBER:host.id@ woudl do it, save that LETTER doesn't exist and @@ escapes. The end goal is as follows (pseudo-code): I need to have a destination for each (HOST). For example all files from hosta## go to /var/log/hosta/ and entries for hostb## go to /var/log/hostb/ I suppose I *might* be able to do a rewrite to add say, a hyphen, and then use csv-parser, but we're talking some heavy traffic and I suspect that doing rewrites on that much traffic would be a performance killer. I'm open to suggestions (that don't involve changing server names, preferably ;) ) as to how to accomplish this. Cheers, Bill

4 10

[Bug 95] New: Missing capabilities support for unix-stream() source
by bugzilla＠bugzilla.balabit.com 18 Oct '10

18 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=95 Summary: Missing capabilities support for unix-stream() source Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: --- Estimated Hours: 0.0 Operations on unix-stream() sources need capabilities: - CAP_CHOWN - needed if owner() or group() are in use - CAP_FOWNER - to force chmod() for sockets with owner != root - CAP_DAC_OVERRIDE - force changes if some parent dir has 000 perm, ie vservers Attached patch fixes all these problems. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[patch] fix linking issue in 3.2beta1
by Guillaume Rousse 18 Oct '10

18 Oct '10

The following patch fixes some linking issue when using -Wl,--no-undefined flag. -- BOFH excuse #126: it has Intel Inside

2 1

Performance analysis (was Re: UDP packet loss with syslog-ng)
by Lars Kellogg-Stedman 18 Oct '10

18 Oct '10

I've run a series of tests against our log server, using loggen-generated logging rates of 100, 1000, 2000, 4000, 8000, 16000, and 32000 messages/sec, and measured the results for each rate using socket buffer sizes ranging from 128KB to 16M. The results show, essentially, what buffer size you need to meet a target rate of message logging. So I'm happy now, and I'm getting the sort of rates I expect to be able to support. I figured other folks might like the data. I've put the raw data online as well as a graph of the results: http://drop.io/syslog_ng The graph shows so_rcvbuf() size along the X axis, and packet loss along the Y axis.

3 3

Bazsi's blog: Syslog-ng correllation
by Balazs Scheidler 18 Oct '10

18 Oct '10

Syslog-ng correllation I think we’ve reached an important milestone with syslog-ng: log message correllation was added to db-parser(). As you probably know dbparser and its sister project patterndb is able to transform unstructured syslog messages into a normalized format: the human readable string content becomes a set of name-value pairs. The problem is that in a lot of cases messages miss one or two details that would really be needed to understand them and this information usually comes in a followup message. For example: one message in postfix logs contain the sender address and while the recipient information comes in the next message. It is trivial to understand that in reality most cases you want the information in sender,recipient pairs. Another example is sshd, where the authentication failure comes in one and the exact reason for the failure comes in the next. Currently what you can do with syslog-ng is to put the separate messages into two SQL tables and join them at query time. This gets ugly quite fast: increased storage needs, the hassle with managing two tables instead of one and not to mention the increase of the time needed to query the database. Sometimes the sole reason for creating SQL tables in this case is to perform the correllation, otherwise you’d be happier with a CSV file. And that’s what became possible now with the latest git commit of syslog-ng 3.2. The idea is simple: when a patterndb rule matches, you can tell syslog-ng to remember that message by adding it to a correllation state. This state is identified with information extracted from the message making it a unique session identifier. When the next line comes in you can reference the information stored earlier. Basically the correllation state is a list of log messages associated with a session id. To add a new message to this state, you need a store rule: <rule id=”…”> <patterns> <pattern>foo session: @STRING:sessionid@, param: @STRING:param@</pattern> </patterns> <store id=”$sessionid” timeout=”60″/> </rule> The id attribute of the store element specifies a template containing any syslog-ng name-value pairs, probably extracted from the current message itself. When the final information comes in you can use the join attribute of the values tag: <rule id=”…”> <patterns> <pattern>bar session: @STRING:sessionid@</pattern> </patterns> <values join=”$sessionid”> <value name=”param”>${param}@1</value> </values> </rule> here the join attribute specifies the session to look up (which must match in the two messages), and if there’s a match all messages stored in the correllation state becomes available when evaluating the name-value pairs associated with the current message. The key here is the new syntax in the template string “@1″ appended to a name-value pair reference. After the “@” character, you can reference a message in the correllation state by specifying the index backward from the current message. This way @0 is the current message, @1 is the one prior to the current one, @2 is before that and so on. There are more complex ways to use/query the contents of the correllation state, but those will appear in a followup post. Stay tuned!

2 8

UDP packet loss with syslog-ng
by Lars Kellogg-Stedman 17 Oct '10

17 Oct '10

Hello all, I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of memory. Using stock kernel settings (e.g., without adjusting net.core.rmem_default), we're not able to handle much more than 100 messages/second (generated from a remote host using the "loggen" tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000 msg/sec, we see closer to 60% packet loss. Our configuration looks approximately like this (template definitions elided for brevity): options { time_reap(30); mark_freq(10); keep_hostname(yes); use_fqdn(yes); dns_cache(2000); dns_cache_expire(86400); }; source s_network { udp(); tcp(port(514)); }; destination d_syslog { file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages" template(t_daily_log) create_dirs(yes) ); file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY" template(t_host_log) create_dirs(yes) ); }; log { source(s_network); destination(d_syslog); }; I didn't think these message rates were terribly high, so I was surprised at the loss. We've confirmed that the loss is entirely between the kernel and the application -- using wireshark, we've verified that all of the packets are arriving at the host, and using this: awk '{print}' /inet/udp/514/0/0 > out Our packet loss is < 1%. If I raise the rmem settings like this: net.core.rmem_default = 512000 net.core.rmem_max = 1024000 Then it looks like I can support messages rates around 1000 msgs/sec. If I try with 2000 msgs/sec, the loss rates jumps up again (to around 30%). Do these numbers make sense? This is an unloaded server. The only log traffic hitting this system is from my loggen runs. The filesystem is ext3 on top of a hardware RAID5 array. I've tried fiddling with some of the syslog-ng global options (e.g., flush_lines(), log_fetch_limit()), but without having much impact on performance. I would appreciate any help you can send our way. Thanks! -- Lars

6 26

syslog-ng OSE 3.2beta1 has been released (source + binaries)
by Balazs Scheidler 16 Oct '10

16 Oct '10

Dear syslog-ng users, I'm happy to announce that an important milestone towards the new syslog-ng release has been reached. syslog-ng OSE 3.2beta1 has been uploaded to our website. It contains a lot of important new functionality compared to 3.2alpha2 and also some care has been taken to shake out release-critical bugs. As of know I don't know about any Release Critical issues, but I do now that a number of such bugs have been fixed since alpha2. So if you had issues with the alpha release and didn't report the issues, it still is probable that this release fixes them. I think this beta is good enough for everyone to try out, so if you like the new features, I'd appreciate if you could report with either negative or positive feedback to the syslog-ng mailing list. If you haven't tracked 3.2 development so far, I'd recommend to read the 3.2alpha2 and 3.2beta1 announcements for a sumary of new features coming in 3.2. You can find these here: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2alp… http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2bet… This time in addition to the source code, I also release syslog-ng binaries for Linux/i386 and Linux/amd64. The binaries are not thouroughly tested and will probably never be. The canonical release format for syslog-ng is the source code. Anyway, if you feel there are problems with the installer/binaries, feedback is welcome, we're willing to fix them. Here's the NEWS entry for the 3.2beta1 release: 3.2beta1 Mon, 11 Oct 2010 12:25:07 +0200 Changes and new features destined to the syslog-ng 3.2 release are complete, and starting with this release, only bugfixes and minor changes are possible. There's only one exception to this: the correllation framework in db-parser() is still considered experimental and is recommended for early adopters only. This beta has gone through some testing and initial blocker problems were fixed before the release. Right now I'm not aware of any serious issues, but as always, testing is appreciated. New features since 3.2alpha2: * Added support for message correllation in db-parser. See the relevant blog posts for more information: http://bazsi.blogs.balabit.com/2010/10/syslog-ng-correllation-updated/ http://bazsi.blogs.balabit.com/2010/09/syslog-ng-correllation/ * Added "pdbtool patternize", which implements automatic patterndb generation from a sample log file. http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ * Added template functions framework and some initial functions: http://bazsi.blogs.balabit.com/2010/09/introducing-template-functions/ The new functions are: $(echo), $(grep) and $(if) * Added support to process native syslog.conf file using the syslogconf SCL plugin. http://bazsi.blogs.balabit.com/2010/09/syslog-ng-now-supports-the-syslog-co… * Added support for comparison operators in filter expressions, e.g. it is now possible to use "$FACILITY_NUM" < "5". String and numeric operators are also provided, the same way as in perl. * syslog-ng now automatically detects if an incoming message is in RFC3164 or RFC5424 format. This means that the syslog driver can be used to process both. * Added pdbtool validation support, using the "pdbtool test --validate". Requires an installed xmllint program. * pdbtool is now able to merge patterndb XML files recursively in order to make it easy to use the results of the patterndb project. * db-parser() automatically assigns class-specific tags to messages, this means that a message classified "system" will get a ".classifier.system" tag in addition to storing the class in a name-value pair named ${.classifier.class} * It is now possible to use multiple program name patterns for a single ruleset in patterndb. * Added $(ipv4-to-int) template function to convert an IP address to its numeric representation. Bugfixes since 3.2alpha2: * Fixed a possible infinite loop in "pdbtool test" in case program/message was missing from the sample message. * SQL: revert don't require the current CVS version of libdbi * Don't report "this config file version is too old" multiple times. * Underscore and dash are assumed to be equivalent in plugin names. * Various memory leaks were plugged. Removed functions: * Removed the use_time_recvd() global and per-destination option, deprecated since 3.0. Can be substituted with $R_ prefix in macro names. Other changes: * Restructured the source tree in order to make compilations of independent plugins easier and faster. Modules go to modules/ subdirectory, the core lives under lib/ and the main executables go into syslog-ng/ * SCL paths are determined relative to ${datadir} instead of ${prefix} to make distribution packaging easier. * Pass -avoid-version when linking modules. * syslog-ng now requires bison 2.4, this is also checked by the configure script. Credits: I'd like to thank the following people who made this release possible. People in no particular order: * Péter Gyöngyösi for his patternize work * Márton Illés for pdbtool test * Róbert Fekete for his feedback on various functions/ideas * Csaba Major for listening to my stupid ideas :) * Martin Grauel for giving a talk on syslog-ng 3.2 and for his number of useful comments * Bálint Kovács for trying patternize, giving me bugreports and for his patternize related ideas * Péter Czanik for asking distributor maintainers to package syslog-ng 3.2 and to return feedback on the build process * Matthew Hall, Martin Holste, Patrick H. for his patterndb feedback and their support on the mailing that gave me time to work on syslog-ng instead of doing syslog-ng related mailing Hopefully I haven't forgotten anyone. I'd like to open the 3.3 branch of syslog-ng now and I'd appreciate if distributions would start adopting the 3.2 package. It comes with big build changes, so maintainers please expect some more time with this release. I'm trying to fix bugs fast as they appear to get a stable 3.2 by the end of this year. Happy logging! -- Bazsi

1 0

anushri kannu wants to chat
by anushri kannu 16 Oct '10

16 Oct '10

----------------------------------------------------------------------- anushri kannu wants to stay in better touch using some of Google's coolest new products. If you already have Gmail or Google Talk, visit: http://mail.google.com/mail/b-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… You'll need to click this link to be able to chat with anushri kannu. To get Gmail - a free email account from Google with over 2,800 megabytes of storage - and chat with anushri kannu, visit: http://mail.google.com/mail/a-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… Gmail offers: - Instant messaging right inside Gmail - Powerful spam protection - Built-in search for finding your messages and a helpful way of organizing emails into "conversations" - No pop-up ads or untargeted banners - just text ads and related information that are relevant to the content of your messages All this, and its yours for free. But wait, there's more! By opening a Gmail account, you also get access to Google Talk, Google's instant messaging service: http://www.google.com/talk/ Google Talk offers: - Web-based chat that you can use anywhere, without a download - A contact list that's synchronized with your Gmail account - Free, high quality PC-to-PC voice calls when you download the Google Talk client We're working hard to add new features and make improvements, so we might also ask for your comments and suggestions periodically. We appreciate your help in making our products even better! Thanks, The Google Team To learn more about Gmail and Google Talk, visit: http://mail.google.com/mail/help/about.html http://www.google.com/talk/about.html (If clicking the URLs in this message does not work, copy and paste them into the address bar of your browser).

4 4

pattern naming problems
by Peter Czanik 16 Oct '10

16 Oct '10

Hello, I ran into an interesting situation. Yesterday I created patterns for uw-imapd. Today I looked at some log files from courier imapd and found, that this imap implementation also uses "imapd" for $PROGRAM (and "imapd-ssl" for port 993 connections). "imapd" and "imapd-ssl" messages looked the same. So, here is a list of questions: - how to handle at file/ruleset/etc level when two applications have the same $PROGRAM - how to handle, when the same application uses different $PROGRAM in different situations Also, I took a look at postfix logs, and they look rather ugly: Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: connect from czp.localnet[192.168.2.179] Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: 8434B41C30: client=czp.localnet[192.168.2.179], sasl_method=PLAIN, sasl_username=czanik Oct 13 21:35:29 ubuntu postfix/cleanup[3946]: 8434B41C30: message-id=<4CB609F2.8(a)blabla.com> Oct 13 21:35:29 ubuntu postfix/qmgr[3570]: 8434B41C30: from=<czanik(a)blabla.com>, size=619, nrcpt=1 (queue active) Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: disconnect from czp.localnet[192.168.2.179] Oct 13 21:35:50 ubuntu postfix/smtp[3947]: connect to targetmachine[1.2.3.4]:25: Connection timed out Oct 13 21:35:50 ubuntu postfix/smtp[3947]: 8434B41C30: to=<czanik@targetmachine>, relay=none, delay=21, delays=0.02/0.01/21/0, dsn=4.4.1, status=deferred (connect to targetmachine[1.2.3.4]:25: Connection timed out) This is an SMTP authentication, and then the e-mail is tried to be delivered to targetmachine. There are many different names as $PROGRAM, $PID also has many different values. But "8434B41C30" could easily be used as session identifier for all of this. The question is the same: how should pattern name be handled? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 4

repeated lines
by Peter Czanik 16 Oct '10

16 Oct '10

Hello, I'm just preparing patterns for uw-imapd. Problem: Oct 12 16:54:12 ubuntu imapd[10555]: connect from 192.168.2.179 (192.168.2.179) Oct 12 16:54:12 ubuntu imapd[10555]: imap service init from 192.168.2.179 Oct 12 16:54:17 ubuntu imapd[10555]: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] Oct 12 16:54:20 ubuntu imapd[10555]: AUTHENTICATE PLAIN failure host=czp.localnet [192.168.2.179] Oct 12 16:54:20 ubuntu imapd[10555]: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] Oct 12 16:54:48 ubuntu imapd[10555]: Unexpected client disconnect, while reading line user=asdf host=czp.localnet [192.168.2.179] As you can see, the line containing most information about the login failure is repeated twice: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] How can I make sure, that only one name value pair is generated from this event? Bazsi: we talked about the importance of time on Monday: messages for a single event arrived in an 8 seconds time span... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1