- syslog-ng - lists.balabit.hu

Re: [syslog-ng] [patch] fix linking issue in 3.2beta1
by Guillaume Rousse 19 Oct '10

19 Oct '10

Le 18/10/2010 10:18, Balazs Scheidler a écrit : > Hi, > > Thanks for the patch, but I it was my intent not to link the plugins > against GLIB & other dependencies. > > Do you have an error message that this patch actually fixes? Build failure, when -Wl,--no-undefined flag is used. > The reason that the plugins are not linked against GLIB and other is > that I have to support an almost static syslog-ng, in which case GLib is > statically linked into the libsyslog-ng.so shared object and plugins > pull it from there. I may indeed disable this flag here, it's primary target is shared library, not plugin. -- BOFH excuse #107: The keyboard isn't plugged in

2 1

pdbtool patternize update and my syslog-ng 3.2 branch
by Peter Gyongyosi 19 Oct '10

19 Oct '10

Hello, As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ Besides the port to the new codebase, it's received some fixes and new features since my original post: * multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing. It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on. It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome. greets, Peter ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.

6 24

decoding messages from sockd (SOCKS proxy)
by Matthew Hall 18 Oct '10

18 Oct '10

Hello all, I am running into some headaches with the poor formatting of sockd messages. How should I decode messages like this? Note I have not applied XML escapes to these yet as that's hard to read but I will do so when inserting them into a patterndb to prevent parse errors. Every message in this group begins with this string on one line: sockd[@NUMBER:pid:@]: @ESTRING:action::@ @ESTRING:phase::@ @IPv4:src:@.@NUMBER:srcport:@ -> Then there are a few different endings which happen in some messages that are giving me problems to decode. Here are three examples from my collected logs: smarthost.company.com.25 host.team.division.company.com.18050: invalid address: 0.0.0.0.18050 company.com.443: Connection reset by peer I am having a hard time figuring out how to break these up into domain name (src / dst as appropriate) and port (srcport / dstport). My best thought so far was to detect this and rewrite them using PCRE before applying patterndb matching. I could find the .[0-9]+ and replace with :\1, then I have the port delimited with ':' and I can pull it apart using: @ESTRING:src::@@NUMBER:srcport:@ Is it possible to do PCRE replacement using backreferences? Or is there another way to get this to work? Thanks, Matthew.

2 2

pattern matching on xxx#
by Bill Anderson 18 Oct '10

18 Oct '10

I have hostnames of the format xxxx# such as host1, hostb1, hostc1. I need to split that into two fields such as (host,1). Unfortunately, since @@ escapes the @ and STRING and it's followers ALSO match digits, I've not found the obvious means to get that out. Conceptually something like @LETTER:host.name@@NUMBER:host.id@ woudl do it, save that LETTER doesn't exist and @@ escapes. The end goal is as follows (pseudo-code): I need to have a destination for each (HOST). For example all files from hosta## go to /var/log/hosta/ and entries for hostb## go to /var/log/hostb/ I suppose I *might* be able to do a rewrite to add say, a hyphen, and then use csv-parser, but we're talking some heavy traffic and I suspect that doing rewrites on that much traffic would be a performance killer. I'm open to suggestions (that don't involve changing server names, preferably ;) ) as to how to accomplish this. Cheers, Bill

4 10

[Bug 95] New: Missing capabilities support for unix-stream() source
by bugzilla＠bugzilla.balabit.com 18 Oct '10

18 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=95 Summary: Missing capabilities support for unix-stream() source Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: --- Estimated Hours: 0.0 Operations on unix-stream() sources need capabilities: - CAP_CHOWN - needed if owner() or group() are in use - CAP_FOWNER - to force chmod() for sockets with owner != root - CAP_DAC_OVERRIDE - force changes if some parent dir has 000 perm, ie vservers Attached patch fixes all these problems. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[patch] fix linking issue in 3.2beta1
by Guillaume Rousse 18 Oct '10

18 Oct '10

The following patch fixes some linking issue when using -Wl,--no-undefined flag. -- BOFH excuse #126: it has Intel Inside

2 1

Performance analysis (was Re: UDP packet loss with syslog-ng)
by Lars Kellogg-Stedman 18 Oct '10

18 Oct '10

I've run a series of tests against our log server, using loggen-generated logging rates of 100, 1000, 2000, 4000, 8000, 16000, and 32000 messages/sec, and measured the results for each rate using socket buffer sizes ranging from 128KB to 16M. The results show, essentially, what buffer size you need to meet a target rate of message logging. So I'm happy now, and I'm getting the sort of rates I expect to be able to support. I figured other folks might like the data. I've put the raw data online as well as a graph of the results: http://drop.io/syslog_ng The graph shows so_rcvbuf() size along the X axis, and packet loss along the Y axis.

3 3

Bazsi's blog: Syslog-ng correllation
by Balazs Scheidler 18 Oct '10

18 Oct '10

Syslog-ng correllation I think we’ve reached an important milestone with syslog-ng: log message correllation was added to db-parser(). As you probably know dbparser and its sister project patterndb is able to transform unstructured syslog messages into a normalized format: the human readable string content becomes a set of name-value pairs. The problem is that in a lot of cases messages miss one or two details that would really be needed to understand them and this information usually comes in a followup message. For example: one message in postfix logs contain the sender address and while the recipient information comes in the next message. It is trivial to understand that in reality most cases you want the information in sender,recipient pairs. Another example is sshd, where the authentication failure comes in one and the exact reason for the failure comes in the next. Currently what you can do with syslog-ng is to put the separate messages into two SQL tables and join them at query time. This gets ugly quite fast: increased storage needs, the hassle with managing two tables instead of one and not to mention the increase of the time needed to query the database. Sometimes the sole reason for creating SQL tables in this case is to perform the correllation, otherwise you’d be happier with a CSV file. And that’s what became possible now with the latest git commit of syslog-ng 3.2. The idea is simple: when a patterndb rule matches, you can tell syslog-ng to remember that message by adding it to a correllation state. This state is identified with information extracted from the message making it a unique session identifier. When the next line comes in you can reference the information stored earlier. Basically the correllation state is a list of log messages associated with a session id. To add a new message to this state, you need a store rule: <rule id=”…”> <patterns> <pattern>foo session: @STRING:sessionid@, param: @STRING:param@</pattern> </patterns> <store id=”$sessionid” timeout=”60″/> </rule> The id attribute of the store element specifies a template containing any syslog-ng name-value pairs, probably extracted from the current message itself. When the final information comes in you can use the join attribute of the values tag: <rule id=”…”> <patterns> <pattern>bar session: @STRING:sessionid@</pattern> </patterns> <values join=”$sessionid”> <value name=”param”>${param}@1</value> </values> </rule> here the join attribute specifies the session to look up (which must match in the two messages), and if there’s a match all messages stored in the correllation state becomes available when evaluating the name-value pairs associated with the current message. The key here is the new syntax in the template string “@1″ appended to a name-value pair reference. After the “@” character, you can reference a message in the correllation state by specifying the index backward from the current message. This way @0 is the current message, @1 is the one prior to the current one, @2 is before that and so on. There are more complex ways to use/query the contents of the correllation state, but those will appear in a followup post. Stay tuned!

2 8

UDP packet loss with syslog-ng
by Lars Kellogg-Stedman 17 Oct '10

17 Oct '10

Hello all, I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of memory. Using stock kernel settings (e.g., without adjusting net.core.rmem_default), we're not able to handle much more than 100 messages/second (generated from a remote host using the "loggen" tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000 msg/sec, we see closer to 60% packet loss. Our configuration looks approximately like this (template definitions elided for brevity): options { time_reap(30); mark_freq(10); keep_hostname(yes); use_fqdn(yes); dns_cache(2000); dns_cache_expire(86400); }; source s_network { udp(); tcp(port(514)); }; destination d_syslog { file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages" template(t_daily_log) create_dirs(yes) ); file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY" template(t_host_log) create_dirs(yes) ); }; log { source(s_network); destination(d_syslog); }; I didn't think these message rates were terribly high, so I was surprised at the loss. We've confirmed that the loss is entirely between the kernel and the application -- using wireshark, we've verified that all of the packets are arriving at the host, and using this: awk '{print}' /inet/udp/514/0/0 > out Our packet loss is < 1%. If I raise the rmem settings like this: net.core.rmem_default = 512000 net.core.rmem_max = 1024000 Then it looks like I can support messages rates around 1000 msgs/sec. If I try with 2000 msgs/sec, the loss rates jumps up again (to around 30%). Do these numbers make sense? This is an unloaded server. The only log traffic hitting this system is from my loggen runs. The filesystem is ext3 on top of a hardware RAID5 array. I've tried fiddling with some of the syslog-ng global options (e.g., flush_lines(), log_fetch_limit()), but without having much impact on performance. I would appreciate any help you can send our way. Thanks! -- Lars

6 26

syslog-ng OSE 3.2beta1 has been released (source + binaries)
by Balazs Scheidler 16 Oct '10

16 Oct '10

Dear syslog-ng users, I'm happy to announce that an important milestone towards the new syslog-ng release has been reached. syslog-ng OSE 3.2beta1 has been uploaded to our website. It contains a lot of important new functionality compared to 3.2alpha2 and also some care has been taken to shake out release-critical bugs. As of know I don't know about any Release Critical issues, but I do now that a number of such bugs have been fixed since alpha2. So if you had issues with the alpha release and didn't report the issues, it still is probable that this release fixes them. I think this beta is good enough for everyone to try out, so if you like the new features, I'd appreciate if you could report with either negative or positive feedback to the syslog-ng mailing list. If you haven't tracked 3.2 development so far, I'd recommend to read the 3.2alpha2 and 3.2beta1 announcements for a sumary of new features coming in 3.2. You can find these here: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2alp… http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2bet… This time in addition to the source code, I also release syslog-ng binaries for Linux/i386 and Linux/amd64. The binaries are not thouroughly tested and will probably never be. The canonical release format for syslog-ng is the source code. Anyway, if you feel there are problems with the installer/binaries, feedback is welcome, we're willing to fix them. Here's the NEWS entry for the 3.2beta1 release: 3.2beta1 Mon, 11 Oct 2010 12:25:07 +0200 Changes and new features destined to the syslog-ng 3.2 release are complete, and starting with this release, only bugfixes and minor changes are possible. There's only one exception to this: the correllation framework in db-parser() is still considered experimental and is recommended for early adopters only. This beta has gone through some testing and initial blocker problems were fixed before the release. Right now I'm not aware of any serious issues, but as always, testing is appreciated. New features since 3.2alpha2: * Added support for message correllation in db-parser. See the relevant blog posts for more information: http://bazsi.blogs.balabit.com/2010/10/syslog-ng-correllation-updated/ http://bazsi.blogs.balabit.com/2010/09/syslog-ng-correllation/ * Added "pdbtool patternize", which implements automatic patterndb generation from a sample log file. http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ * Added template functions framework and some initial functions: http://bazsi.blogs.balabit.com/2010/09/introducing-template-functions/ The new functions are: $(echo), $(grep) and $(if) * Added support to process native syslog.conf file using the syslogconf SCL plugin. http://bazsi.blogs.balabit.com/2010/09/syslog-ng-now-supports-the-syslog-co… * Added support for comparison operators in filter expressions, e.g. it is now possible to use "$FACILITY_NUM" < "5". String and numeric operators are also provided, the same way as in perl. * syslog-ng now automatically detects if an incoming message is in RFC3164 or RFC5424 format. This means that the syslog driver can be used to process both. * Added pdbtool validation support, using the "pdbtool test --validate". Requires an installed xmllint program. * pdbtool is now able to merge patterndb XML files recursively in order to make it easy to use the results of the patterndb project. * db-parser() automatically assigns class-specific tags to messages, this means that a message classified "system" will get a ".classifier.system" tag in addition to storing the class in a name-value pair named ${.classifier.class} * It is now possible to use multiple program name patterns for a single ruleset in patterndb. * Added $(ipv4-to-int) template function to convert an IP address to its numeric representation. Bugfixes since 3.2alpha2: * Fixed a possible infinite loop in "pdbtool test" in case program/message was missing from the sample message. * SQL: revert don't require the current CVS version of libdbi * Don't report "this config file version is too old" multiple times. * Underscore and dash are assumed to be equivalent in plugin names. * Various memory leaks were plugged. Removed functions: * Removed the use_time_recvd() global and per-destination option, deprecated since 3.0. Can be substituted with $R_ prefix in macro names. Other changes: * Restructured the source tree in order to make compilations of independent plugins easier and faster. Modules go to modules/ subdirectory, the core lives under lib/ and the main executables go into syslog-ng/ * SCL paths are determined relative to ${datadir} instead of ${prefix} to make distribution packaging easier. * Pass -avoid-version when linking modules. * syslog-ng now requires bison 2.4, this is also checked by the configure script. Credits: I'd like to thank the following people who made this release possible. People in no particular order: * Péter Gyöngyösi for his patternize work * Márton Illés for pdbtool test * Róbert Fekete for his feedback on various functions/ideas * Csaba Major for listening to my stupid ideas :) * Martin Grauel for giving a talk on syslog-ng 3.2 and for his number of useful comments * Bálint Kovács for trying patternize, giving me bugreports and for his patternize related ideas * Péter Czanik for asking distributor maintainers to package syslog-ng 3.2 and to return feedback on the build process * Matthew Hall, Martin Holste, Patrick H. for his patterndb feedback and their support on the mailing that gave me time to work on syslog-ng instead of doing syslog-ng related mailing Hopefully I haven't forgotten anyone. I'd like to open the 3.3 branch of syslog-ng now and I'd appreciate if distributions would start adopting the 3.2 package. It comes with big build changes, so maintainers please expect some more time with this release. I'm trying to fix bugs fast as they appear to get a stable 3.2 by the end of this year. Happy logging! -- Bazsi

1 0