- syslog-ng - lists.balabit.hu

password mailing-list
by laugello 28 Oct '10

28 Oct '10

Hello Sorry of the o-topic I forgot password of the mailing-list How I can recover it tanks Luigi -- Augello Luigi Amministratore di Sistema Poli didattici di Agrigento, Caltanissetta e Trapani Università degli Studi di Palermo tel 093420928 VoIP 09123865802

2 1

AppArmor
by Peter Czanik 26 Oct '10

26 Oct '10

Hello, I wonder, if anyone here uses syslog-ng together with AppArmor. SCL calls external applications, or using program() in syslog-ng.conf also calls an external app. I spent quite some time with AppArmor documentation, tried logprof many times, asked on openSUSE mailing lists, but no luck. I could not execute external apps when AppArmor was enabled. Does anybody has a success story here? I'd be very interested in your AppArmor profile for syslog-ng! BTW: if you use openSUSE, you could test my 3.2beta1 packages. See my blog for details! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Should configure error out if lex/flex is missing?
by Lars Kellogg-Stedman 19 Oct '10

19 Oct '10

After successfully running ./configure for syslog-ng 3.2beta1, the build immediately failed because I didn't have flex installed. I see that configure is already calling AM_PROG_LEX...so maybe this is just the expected behavior? After installing flex the build completed successfully.

2 3

[Bug 52] CAP_DAC_OVERRIDE is missing when capabilities are enabled, unable to write to files created with the owner() option.
by bugzilla＠bugzilla.balabit.com 19 Oct '10

19 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=52 Balazs Scheidler <bazsi(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #4 from Balazs Scheidler <bazsi(a)balabit.hu> 2010-10-19 17:23:38 --- I've integrated this as well, thanks Zbigniew commit 967b1720c8487f3cbe49292c7e5ed3b871ab2de5 Author: Zbigniew Krzystolik <zbyniu(a)pld-linux.org> Date: Tue Oct 19 17:21:23 2010 +0200 fix capability in the file() destination fixes for settings like owner(), group(), perm() and analogous dir_* if create_dirs=yes for destination files - CAP_DAC_OVERRIDE force changes if parent dir has 000 perm, ie vservers (is inerhitted for dir_* too) - CAP_CHOWN - needed if dir_owner() or dir_group() are in use - CAP_FOWNER - to force chmod() for dirs with owner != root -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Buffering AF_UNIX Destination, Batch Post Processing Messages
by Matthew Hall 19 Oct '10

19 Oct '10

Hello All, I want to configure an AF_UNIX SOCK_DGRAM syslog-ng destination which sends certain log messages to an external program for further processing and analysis. This program should batch up the messages into 60 second batches for processing. Currently I am running into an architectural challenge in how I should process the 60 second batch without slowing down the select which is collecting the messages from the destination. In the past when creating a similar kind of application in Java I handled this by creating a huge dynamic array to store objects creating from each incoming message, then passed the array reference to another background thread for processing, and began building a new array in the select thread. Currently I am trying to solve this same basic problem in Perl, which has poor threading support. I am investigating a few different options: * use threads anyway-- not recommended by more expert Perl devs I asked * prefork a process which listens to the AF_UNIX from syslog-ng, and writes to some kind of buffered non blocking pipe with a really big buffer-- not sure if such a pipe device actually exists, many pipes block * postfork a worker process which handles the 60 second batch-- problem here is that you want to have a whole lot of long term state data which is maintained between batches to help separate the needles from the haystacks, and you could get weird behavior on the duplicated FDs that are still being select()ed in the parent process which are copied into the child. * use some kind of message or job queue to copy things from a producer process to a consumer process-- gearman, theschwartz, beanstalk, rabbitmq, activemq, and poe::component::mq have been suggested-- this would probably cause a lot of context switching and unwanted buffer copies * see if there is something existing in syslog-ng that can help with this situation. can it somehow be convinced to buffer things internally for my process when my process is busy on a 60 second batch, or send in 60 second batch, etc. / whatever other clever people can dream up? Is this a problem other people have dealt with before? What did you do about this one? I want to get this right and avoid making a big mess or reinventing the wheel. Matthew.

4 18

Re: [syslog-ng] [patch] fix linking issue in 3.2beta1
by Guillaume Rousse 19 Oct '10

19 Oct '10

Le 18/10/2010 10:18, Balazs Scheidler a écrit : > Hi, > > Thanks for the patch, but I it was my intent not to link the plugins > against GLIB & other dependencies. > > Do you have an error message that this patch actually fixes? Build failure, when -Wl,--no-undefined flag is used. > The reason that the plugins are not linked against GLIB and other is > that I have to support an almost static syslog-ng, in which case GLib is > statically linked into the libsyslog-ng.so shared object and plugins > pull it from there. I may indeed disable this flag here, it's primary target is shared library, not plugin. -- BOFH excuse #107: The keyboard isn't plugged in

2 1

pdbtool patternize update and my syslog-ng 3.2 branch
by Peter Gyongyosi 19 Oct '10

19 Oct '10

Hello, As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ Besides the port to the new codebase, it's received some fixes and new features since my original post: * multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing. It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on. It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome. greets, Peter ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.

6 24

decoding messages from sockd (SOCKS proxy)
by Matthew Hall 18 Oct '10

18 Oct '10

Hello all, I am running into some headaches with the poor formatting of sockd messages. How should I decode messages like this? Note I have not applied XML escapes to these yet as that's hard to read but I will do so when inserting them into a patterndb to prevent parse errors. Every message in this group begins with this string on one line: sockd[@NUMBER:pid:@]: @ESTRING:action::@ @ESTRING:phase::@ @IPv4:src:@.@NUMBER:srcport:@ -> Then there are a few different endings which happen in some messages that are giving me problems to decode. Here are three examples from my collected logs: smarthost.company.com.25 host.team.division.company.com.18050: invalid address: 0.0.0.0.18050 company.com.443: Connection reset by peer I am having a hard time figuring out how to break these up into domain name (src / dst as appropriate) and port (srcport / dstport). My best thought so far was to detect this and rewrite them using PCRE before applying patterndb matching. I could find the .[0-9]+ and replace with :\1, then I have the port delimited with ':' and I can pull it apart using: @ESTRING:src::@@NUMBER:srcport:@ Is it possible to do PCRE replacement using backreferences? Or is there another way to get this to work? Thanks, Matthew.

2 2

pattern matching on xxx#
by Bill Anderson 18 Oct '10

18 Oct '10

I have hostnames of the format xxxx# such as host1, hostb1, hostc1. I need to split that into two fields such as (host,1). Unfortunately, since @@ escapes the @ and STRING and it's followers ALSO match digits, I've not found the obvious means to get that out. Conceptually something like @LETTER:host.name@@NUMBER:host.id@ woudl do it, save that LETTER doesn't exist and @@ escapes. The end goal is as follows (pseudo-code): I need to have a destination for each (HOST). For example all files from hosta## go to /var/log/hosta/ and entries for hostb## go to /var/log/hostb/ I suppose I *might* be able to do a rewrite to add say, a hyphen, and then use csv-parser, but we're talking some heavy traffic and I suspect that doing rewrites on that much traffic would be a performance killer. I'm open to suggestions (that don't involve changing server names, preferably ;) ) as to how to accomplish this. Cheers, Bill

4 10

[Bug 95] New: Missing capabilities support for unix-stream() source
by bugzilla＠bugzilla.balabit.com 18 Oct '10

18 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=95 Summary: Missing capabilities support for unix-stream() source Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: --- Estimated Hours: 0.0 Operations on unix-stream() sources need capabilities: - CAP_CHOWN - needed if owner() or group() are in use - CAP_FOWNER - to force chmod() for sockets with owner != root - CAP_DAC_OVERRIDE - force changes if some parent dir has 000 perm, ie vservers Attached patch fixes all these problems. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[patch] fix linking issue in 3.2beta1
by Guillaume Rousse 18 Oct '10

18 Oct '10

The following patch fixes some linking issue when using -Wl,--no-undefined flag. -- BOFH excuse #126: it has Intel Inside

2 1

Performance analysis (was Re: UDP packet loss with syslog-ng)
by Lars Kellogg-Stedman 18 Oct '10

18 Oct '10

I've run a series of tests against our log server, using loggen-generated logging rates of 100, 1000, 2000, 4000, 8000, 16000, and 32000 messages/sec, and measured the results for each rate using socket buffer sizes ranging from 128KB to 16M. The results show, essentially, what buffer size you need to meet a target rate of message logging. So I'm happy now, and I'm getting the sort of rates I expect to be able to support. I figured other folks might like the data. I've put the raw data online as well as a graph of the results: http://drop.io/syslog_ng The graph shows so_rcvbuf() size along the X axis, and packet loss along the Y axis.

3 3

Bazsi's blog: Syslog-ng correllation
by Balazs Scheidler 18 Oct '10

18 Oct '10

Syslog-ng correllation I think we’ve reached an important milestone with syslog-ng: log message correllation was added to db-parser(). As you probably know dbparser and its sister project patterndb is able to transform unstructured syslog messages into a normalized format: the human readable string content becomes a set of name-value pairs. The problem is that in a lot of cases messages miss one or two details that would really be needed to understand them and this information usually comes in a followup message. For example: one message in postfix logs contain the sender address and while the recipient information comes in the next message. It is trivial to understand that in reality most cases you want the information in sender,recipient pairs. Another example is sshd, where the authentication failure comes in one and the exact reason for the failure comes in the next. Currently what you can do with syslog-ng is to put the separate messages into two SQL tables and join them at query time. This gets ugly quite fast: increased storage needs, the hassle with managing two tables instead of one and not to mention the increase of the time needed to query the database. Sometimes the sole reason for creating SQL tables in this case is to perform the correllation, otherwise you’d be happier with a CSV file. And that’s what became possible now with the latest git commit of syslog-ng 3.2. The idea is simple: when a patterndb rule matches, you can tell syslog-ng to remember that message by adding it to a correllation state. This state is identified with information extracted from the message making it a unique session identifier. When the next line comes in you can reference the information stored earlier. Basically the correllation state is a list of log messages associated with a session id. To add a new message to this state, you need a store rule: <rule id=”…”> <patterns> <pattern>foo session: @STRING:sessionid@, param: @STRING:param@</pattern> </patterns> <store id=”$sessionid” timeout=”60″/> </rule> The id attribute of the store element specifies a template containing any syslog-ng name-value pairs, probably extracted from the current message itself. When the final information comes in you can use the join attribute of the values tag: <rule id=”…”> <patterns> <pattern>bar session: @STRING:sessionid@</pattern> </patterns> <values join=”$sessionid”> <value name=”param”>${param}@1</value> </values> </rule> here the join attribute specifies the session to look up (which must match in the two messages), and if there’s a match all messages stored in the correllation state becomes available when evaluating the name-value pairs associated with the current message. The key here is the new syntax in the template string “@1″ appended to a name-value pair reference. After the “@” character, you can reference a message in the correllation state by specifying the index backward from the current message. This way @0 is the current message, @1 is the one prior to the current one, @2 is before that and so on. There are more complex ways to use/query the contents of the correllation state, but those will appear in a followup post. Stay tuned!

2 8

UDP packet loss with syslog-ng
by Lars Kellogg-Stedman 17 Oct '10

17 Oct '10

Hello all, I'm deploying syslog-ng 3.0.8 on a quad-core 2.4Ghz system with 4GB of memory. Using stock kernel settings (e.g., without adjusting net.core.rmem_default), we're not able to handle much more than 100 messages/second (generated from a remote host using the "loggen" tool). At 500 msg/sec (-r 500), we see about 50% loss, and at 1000 msg/sec, we see closer to 60% packet loss. Our configuration looks approximately like this (template definitions elided for brevity): options { time_reap(30); mark_freq(10); keep_hostname(yes); use_fqdn(yes); dns_cache(2000); dns_cache_expire(86400); }; source s_network { udp(); tcp(port(514)); }; destination d_syslog { file("/srv/syslog/bydate/$YEAR-$MONTH-$DAY/messages" template(t_daily_log) create_dirs(yes) ); file("/srv/syslog/byhost/$FULLHOST_FROM/$YEAR-$MONTH-$DAY" template(t_host_log) create_dirs(yes) ); }; log { source(s_network); destination(d_syslog); }; I didn't think these message rates were terribly high, so I was surprised at the loss. We've confirmed that the loss is entirely between the kernel and the application -- using wireshark, we've verified that all of the packets are arriving at the host, and using this: awk '{print}' /inet/udp/514/0/0 > out Our packet loss is < 1%. If I raise the rmem settings like this: net.core.rmem_default = 512000 net.core.rmem_max = 1024000 Then it looks like I can support messages rates around 1000 msgs/sec. If I try with 2000 msgs/sec, the loss rates jumps up again (to around 30%). Do these numbers make sense? This is an unloaded server. The only log traffic hitting this system is from my loggen runs. The filesystem is ext3 on top of a hardware RAID5 array. I've tried fiddling with some of the syslog-ng global options (e.g., flush_lines(), log_fetch_limit()), but without having much impact on performance. I would appreciate any help you can send our way. Thanks! -- Lars

6 26

syslog-ng OSE 3.2beta1 has been released (source + binaries)
by Balazs Scheidler 16 Oct '10

16 Oct '10

Dear syslog-ng users, I'm happy to announce that an important milestone towards the new syslog-ng release has been reached. syslog-ng OSE 3.2beta1 has been uploaded to our website. It contains a lot of important new functionality compared to 3.2alpha2 and also some care has been taken to shake out release-critical bugs. As of know I don't know about any Release Critical issues, but I do now that a number of such bugs have been fixed since alpha2. So if you had issues with the alpha release and didn't report the issues, it still is probable that this release fixes them. I think this beta is good enough for everyone to try out, so if you like the new features, I'd appreciate if you could report with either negative or positive feedback to the syslog-ng mailing list. If you haven't tracked 3.2 development so far, I'd recommend to read the 3.2alpha2 and 3.2beta1 announcements for a sumary of new features coming in 3.2. You can find these here: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2alp… http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2bet… This time in addition to the source code, I also release syslog-ng binaries for Linux/i386 and Linux/amd64. The binaries are not thouroughly tested and will probably never be. The canonical release format for syslog-ng is the source code. Anyway, if you feel there are problems with the installer/binaries, feedback is welcome, we're willing to fix them. Here's the NEWS entry for the 3.2beta1 release: 3.2beta1 Mon, 11 Oct 2010 12:25:07 +0200 Changes and new features destined to the syslog-ng 3.2 release are complete, and starting with this release, only bugfixes and minor changes are possible. There's only one exception to this: the correllation framework in db-parser() is still considered experimental and is recommended for early adopters only. This beta has gone through some testing and initial blocker problems were fixed before the release. Right now I'm not aware of any serious issues, but as always, testing is appreciated. New features since 3.2alpha2: * Added support for message correllation in db-parser. See the relevant blog posts for more information: http://bazsi.blogs.balabit.com/2010/10/syslog-ng-correllation-updated/ http://bazsi.blogs.balabit.com/2010/09/syslog-ng-correllation/ * Added "pdbtool patternize", which implements automatic patterndb generation from a sample log file. http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ * Added template functions framework and some initial functions: http://bazsi.blogs.balabit.com/2010/09/introducing-template-functions/ The new functions are: $(echo), $(grep) and $(if) * Added support to process native syslog.conf file using the syslogconf SCL plugin. http://bazsi.blogs.balabit.com/2010/09/syslog-ng-now-supports-the-syslog-co… * Added support for comparison operators in filter expressions, e.g. it is now possible to use "$FACILITY_NUM" < "5". String and numeric operators are also provided, the same way as in perl. * syslog-ng now automatically detects if an incoming message is in RFC3164 or RFC5424 format. This means that the syslog driver can be used to process both. * Added pdbtool validation support, using the "pdbtool test --validate". Requires an installed xmllint program. * pdbtool is now able to merge patterndb XML files recursively in order to make it easy to use the results of the patterndb project. * db-parser() automatically assigns class-specific tags to messages, this means that a message classified "system" will get a ".classifier.system" tag in addition to storing the class in a name-value pair named ${.classifier.class} * It is now possible to use multiple program name patterns for a single ruleset in patterndb. * Added $(ipv4-to-int) template function to convert an IP address to its numeric representation. Bugfixes since 3.2alpha2: * Fixed a possible infinite loop in "pdbtool test" in case program/message was missing from the sample message. * SQL: revert don't require the current CVS version of libdbi * Don't report "this config file version is too old" multiple times. * Underscore and dash are assumed to be equivalent in plugin names. * Various memory leaks were plugged. Removed functions: * Removed the use_time_recvd() global and per-destination option, deprecated since 3.0. Can be substituted with $R_ prefix in macro names. Other changes: * Restructured the source tree in order to make compilations of independent plugins easier and faster. Modules go to modules/ subdirectory, the core lives under lib/ and the main executables go into syslog-ng/ * SCL paths are determined relative to ${datadir} instead of ${prefix} to make distribution packaging easier. * Pass -avoid-version when linking modules. * syslog-ng now requires bison 2.4, this is also checked by the configure script. Credits: I'd like to thank the following people who made this release possible. People in no particular order: * Péter Gyöngyösi for his patternize work * Márton Illés for pdbtool test * Róbert Fekete for his feedback on various functions/ideas * Csaba Major for listening to my stupid ideas :) * Martin Grauel for giving a talk on syslog-ng 3.2 and for his number of useful comments * Bálint Kovács for trying patternize, giving me bugreports and for his patternize related ideas * Péter Czanik for asking distributor maintainers to package syslog-ng 3.2 and to return feedback on the build process * Matthew Hall, Martin Holste, Patrick H. for his patterndb feedback and their support on the mailing that gave me time to work on syslog-ng instead of doing syslog-ng related mailing Hopefully I haven't forgotten anyone. I'd like to open the 3.3 branch of syslog-ng now and I'd appreciate if distributions would start adopting the 3.2 package. It comes with big build changes, so maintainers please expect some more time with this release. I'm trying to fix bugs fast as they appear to get a stable 3.2 by the end of this year. Happy logging! -- Bazsi

1 0

anushri kannu wants to chat
by anushri kannu 16 Oct '10

16 Oct '10

----------------------------------------------------------------------- anushri kannu wants to stay in better touch using some of Google's coolest new products. If you already have Gmail or Google Talk, visit: http://mail.google.com/mail/b-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… You'll need to click this link to be able to chat with anushri kannu. To get Gmail - a free email account from Google with over 2,800 megabytes of storage - and chat with anushri kannu, visit: http://mail.google.com/mail/a-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… Gmail offers: - Instant messaging right inside Gmail - Powerful spam protection - Built-in search for finding your messages and a helpful way of organizing emails into "conversations" - No pop-up ads or untargeted banners - just text ads and related information that are relevant to the content of your messages All this, and its yours for free. But wait, there's more! By opening a Gmail account, you also get access to Google Talk, Google's instant messaging service: http://www.google.com/talk/ Google Talk offers: - Web-based chat that you can use anywhere, without a download - A contact list that's synchronized with your Gmail account - Free, high quality PC-to-PC voice calls when you download the Google Talk client We're working hard to add new features and make improvements, so we might also ask for your comments and suggestions periodically. We appreciate your help in making our products even better! Thanks, The Google Team To learn more about Gmail and Google Talk, visit: http://mail.google.com/mail/help/about.html http://www.google.com/talk/about.html (If clicking the URLs in this message does not work, copy and paste them into the address bar of your browser).

4 4

pattern naming problems
by Peter Czanik 16 Oct '10

16 Oct '10

Hello, I ran into an interesting situation. Yesterday I created patterns for uw-imapd. Today I looked at some log files from courier imapd and found, that this imap implementation also uses "imapd" for $PROGRAM (and "imapd-ssl" for port 993 connections). "imapd" and "imapd-ssl" messages looked the same. So, here is a list of questions: - how to handle at file/ruleset/etc level when two applications have the same $PROGRAM - how to handle, when the same application uses different $PROGRAM in different situations Also, I took a look at postfix logs, and they look rather ugly: Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: connect from czp.localnet[192.168.2.179] Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: 8434B41C30: client=czp.localnet[192.168.2.179], sasl_method=PLAIN, sasl_username=czanik Oct 13 21:35:29 ubuntu postfix/cleanup[3946]: 8434B41C30: message-id=<4CB609F2.8(a)blabla.com> Oct 13 21:35:29 ubuntu postfix/qmgr[3570]: 8434B41C30: from=<czanik(a)blabla.com>, size=619, nrcpt=1 (queue active) Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: disconnect from czp.localnet[192.168.2.179] Oct 13 21:35:50 ubuntu postfix/smtp[3947]: connect to targetmachine[1.2.3.4]:25: Connection timed out Oct 13 21:35:50 ubuntu postfix/smtp[3947]: 8434B41C30: to=<czanik@targetmachine>, relay=none, delay=21, delays=0.02/0.01/21/0, dsn=4.4.1, status=deferred (connect to targetmachine[1.2.3.4]:25: Connection timed out) This is an SMTP authentication, and then the e-mail is tried to be delivered to targetmachine. There are many different names as $PROGRAM, $PID also has many different values. But "8434B41C30" could easily be used as session identifier for all of this. The question is the same: how should pattern name be handled? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 4

repeated lines
by Peter Czanik 16 Oct '10

16 Oct '10

Hello, I'm just preparing patterns for uw-imapd. Problem: Oct 12 16:54:12 ubuntu imapd[10555]: connect from 192.168.2.179 (192.168.2.179) Oct 12 16:54:12 ubuntu imapd[10555]: imap service init from 192.168.2.179 Oct 12 16:54:17 ubuntu imapd[10555]: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] Oct 12 16:54:20 ubuntu imapd[10555]: AUTHENTICATE PLAIN failure host=czp.localnet [192.168.2.179] Oct 12 16:54:20 ubuntu imapd[10555]: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] Oct 12 16:54:48 ubuntu imapd[10555]: Unexpected client disconnect, while reading line user=asdf host=czp.localnet [192.168.2.179] As you can see, the line containing most information about the login failure is repeated twice: Login failed user=asdf auth=asdf host=czp.localnet [192.168.2.179] How can I make sure, that only one name value pair is generated from this event? Bazsi: we talked about the importance of time on Monday: messages for a single event arrived in an 8 seconds time span... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

Diagnosing patterndb load errors
by Matthew Hall 15 Oct '10

15 Oct '10

I am getting the following sequence of load errors for my patterndbs. Duplicates eliminated, timestamps zeroed out, host info anonymized. Has anybody else gotten these and learned how to fix them? I'm having a hard time figuring out what 'type', 'name', and 'value' mean, to track them down in the XML files. Thanks, Matthew Hall. Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Name-value pairs cannot have a zero-length name; value='' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='Neighbor Down: Dead timer expired', value='19' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='36' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='39' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='40' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='41' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='44' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=', domain = <not specified>', value='8' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='12' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='13' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='20' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='21' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='22' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='23' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='24' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='27' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='29' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='30' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='31' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='33' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='35' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='4', name='None', value='38' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='vpn-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='load-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='firewall-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='group', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='. Please check authentication server radius/ldap/tacacs logs.', value='50' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='wireless-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Termination requested via signal, terminating; Jan 01 00:00:00 s_internal@host syslog-ng[23005]: syslog-ng shutting down; version='3.1.1' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='wireless-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='vpn-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='load-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='firewall-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: syslog-ng starting up; version='3.1.1'

3 3

Bazsi's blog: syslog-ng correllation updated
by Balazs Scheidler 15 Oct '10

15 Oct '10

syslog-ng correllation updated I’m trying to push syslog-ng 3.2beta1 out on the door, but as I was writing the NEWS entry I had to realize that the latest state of the patterndb correllation functions are undocumented so far. So here goes a blog post which tries to summarize how it works, so that I can include it in the NEWS entry :) My previous post on the topic used a syntax that included explicit “store” and “join” elements, but I’ve decided to drop those, as they stood in the way for some more juicy functionality. What remained is that the correllation is still focused around a “correllation state”, or as I’ve called internally a “context”. A context consists of a series of log messages related to each other in some way. As new messages come in, they may be associated with a context (e.g. added to the context). Also, when an incoming message is identified it can trigger some actions to be performed. And these actions can use all the information that was stored previously in the context. Let’s see how this work out with concrete examples: each rule in the patterndb has a “context-id” attribute telling db-parser() which context the given message should be associated with. This example covers an SSH login message: <rule id=”…” context-id=”$HOST:$PROGRAM:$PID” context-timeout=”86400″ context-scope=”global”> <patterns> <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern> </patterns> … </rule> Since multiple rules can reference the same context, multiple different kind of messages may be added into the same context as a result. E.g. the logout event looks like this: <rule context-id=”$HOST:$PROGRAM:$PID” context-timeout=”0″ context-scope=”global”> <patterns> <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:usracct.username:@</pattern> </patterns> … </rule> As you can see a “session” is identified using the triplet ($HOST, $PROGRAM, $PID) and these two rules correllate the login/logout events into the same context, which means that you can create a derived event that contains information from both of them. Please note that it is fairly common that messages only need to be correllated if they originate from the same host: e.g. this SSH login message needs only to be correllated to its logout counterpart if they both originate from the same host. In the previous example this was achieved using explicit macros in the context-id attribute, however since this is quite often the case, this was worked into a function of its own right: each rule can have a context-scope attribute: <rule id=”…” context-scope=”process” context-id=”ssh-login” context-timeout=”5″> </rule> The context-scope tells syslog-ng which messages need to be considered when looking for correllations: * process: only consider messages that have matching $HOST, $PROGRAM and $PID values * program: only consider messages that have matching $HOST and $PROGRAM values * host: only consider messages that have matching $HOST values * global: any kind of message is fine The default is to use “process”, which means that if it is true that the same process is emitting all the messages that you want to correllate, then you don’t need to use a variable part in your context-id attribute. But it is also important to know that it is way faster to specify the scope this way than it’d be to add all relevant macros to your context-id attribute. So far so good, we have all the functions that we used to have with the previous versions of the functionality. But I mentioned something about “actions” to be performed. Until now a patterndb rule basically only identified the incoming message, possibly associated tags and name-value pairs, but didn’t perform anything else. This is being changed: one or more actions can be associated with a patterndb rule in order to make it possible to react to more complex situations. Here’s an example action: <action rate=”1/86400″> <message> <values> <value name=”MESSAGE”>a patterndb rule matched</value> </values> </message> </action> Right now the only real response to a message is to generate another message, but this allows us to do a couple of powerful transformations, especially with the following options that you can specify for an action tag: * condition: specifies a syslog-ng filter expression that needs to be matched in order to really perform the action. It is evaluated on the current message that matched the rule. * rate: <num>/<period> specifies how much messages are to be generated (num), in the specified time period (period). Excess messages are dropped. For example: “1/60″ allows 1 message / minute. Rates apply to the given scope for the given rule/action. E.g. context-scope=”host”, rate=”1/60″ means that one message gets generated for _each host_ per minute. * trigger: specifies when to execute the action, there are two possible triggers right now: * match: execute immediately once the rule matches * timeout: execute when the correllation timer expires I’d like to highlight two things: * it is possible to react to the expiration of a correllation timer (e.g. trigger=”timeout”) * it is possible to generate a message only in case a given condition is met (e.g. “$PID” == “”) Right now new messages are posted to the internal() driver. This is not the way I wanted it to be, but doing my original plan would require an enormous refactorization of the code, and it is too late for that too happen. My original idea was to let the db-parser() emit multiple messages, but since the current state of affairs in syslog-ng assumes that only sources generate messages, that needs a lot of work. But hey, we need something to do for syslog-ng 3.3, right?

5 10

syslog-ng and ntsyslog
by Fiorenzi Alessandro 15 Oct '10

15 Oct '10

Hi, We have syslog-ng 3.05 as log server, and datagram syslog agent on windows system (originary ntsyslog) form e windows 2003 server with syslogagent configure I have this event on eventviewer Event Type: Success Audit Event Source: Security Event Category: Logon/Logoff Event ID: 538 Date: 10/12/2010 Time: 12:26:43 PM User: DOMAINXXX\A.Fiorenzi Computer: XXXXXX Description: User Logoff: User Name: A.Fiorenzi Domain: DOMAINXXX Logon ID: (0x0,0xF78F137) Logon Type: 10 and on syslog-ng server i get this: Oct 12 12:26:43 XXXXXX security[success]: 538 DOMAINXXX\a.fiorenzi User Logoff User Name: A.Fiorenz Domain: DOMAINXX Logo n ID: (0x0,0xF78F137 Logon Type: 1 where the descrition field has UserName, Domain, logon ID an Logon Type cutted. I have record the network traffic via tcpdump and I have seen data arrive correctly. So have set in syslog-ng.conf options the statement log_msg_size(8192); The problem is still open and I do not know how to solve, anyone can help me? Alessandro Fiorenzi Prima di stampare, pensa all'ambiente ** Think about the environment before printing ________________________________ Il presente messaggio, inclusi gli eventuali allegati, ha natura aziendale e potrebbe contenere informazioni confidenziali e/o riservate. Chiunque lo ricevesse per errore, ? pregato di avvisare tempestivamente il mittente e di cancellarlo. E' strettamente vietata qualsiasi forma di utilizzo, riproduzione o diffusione non autorizzata del contenuto di questo messaggio o di parte di esso. Pur essendo state assunte le dovute precauzioni per ridurre al minimo il rischio di trasmissione di virus, si suggerisce di effettuare gli opportuni controlli sui documenti allegati al presente messaggio. Non si assume alcuna responsabilit? per eventuali danni o perdite derivanti dalla presenza di virus. *** This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden. We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses. For the conduct of investment business in the UK, the Company is authorized by Bank of Italy and regulated by the Financial Services Authority.

4 3

Open source versions of syslog-ng with sql() destination
by Liam Kirsher 15 Oct '10

15 Oct '10

Hi -- Just upgraded to a new server (CentOS 5) and the yum installed version of syslog-ng (2.1.4) is gagging on the previous version of syslog-ng.conf, which was working fine for syslog-ng (1.6.12). I commented out a couple of lines like @version and file() but find I can't get past sql(). The documentation says that for versions 2.x, sql() is only available for Premium version. Just want to verify that I will have to roll back to the old 1.6.12 version to have the sql() feature, which we are using quite a bit. Is that version still available? Thanks, Liam -- Liam Kirsher PGP: http://liam.numenet.com/pgp/

4 4

[Bug 52] CAP_DAC_OVERRIDE is missing when capabilities are enabled, unable to write to files created with the owner() option.
by bugzilla＠bugzilla.balabit.com 15 Oct '10

15 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=52 --- Comment #3 from Balazs Scheidler <bazsi(a)balabit.hu> 2010-10-15 22:09:19 --- sounds great. I'll try to get this integrated. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Re: [syslog-ng] error in syslog-ng 3.2alpha2: missing m4 macros directory
by Guillaume Rousse 15 Oct '10

15 Oct '10

Le 12/10/2010 11:37, Guillaume Rousse a écrit : > Le 06/10/2010 15:40, Balazs Scheidler a écrit : >> On Sat, 2010-10-02 at 23:01 +0200, Guillaume Rousse wrote: >>> Hello. >>> >>> configure.in has this macro: >>> AC_CONFIG_MACRO_DIR([m4]) >>> >>> However, the m4 directory doesn't exist. As a consequence, running >>> 'autoreconf' fails immediatly. I guess it's just missing, as previous >>> release had it. >> >> Right. I've fixed this in the git tree already. Could you try using a >> git snapshot? > I just tried, the tree still doesn't have any m4 directory... Forget this, I just understood it is dynamically created by autogen.sh script. [..] > Anyway, I just realized I didn't need to regenerate autotools toolchain, > as my patch doesn't impact any relevant file. Here it is, its goal is > just to ensure than syslog-ng doesn't create files directly under > $localstatedir directory, which is actually /var, but only under its own > specific directory, $localstatedir/lib/syslog-ng. It avoid the need to > override configure defaults through --localstatedir option, as does the > spec file shipped in the tree. Actually, the patch itself only deals with localestatedir, whereas the same issue appears with other shared directories (datadir and sysconfdir). By default, with a standard prefix, all those locations matches shared directories (/etc, /usr/share, /var), forcing packagers to override all of them to isolate syslog-ng specific content. Having all of them correct by default would be nicer for FHS-addicts. However, it also allow people using a distinct prefix (/opt/syslog-ng for instance) to avoid uselessly long path for different component (/opt/syslog-ng/share instead of /opt/syslog-ng/share/syslog-ng). I guess this is the goal here. -- BOFH excuse #368: Failure to adjust for daylight savings time.

2 1

Re: [syslog-ng] error in syslog-ng 3.2alpha2: missing m4 macros directory
by Guillaume Rousse 15 Oct '10

15 Oct '10

Le 06/10/2010 15:40, Balazs Scheidler a écrit : > On Sat, 2010-10-02 at 23:01 +0200, Guillaume Rousse wrote: >> Hello. >> >> configure.in has this macro: >> AC_CONFIG_MACRO_DIR([m4]) >> >> However, the m4 directory doesn't exist. As a consequence, running >> 'autoreconf' fails immediatly. I guess it's just missing, as previous >> release had it. > > Right. I've fixed this in the git tree already. Could you try using a > git snapshot? I just tried, the tree still doesn't have any m4 directory... > No need to use a git client, the gitweb interface is able to generate a > tar.gz on the fly: > > http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=snapshot;h=master > > In order to make syslog-ng compile from the git snapshot, rather than a > tar.gz, you'll a couple more tools, like (IIRC): > > * bison 2.4.1 > * automake/autoconf/libtool > > Once you have those you should run ./autogen.sh from the source tree, > then you'll have everything starting with a configure script to compile > it. Excepted the hack on PKG_CONFIG minimal version, autoreconf should work as well. Anyway, I just realized I didn't need to regenerate autotools toolchain, as my patch doesn't impact any relevant file. Here it is, its goal is just to ensure than syslog-ng doesn't create files directly under $localstatedir directory, which is actually /var, but only under its own specific directory, $localstatedir/lib/syslog-ng. It avoid the need to override configure defaults through --localstatedir option, as does the spec file shipped in the tree. -- BOFH excuse #334: 50% of the manual is in .pdf readme files

2 1