- syslog-ng - lists.balabit.hu

[PATCH] afmongodb: Fix a deadlock at deinit time.
by Gergely Nagy 03 Feb '12

03 Feb '12

There is a race condition between the writer thread and stop_threads (via deinit): if deinit sends the signal just at the wrong time, then the worker thread willl never wake up, as the main thread is already waiting for a join to finish. The fix is to lock the queue_mutex before signalling, that eliminates the race. This patch does just that. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 42c140e..dade669 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -1,6 +1,6 @@ /* - * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary - * Copyright (c) 2010-2011 Gergely Nagy <algernon(a)balabit.hu> + * Copyright (c) 2010-2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2010-2012 Gergely Nagy <algernon(a)balabit.hu> * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License version 2 as published @@ -403,7 +403,9 @@ static void afmongodb_dd_stop_thread (MongoDBDestDriver *self) { self->writer_thread_terminate = TRUE; + g_mutex_lock(self->queue_mutex); g_cond_signal(self->writer_thread_wakeup_cond); + g_mutex_unlock(self->queue_mutex); g_thread_join(self->writer_thread); } -- 1.7.8.3

2 1

Bazsi's blog: syslog-ng flexibility improvements
by Balazs Scheidler 03 Feb '12

03 Feb '12

syslog-ng flexibility improvements syslog-ng is often referred as a very flexible application when it comes to processing logs. Over the years however, I began to feel that some things are a bit more difficult to achieve in the configuration language than it should be. For instance it is sometimes too rigid when you need a combination of parsers (patterndb with db-parser) and rewrite rules to achieve the goal you wanted. Parsers and rewrite rules are distinct part of the configuration, it is not possible to combine them into a single functionality. Also, declaring objects first and then referencing them later, makes the configuration easy to read, however sometimes it is quite cumbersome, when you only need to invert the result of an already existing filter. To solve this situation, I’ve set out to implement an idea I had on mind for some time now. It is quite difficult to describe the feature in clear and concise words, as it is a combination of various changes that together makes syslog-ng configuration more flexible and easier to use, without sacrificing readability. Curious? Please read on. In-line objects Perhaps the simplest of all features is that you can now define the contents of a given object right on the spot, without having to use a separate statement. For example, earlier you had to write: log { source(s_local); filter(f_postfix); destination(d_postfix); }; Sometimes, f_postfix filter is only used once and is trivial. This can now be written as: log { source(s_local); filter { program("^postfix/"); }; destination(d_postfix); }; Furthermore both the source() and destination() options can be written in-line, you simply use braces instead of parentheses. The same functionality applies to everything: sources, destinations, filters, parsers and rewrite rules. Junctions A limited form of junctions has been supported since syslog-ng 3.0 in the form of “embedded log statements”, which has been generalized now. Within syslog-ng, when a message is received it is dispatched to a log processing path or pipeline, which carries out the task at hand. A junction is a point in the log processing path where the processing is performed on multiple independent branches, each doing its own specific thing with the message. The limited functionality in 3.0 only allowed the processing tree to split (or fork) into independent branches, each of the branches was a “sink”, where processing also ended. Configuration example: log { source(s_all); filter(f); log { filter(f1); destination(d1); }; log { filter(f2); destination(d2); }; }; This sample forks the processing path into two branches starting with the “log” keyword within the top-level log statement. The first branch evaluates the filter f1 and the writes matching messages to the d1 destination, effectively sending all messages that match (f AND f1) to d1. Likewise, d2 receives all messages that match (f AND f2). The limitation of the embedded log statement concept was simple: it could only be listed at the very end of a log statement, and the end-result of the branches couldn’t be processed further. Effectively the message at the end of each branch “fell off”. Junctions on the other hand makes it possible to do things to messages once the branches converge to the same point again. Repeating the sample above, it is now possible to write: log { source(s_all); filter(f); junction { log { filter(f1); destination(d1); }; log { filter(f2); destination(d2); }; }; destination(d_all); }; The new thing is that you can now add processing after the branches finish their processing. A bit more useful example would be: log { source(s_apache_files); source(s_syslog); junction { log { filter(f_apache_files); rewrite(r_apache_remove_file_header); parser(p_apache); flags(final); }; log { filter(f_apache_syslog); parser(p_apache); flags(final); }; }; destination(d_files); }; This example does an alternative processing of incoming logs based on where the message came from. Everything is a log expression This feature is probably the most complicated, however provides very nice properties and expressiveness to the configuration. From now on, not just the well known log statement allows the specification of log processing rules, but all the objects in the syslog-ng configuration file can use the same expressive power. It is now possible to use embedded log statements, junctions and in-line object definitions within source, destination, filter, rewrite and parser definitions. Huh, you could ask: what does it bring to me as a benefit? Well, until now, objects of different types were separate entities, connected using log statements, with this change a source can also specify a rewrite rule and that combination used as a log source in a log statement. For instance, a usual source definition looked like this: source s_apache { file("/var/log/apache/error.log"); }; If you wanted to process this log file in a specific way, you needed to define the accompanying processing rules (parsers and rewrite expressions) and combine them in a log statement. But how about this: source s_apache { log { source { file("/var/log/apache/error.log"); }; parser(p_apache_parser); }; }; }; log { source(s_apache); ... }; Can you see? The s_apache source used a file source and the reference of a specific parser and all messages read from the apache error log file would be processed by that parser. The log statement is just as simple as if s_apache would be a “normal” source definition. This feature allows pairing the essential log preprocessing functionality very close to the source itself, making it very easy to write and read the log statements. As an added bonus, it becomes very easy to distribute application specific source & parser definitions as an SCL configuration snippet. Where? This stuff is available in the syslog-ng 3.4 git tree, on master. It passes the included regression test, so it is at least dogfoodable. The nice thing about the implementation is that it only slightly increased the code size, but brought a lot of new features. If you have trouble getting the code from git, let me know, I’m willing to create an alpha release, so that it becomes easier to play with it. Feedback I see a lot of potential in this functionality, however my examples may have not been the best ones. I would really appreciate any kind of feedback, please be sure to send those to the syslog-ng mailing list or post me as a private email.

4 13

Patch for tests/loggen.c (bind src addr and full SendQ sleep)
by Russell Tait Milne 03 Feb '12

03 Feb '12

A colleague of mine has patched the loggen utility to allow for: - binding to a particular source address - adding a usleep() when the SendQ is full This patch was applied and tested on syslog-ng-3.3.3 found in the FreeBSD ports tree. I apologize in advance if this is not the proper way to submit a patch. (See attached patch) Best, Russ This communication, including any attachments, does not necessarily represent official policy of Seccuris Inc. Please see http://www.seccuris.com/Contact-PrivacyPolicy.htm for further details about Seccuris Inc.'s Privacy Policy. If you have received this communication in error, please notify Seccuris Inc. at info(a)seccuris.com or at 1-866-644-8442.

2 3

Dumping syslog-ng log files to mysql db
by Anup Shetty 02 Feb '12

02 Feb '12

Hello, With recent addition of MYSQL DB to my logging infrastructure, would I be able to dump my previous logs collected from the by syslog-ng on the database? If I configure the source to be the local log file that has logs collected from devices over the network and the destination to be the DB, would I be loosing the source IP address when I log these to the DB and instead get the source of the message to be localhost or 127.0.0.1 instead of the original source? -- Thanks and regards, Anup

2 1

parsing what gets into syslog from windows machines.
by greg whynott 02 Feb '12

02 Feb '12

Hello, We have all our windows servers/controllers send system messages (using snare) to our central log server which is a linux machine running syslog-ng. They are very verbose, multi line messages. Is there a way to have syslog-ng trim these down before the are wrote out to the log file? Or perhaps you know of another method I should be looking at? Thanks very much. here is a typical windows server message: Feb 1 16:08:31 10.111.1.11 MSWinEventLog 1 Security 5043398 Wed Feb 01 16:08:29 2012 4624 Microsoft-Windows-Security-Auditing NT AUTHORITY\ANONYMOUS LOGON N/A Success Audit AD02.domain.com Logon An account was successfully logged on. Subject: Security ID: S-1-0-0 Account Name: - Account Domain: - Logon ID: 0x0 Logon Type: 3 New Logon: Security ID: S-1-5-7 Account Name: ANONYMOUS LOGON Account Domain: NT AUTHORITY Logon ID: 0x45ab55d Logon GUID: {00000000-0000-0000-0000-000000000000} Process Information: Process ID: 0x0 Process Name: - Network Information: Workstation Name: AC010 Source Network Address: 10.101.2.44 Source Port: 60711 Detailed Authentication Information: Logon Process: NtLmSsp Authentication Package: NTLM Transited Services: - Package Name (NTLM only): NTLM V1 Key Length: 128 This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network). The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on. The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The authentication information fields provide detailed information about this specific logon request. - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of the generated session key. This will be 0 if no session key was requested. 5040030 I'd like to parse out the redundant data which is in every similar message, several times a second... the ultimate goal would be for that message to be trimmed down to: Feb 1 16:08:31 10.111.1.11 5043398 4624 Microsoft-Windows-Security-Auditing NT AUTHORITY\ANONYMOUS LOGON N/A Success Audit AD02.domain.com Logon Workstation Name: AC110 Source Network Address: 10.101.2.110 Source Port: 60711 or similar... I can define aregx to do this, but I am not sure where the best place would be to intercept and parse this before it is wrote out, or if syslog-ng has the parts to allow this to happen. thanks for your time, greg

3 3

syslog-ng 3.3
by Peter Czanik 01 Feb '12

01 Feb '12

Hello, As syslog-ng 3.3 was released for a couple of months now, I'd like to know, if you already switched to this version. If you already use 3.3, please share your success story! Why did you upgrade (which feature)? Size of installation? Or any other info you find interesting. If you still use an earlier release, please let us know, what keeps you back from upgrading! Not yet available as a package for your platform? Something is broken, what previously worked? Something else? For those, who don't want to share their story with the rest of the world, you could still help our work by answering these questions in a private e-mail directly to me. And finally, here is a teaser for those, who did not upgrade yet: what's new in syslog-ng since 2.0: http://czanik.blogs.balabit.com/2012/02/whats-new-in-syslog-ng-3-x/ Thank you for your help, Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

4 3

Receives multiple messages in one packet
by 大町靖 01 Feb '12

01 Feb '12

Hi experts, I want to receive multiple messages in one packet by using syslog-ng OSE. I checked the result after getting two messages in one packet. (The message format is that <PRI> Message1 \r\n<PRI> Message2\rn .) Message1 <PRI> Message2 I guess that syslog-ng recognize the first <PRI> as <PRI> ,though not recognise the second <PRI>. It recognized the second <PRI> message. If I want to fix this issue, what should I do. Please let me know. Thanks, Yasu

2 1

kernel logging feature requests
by Patrick H. 01 Feb '12

01 Feb '12

While setting up a new server at home I've come across 2 feature requests that would be fairly nice to have (and not that hard to implement I would think). 1) When the 'kernel' flag is set on a file() source (like for reading /proc/kmsg), look for the printk time (eg "[ 1234.567890]") and calculate when the message would have occurred instead of just using when the line was read off the file. Basically check to see the number of seconds the system has been up, subtract the printk time, and then subtract that from current time. 2) I grab all kernel messages with priority of crit or higher and send it to the usertty() destination, but this destination doesnt support templates. It'd be nice to be able to define the template. I mostly just want to change the time format and remove the hostname (since these will only come from localhost on my setup).

4 8

Relicensing syslog-ng man pages
by Fekete Robert 01 Feb '12

01 Feb '12

Hi, to make the syslog-ng manual pages more consistent with the source code and also to make the life of package maintainers a bit easier, we have relicensed the man pages of syslog-ng under the GPLv2 license. Bazsi will add source of the manual pages (docbook xml) to the syslog-ng source code as well. Please find attached the xml sources of the current man pages of syslog-ng OSE 3.3. I will also update the adminguide to include the relicensed man pages. Generating the man pages from the sources is relatively simple, just dump them into a directory and execute the following: (you need to have xsltproc and docbook-xsl installed, and might have to adjust the path for your environment) for i in *.xml; do xsltproc /usr/share/xml/docbook/stylesheet/docbook-xsl/manpages/docbook.xsl $i; done Sidenote: the syslog-ng.8 man page contained outdated information about chrooting syslog-ng, I have corrected that as well. Regards, Robert

2 1

afsql error
by Hery Fanomezantsoa 31 Jan '12

31 Jan '12

When I'm starting syslog-ng 3.3.1 I got an error saying an afsql error. I compiled it with libdbi and libdbi-drivers but when I do an ldd syslog-ng there is no libdbi found in the link. What's the matter?

2 5