- syslog-ng - lists.balabit.hu

Multi-threaded in open source?
by Avleen Vig 25 Nov '11

25 Nov '11

Hi everyone, I'm running an infrastructure with several hundred hosts logging many hundreds of Gb of logs to a central server. Syslog-ng on the central server is using 100% CPU on a single core, and I see here: http://bazsi.blogs.balabit.com/2011/07/syslog-ng-multithreaded-performance/ that there is some multi-threading option available! Is this only in the PE version or is there an OSE edition with this feature?

5 6

[PATCH (3.4)] value-pairs: Fix the error handling in the command-line parser.
by Gergely Nagy 25 Nov '11

25 Nov '11

When an error occurs during command-line parsing, do not attempt to free argv: it's not owned by us anymore. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/value-pairs.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/lib/value-pairs.c b/lib/value-pairs.c index fcaa401..c58d294 100644 --- a/lib/value-pairs.c +++ b/lib/value-pairs.c @@ -465,7 +465,6 @@ value_pairs_new_from_cmdline (GlobalConfig *cfg, { value_pairs_free (vp); g_option_context_free (ctx); - g_free (argv); return NULL; } g_option_context_free (ctx); -- 1.7.7.3

2 1

[PATCH] logproto: Fix compilation on systems without IOV_MAX.
by Gergely Nagy 25 Nov '11

25 Nov '11

Since IOV_MAX is not defined on Hurd its usage is made conditional. Errors are handled by g_malloc0() which aborts on failure, so no return value has to be checked. Reported-By: Svante Signell <svante.signell(a)telia.com> Patch-From: Svante Signell <svante.signell(a)telia.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logproto.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/lib/logproto.c b/lib/logproto.c index 282064c..16ca0d7 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -420,9 +420,11 @@ log_proto_file_writer_new(LogTransport *transport, gint flush_lines) if (flush_lines == 0) /* the flush-lines option has not been specified, use a default value */ flush_lines = 1; +#ifdef IOV_MAX if (flush_lines > IOV_MAX) /* limit the flush_lines according to the current platform */ flush_lines = IOV_MAX; +#endif /* allocate the structure with the proper number of items at the end */ LogProtoFileWriter *self = (LogProtoFileWriter *)g_malloc0(sizeof(LogProtoFileWriter) + sizeof(struct iovec)*flush_lines); -- 1.7.7.3

2 1

[PATCH] logrewrite: Fix the logrewrite cloning functions.
by Gergely Nagy 25 Nov '11

25 Nov '11

Repair the rewrite clone funcions, to correctly clone the value_handle and conditions aswell. Without this, if we use rewrite in multiple logpaths, only the first will behave correctly, the cloned ones will all rewrite the $MESSAGE macro instead, because that's the default. Signed-off-By: Zoltan Fried <deirf(a)balabit.hu> Signed-off-By: Gergely Nagy <algernon(a)balabit.hu> --- lib/logrewrite.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/lib/logrewrite.c b/lib/logrewrite.c index d282f6f..a578439 100644 --- a/lib/logrewrite.c +++ b/lib/logrewrite.c @@ -165,6 +165,8 @@ log_rewrite_subst_clone(LogProcessPipe *s) cloned = (LogRewriteSubst *) log_rewrite_subst_new(self->replacement->template); cloned->matcher = log_matcher_ref(self->matcher); + cloned->super.value_handle = self->super.value_handle; + cloned->super.condition = self->super.condition; return &cloned->super.super.super; } @@ -227,6 +229,8 @@ log_rewrite_set_clone(LogProcessPipe *s) LogRewriteSet *cloned; cloned = (LogRewriteSet *) log_rewrite_set_new(self->value_template->template); + cloned->super.value_handle = self->super.value_handle; + cloned->super.condition = self->super.condition; return &cloned->super.super.super; } -- 1.7.7.3

2 1

[PATCH] Fixed compounded filter initialization.
by Szalay Attila 25 Nov '11

25 Nov '11

If a filter link to an other filter an init function called to set up this link. But if this filter is compunded then this callback is not called so the filter is not initialized but silently evaluated to false. For example in the following situation the f_message is always true: filter f_auth { facility(auth, authpriv); }; filter f_cron { facility(cron); }; filter f_mail { facility(mail); }; filter f_messages { not filter(f_auth) and not filter(f_cron) and not filter(f_mail); }; Signed-off-by: Szalay Attila <sasa(a)balabit.hu> --- lib/filter.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 39 insertions(+), 0 deletions(-)

2 1

preserving tags across the network
by Matt Zagrabelny 24 Nov '11

24 Nov '11

Hi, I am trying to ship a bunch of apache log files across the network and on the syslog-ng server side then break them into their individual files again. I am not sure the best way to do this, but it looks like tags might be helpful. On my apache system I have the following snippets: -----{syslog-ng client begin}----- source s_apache_access { file("/var/log/apache2/access.log" tags("main_access") ); }; source s_apache_other_vhost_access { file("/var/log/apache2/other_vhost.log" tags("other_vhost_access") ); }; destination d_server { syslog( "10.0.0.1" transport("tls") port(6514) tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) ); }; log { source(s_apache_access); source(s_apache_other_vhost_access); destination(d_server); }; -----{syslog-ng client end}----- On my server I have the following: -----{syslog-ng server begin}----- source s_tls { syslog( ip(0.0.0.0) port(6514) transport("tls") tls( peer-verify(required-trusted) ca_dir('/etc/syslog-ng/ssl/ca.d') key_file('/etc/syslog-ng/ssl/server.key') cert_file('/etc/syslog-ng/ssl/server.crt') ) max_connections(1000) keep_hostname(yes) ); }; filter f_main_apache_access { tags("main_access"); }; destination d_main_access { file("/var/log/apache2/access.log"); }; log { source(s_tls); filter(f_main_apache_access); destination(d_main_access); }; -----{syslog-ng server end}----- It doesn't look like the "tag" is being preserved through the network. Does anyone know if tags is the right mechanism for splitting apart (filtering) the aggregate stream coming in on the server? Any other advice would be greatly appreciated. Thanks! -matt zagrabelny

4 6

How to detect loss of connection of remote syslog-ng client.
by Pramod Pillai 24 Nov '11

24 Nov '11

Hi I am using syslog-ng server on Solaris box and the syslog-ng clients are running on Linux box. When client ( linux ) box crashes the streaming stops. But the syslog-ng server assumes that the client is connected and keeps the connection. Is there any way to detect this condition and close the socket connection to the client which no longer exists. Regards Pramod

2 2

[PATCH 0/1]: value-pairs: key rewriting support
by Gergely Nagy 24 Nov '11

24 Nov '11

After going over the code a few more times, and after a quick chat with Bazsi, I cleaned up the patch considerably. So much so, that I'm actually starting to like it! Functionality remained the same, but the code behind it is shorter and contains less magic. Instead of playing tricks with both scratch buffers and allocating gchar* strings all over the place, the transformation functions now operate directly on scratch buffers, and we steal the results. The patch coming in a follow-up message is something I consider ready for review. The only thing left to do is adding a couple of test cases (there's already a simple one included, but I'd like a few more).

1 1

Enterprise Log Search and Archive (ELSA) Beta Available
by Martin Holste 23 Nov '11

23 Nov '11

The new auto-installer script is working well enough that I think most people should be able to get the beta of ELSA installed now. I put up a short post with the details and screenshots on my blog here: http://ossectools.blogspot.com/2011/11/elsa-beta-available.html . >From the project page (http://code.google.com/p/enterprise-log-search-and-archive/) Features: * High-volume receiving/indexing (a single node can receive > 30k logs/sec, sustained) * Full Active Directory/LDAP integration for authentication, authorization, email settings * Instant ad-hoc reports/graphs on arbitrary queries even on enormous data sets * Email alerting, scheduled reports * Plugin architecture for web interface * Distributed architecture for clusters * Ships with normalization for some Cisco logs, Snort/Suricata, Bro, and Windows via Eventlog-to-Syslog or Snare Many thanks to Peter for helping me test the auto-installer, as well as all of Balabit for providing such an amazingly fast syslog parser with PatternDB. Please let me know if you run into issues installing. Ubuntu, openSUSE, and CentOS have been tested, but variations of those distros should work fine.

1 0

Query regarding syslog-ng.conf and creation of logging files
by Ankit Mahawar 23 Nov '11

23 Nov '11

Hi, Please help regarding doubts of syslog-ng.conf file 1 . The destination files specified in syslog-ng.conf is created by syslog-ng ?? 2. If they are created by syslog-ng then , are they created only when the log gets filtered for that destination file ? 3 . Does this happen in syslog as well,because I think in syslog the file gets created even before logging into the file I am new to syslog utilities and it is just my assumption that the files might be created using the conf file ,because when I killed syslog daemon and restarted it ,the file I removed got created again but not in the case of syslog-ng. Please help regarding this issue . Thanks Ankit =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you

5 5