- syslog-ng - lists.balabit.hu

[PATCH] gprocess: Implement stricter CAP_SYSLOG checking.
by Gergely Nagy 27 Sep '11

27 Sep '11

Some distributions ship libcap with headers generated from the kernel sources, instead of the headers of libcap itself, resulting in the headers and the library diagreeing about what capabilities are supported. For this reason, before deciding whether CAP_SYSLOG is supported, verify that libcap can actually parse it, and fall back to CAP_SYS_ADMIN if it can't. The benefit of this is that even if ran on such a system, syslog-ng will still continue to work (albeit triggering a kernel warning) instead of terminating. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/lib/gprocess.c b/lib/gprocess.c index e9315d9..b0925fb 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -328,6 +328,14 @@ g_process_check_cap_syslog(void) if (ret == -1) return FALSE; + ret = cap_from_name("cap_syslog", NULL); + if (ret == -1) + { + fprintf (stderr, "CAP_SYSLOG seems to be supported by the system, but " + "libcap can't parse it. Falling back to CAP_SYS_ADMIN!\n"); + return FALSE; + } + have_capsyslog = TRUE; return TRUE; } -- 1.7.6.3

2 1

[PATCH] afmongodb, afsql: Fix a race condition in the queue handling.
by Gergely Nagy 27 Sep '11

27 Sep '11

On occassions where we received a message, woke up the worker thread, then received two two others before the worker finished, we had a high chance of triggering an assertion, because we ended up calling the insert before the parallel_push stuff managed to wake up the thread and clear the callbacks. Now we explicitly clear parallel_push before popping the queue head, for the simple reason that we do not need it at that point anymore. Many thanks to Balazs Scheidler <bazsi(a)balabit.hu> for pointing out what happens. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 1 + modules/afsql/afsql.c | 1 + 2 files changed, 2 insertions(+), 0 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 0f01f05..b2bd615 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -263,6 +263,7 @@ afmongodb_worker_insert (MongoDBDestDriver *self) afmongodb_dd_connect(self, TRUE); g_mutex_lock(self->queue_mutex); + log_queue_reset_parallel_push(self->queue); success = log_queue_pop_head(self->queue, &msg, &path_options, FALSE, FALSE); g_mutex_unlock(self->queue_mutex); if (!success) diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index 20db131..6fbd656 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -682,6 +682,7 @@ afsql_dd_insert_db(AFSqlDestDriver *self) else { g_mutex_lock(self->db_thread_mutex); + log_queue_reset_parallel_push(self->queue); success = log_queue_pop_head(self->queue, &msg, &path_options, (self->flags & AFSQL_DDF_EXPLICIT_COMMITS), FALSE); g_mutex_unlock(self->db_thread_mutex); if (!success) -- 1.7.6.3

2 1

How to send tracebacks through syslog-ng?
by Luis Pugoy 27 Sep '11

27 Sep '11

Hello, We have a Python process that logs into a file. Included in the file are tracebacks. We would like to send these log entries through syslog-ng. To solve the problem of newlines in tracebacks, our plan was to modify the Python log formatter to replace all newlines with the Unicode character for line separator, U+2028. Syslog-ng then tails the log file and sends the entries to the central logging server through tcp, which is setup to replace the Unicode character with a newline and record it in a file. The problem with this approach is that the Unicode character does not seem to be persisted in the file. A lot of rewrite rules were tried to replace the Unicode character. These are: rewrite r_newlines { subst("\p{Zl}", " test ", value("MESSAGE") type("pcre") flags("utf8" "global")); }; rewrite r_newlines { subst("\x{2028}", " test ", value("MESSAGE") type("pcre") flags("utf8" "global")); }; rewrite r_newlines { subst("\u2028", " test ", value("MESSAGE") flags("global")); }; but none of them worked. The file source was also set to have UTF-8 encoding, as well as the tcp source (from the view of the central logging server) but these didn't work either. The problem seems to be either the rewrite rule or the Unicode character is not persisted in the log file. We also tried logging to syslog directly from Python (through /dev/log) but that also failed. Replacing newlines with a string such as " [NEWLINE] " and using syslog-ng to rewrite that to newlines worked, so it seems to be a problem with the Unicode character. This solution is a last resort though since it could be problematic. Thanks in advance.

3 3

Load balancing in syslog-ng(verison 3.0)
by Jain, Vaibhav (GE Healthcare) 27 Sep '11

27 Sep '11

Hi, In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know how to do this ? Thanks Vaibhav

7 9

[Bug 133] New: syslog-ng-3.3. 0beta2 stops to receive messages from systemd activated /dev/log
by bugzilla＠bugzilla.balabit.com 27 Sep '11

27 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=133 Summary: syslog-ng-3.3.0beta2 stops to receive messages from systemd activated /dev/log Product: syslog-ng Version: 3.2.x Platform: PC URL: https://bugzilla.novell.com/show_bug.cgi?id=719102 OS/Version: Linux Status: NEW Severity: critical Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mt(a)suse.com Type of the Report: --- Estimated Hours: 0.0 Even all looks fine and syslog-ng-3.3.0beta2 starts to work correctly: > systemctl status syslog.service syslog.service - System Logging Service Loaded: loaded (/lib/systemd/system/syslog.service; enabled) Active: active (running) since Tue, 20 Sep 2011 14:08:31 +0200; 19h ago Process: 2317 ExecReload=/sbin/killproc -p /var/run/syslog-ng.pid -HUP /sbin/syslog-ng (code=exited, status=0/SUCCESS) Process: 740 ExecStart=/sbin/syslog-ng (code=exited, status=0/SUCCESS) Process: 738 ExecStartPre=/var/run/syslog-ng/addsockets (code=exited, status=0/SUCCESS) Process: 735 ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service (code=exited, status=0/SUCCESS) Main PID: 763 (syslog-ng) CGroup: name=systemd:/system/syslog.service ├ 762 supervising syslog-ng └ 763 /sbin/syslog-ng > lsof | grep /dev/log systemd 1 root 25u unix 0xffff8802211690c0 0t0 4225 /dev/log syslog-ng 762 root 3u unix 0xffff8802211690c0 0t0 4225 /dev/log syslog-ng 763 root 3u unix 0xffff8802211690c0 0t0 4225 /dev/log > ps ax | grep syslog 762 ? S 0:00 supervising syslog-ng 763 ? Ss 0:00 /sbin/syslog-ng > ls -l /proc/763/fd/ total 0 lr-x------ 1 root root 64 Sep 20 14:10 0 -> /dev/null l-wx------ 1 root root 64 Sep 20 14:10 1 -> /dev/null lrwx------ 1 root root 64 Sep 20 14:10 10 -> anon_inode:[eventfd] lrwx------ 1 root root 64 Sep 20 14:10 11 -> anon_inode:[eventfd] lrwx------ 1 root root 64 Sep 20 14:10 12 -> anon_inode:[eventfd] lrwx------ 1 root root 64 Sep 20 14:10 13 -> anon_inode:[eventfd] lrwx------ 1 root root 64 Sep 20 14:10 19 -> /dev/xconsole l-wx------ 1 root root 64 Sep 20 14:10 2 -> /dev/null lrwx------ 1 root root 64 Sep 20 14:10 3 -> socket:[4225] lrwx------ 1 root root 64 Sep 20 14:10 4 -> anon_inode:[eventpoll] l-wx------ 1 root root 64 Sep 20 14:10 5 -> pipe:[8583] lrwx------ 1 root root 64 Sep 20 14:10 6 -> socket:[8602] lrwx------ 1 root root 64 Sep 20 14:10 7 -> anon_inode:[eventfd] lrwx------ 1 root root 64 Sep 20 14:10 8 -> /var/lib/syslog-ng/syslog-ng.persist lr-x------ 1 root root 64 Sep 20 14:10 9 -> /proc/kmsg it stops receiving at the socket after some time (hours) and tries to send something to /dev/log gets blocked: > strace logger =========================================== ... socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 1 connect(1, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0 sendto(1, "<13>Sep 21 09:26:30 logger: ===="..., 71, MSG_NOSIGNAL, NULL, 0 --- blocks --- Syslog-ng shows no reaction on the data send to the systemd /dev/log socket, it just waits in epoll: > strace -f -s128 -p 763 Process 763 attached - interrupt to quit clock_gettime(CLOCK_MONOTONIC_RAW, {69938, 341467128}) = 0 epoll_wait(4, {}, 9, 21975) = 0 clock_gettime(CLOCK_MONOTONIC_RAW, {69960, 335242759}) = 0 epoll_wait(4, ^C and logs its statistics and -- MARK -- messages from time to time to /var/log/messages. After a "systemctl restart syslog.service", it starts reading messages from /dev/log again -- still the using the same socket node 4225 from systemd. I don't think some of our patches are responsible for this problem as all are from git and IMO unrelated. See also our bug reference in Url. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

Spoof-source option on Cygwin
by pedro castro 27 Sep '11

27 Sep '11

Hi we have Windows 2003 and Windows 2008 with cygwin. We receive events from several devices on syslog-ng server and redirect them to other systems. I want to use spoof-source option but i see that i need to compile syslog-ng from source. I have tried it but it fails in make. Does anyone have any idea? Regards

2 4

Re: [syslog-ng] syslog-ng Digest, Vol 77, Issue 34
by Luis Pugoy 27 Sep '11

27 Sep '11

> > Message: 1 > Date: Mon, 26 Sep 2011 18:07:00 +0800 > From: Luis Pugoy <lpugoy(a)insynchq.com> > Subject: Re: [syslog-ng] How to send tracebacks through syslog-ng? > (Jakub Jankowski) > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <CA+WGxEhGFX7V9C+71BgcU1TZsb21+FbzX5GdCZ=ZLPEDJ4fYFg(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > On Sun, Sep 25, 2011 at 1:57 AM, Luis Pugoy <lpugoy(a)insynchq.com> wrote: > > > Message: 4 > >> Date: Sat, 24 Sep 2011 09:31:34 +0200 (CEST) > >> From: Jakub Jankowski <shasta(a)toxcorp.com> > >> Subject: Re: [syslog-ng] How to send tracebacks through syslog-ng? > >> To: Syslog-ng users' and developers' mailing list > >> <syslog-ng(a)lists.balabit.hu> > >> Message-ID: <alpine.LNX.2.00.1109240928240.15891(a)bluzg.slackware.pl> > >> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > >> > >> On 2011-09-24, Luis Pugoy wrote: > >> > >> > rewrite r_rewrite_subst { subst("a*", "?", value("MESSAGE") > >> type("pcre") > >> > flags("utf8" "global")); }; > >> > > >> > I thought that it would replace 0 or more occurrences of 'a' with a > >> question > >> > mark. However, when the input is 'a', the output becomes 'a?'. It just > >> > appends a question mark at the end. 'aa' also becomes 'aa?'. What is > >> weirder > >> > still is that even if the input does not contain 'a', the question > mark > >> is > >> > still appended. So '1' becomes '1?'. > >> > >> Well, does "1" match /a*/ ? :) > >> It doesn't make any difference if that's /a*/, /b*/ or /.*/ - this kind > of > >> regexp matches everything, even an empty string. > >> > >> HTH > >> > >> -- > >> Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ > >> GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D > > > > > > Hello Jakub. > > > > Thanks for the reply. You're right in that "1" matches /a*/, but wouldn't > > the resulting replacement be "?" instead of "1?" ? Also, if I just use > /a/ > > and use "a" as input, it does not seem to match. The text is still > outputted > > as "a". > > > > I don't know the policy regarding bumping threads here, but I'd like to > bump > this. Can anyone confirm the results I'm getting? Is there also something > wrong with my understanding of pcre? I'm sorry if this is not allowed. > I was able to solve my initial problem. The "\p{Zl}" must be changed to "\\p}{Zl}".

1 0

[PATCH] correlation: explicit context timeout action
by Balint Kovacs 23 Sep '11

23 Sep '11

Hi, I'm trying to use patterndb correlation to find matching firewall connection startup and ending log messages and emit a consolidated message. I found that with high-volume load, the memory usage of syslog-ng climbs rapidly and it would be beneficial to be able to kick already ended events from the memory. Please find a patch below (against 3.4) that does this, I thought the easy place for the user would be the <action> part of the ending rule, this way it can even be specified when the context should be ended. Balint From: Balint Kovacs <blint(a)balabit.hu> Date: Wed, 21 Sep 2011 11:46:30 +0200 Subject: [PATCH] correlation: add action to explicitly end context If there is a message that can be associated with the end of a context, the below action can explicitly end it, reducing the memory footprint. Example: <action> <end-context /> </action> Signed-off-by: Balint Kovacs <blint(a)balabit.hu> --- modules/dbparser/patterndb-int.h | 3 ++- modules/dbparser/patterndb.c | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletions(-) diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..eff77e6 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -115,7 +115,8 @@ enum enum { RAC_NONE, - RAC_MESSAGE + RAC_MESSAGE, + RAC_EXPIRE }; /* a rule may contain one or more actions to be performed */ diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 343ef6d..72e97e0 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -577,6 +577,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con emit(genmsg, TRUE, emit_data); log_msg_unref(genmsg); break; + case RAC_EXPIRE: + if (context) + { + msg_debug("Expiring patterndb correllation context as directed by action", + evt_tag_str("last_rule", context->rule->rule_id), + evt_tag_long("remaining_context_count", g_hash_table_size(context->db->state)), + NULL); + g_hash_table_remove(context->db->state, &context->key); + timer_wheel_del_timer(db->timer_wheel, context->timer); + pdb_context_unref(context); + } + break; default: g_assert_not_reached(); break; @@ -921,6 +933,16 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name state->current_action->content_type = RAC_MESSAGE; state->current_message = &state->current_action->content.message; } + else if (strcmp(element_name, "end-context") == 0) + { + if (!state->in_action) + { + *error = g_error_new(1, 0, "Unexpected <end-context> element, it must be inside an action"); + return; + } + state->current_action->content_type = RAC_EXPIRE; + state->current_message = &state->current_action->content.message; + } } void -- 1.7.0.4

1 2

Re: [syslog-ng] Is it possible to replace a string with a new line using rewrite?
by Luis Pugoy 22 Sep '11

22 Sep '11

> > > Message: 4 > Date: Wed, 21 Sep 2011 21:20:44 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Is it possible to replace a string with a new > line using rewrite? > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87pqitr9bn.fsf(a)luthien.mhp> > Content-Type: text/plain > > Luis Pugoy <lpugoy(a)insynchq.com> writes: > > > I am trying to replace a string with a newline using rewrite. Is it > > possible? Thanks. > > rewrite r_nl { > subst("<insert regexp here>", > "\n", value("MSG")); > }; > > Or, if you want to replace the whole MSG with a newline: > > rewrite r_nlmsg { set("\n", value("MSG")); }; > > -- > |8] > > Thanks. I was using single quotes ('\n') so it wasn't working for me before. I didn't realize that there's a distinction between single and double quotes in syslog-ng.

1 0

Is it possible to replace a string with a new line using rewrite?
by Luis Pugoy 21 Sep '11

21 Sep '11

Hello, I am trying to replace a string with a newline using rewrite. Is it possible? Thanks.

2 1