[syslog-ng] [Bug 260] exit with code 134

[bugzilla at bugzilla.balabit.com]

https://bugzilla.balabit.com/show_bug.cgi?id=260





--- Comment #7 from Balazs Scheidler <bazsi at balabit.hu>  2013-11-24 20:55:01 ---
Hi,

Now I had some time to review to stuff you sent me. Thanks for the efforts.

As it seems, strace is unusable, the 133 exit code you have seen indicates a SIGTRAP, which is the signal raised when a breakpoint is hit. Since the 
only remotely debugger-like is strace, it seems we have hit some kind of strace issue, one that injects a SIGTRAP to the child. Could be a race or something.

But fortunately, I have found a clue in one of the core dumps.

One of the dumps was a segfault and not an abort:

Core was generated by `/usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.server.c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fa668c88381 in __strlen_sse2 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install UVic-syslog-ng-3.4.4-99.el6.x86_64
(gdb) bt
#0  0x00007fa668c88381 in __strlen_sse2 () from /lib64/libc.so.6
#1  0x00007fa668c88076 in strdup () from /lib64/libc.so.6
#2  0x00007fa6697ed94b in evt_tag_str (tag=0x7fa66a574d00 "encoding", value=0x6479786f72 <Address 0x6479786f72 out of bounds>) at evttags.c:76
#3  0x00007fa66a5341bd in log_proto_server_options_validate (options=0x14cd298) at logproto-server.c:138
#4  0x00007fa66a539c10 in log_proto_server_validate_options (self=0x97d1b0) at logproto-server.h:82
#5  0x00007fa66a53aecf in log_reader_init (s=0xc4f040) at logreader.c:638
#6  0x00007fa6683dab9f in log_pipe_init (s=0xc4f040, cfg=0x0) at ../../lib/logpipe.h:253
#7  0x00007fa6683db068 in afsocket_sc_init (s=0xe45280) at afsocket-source.c:149
#8  0x00007fa6683dab9f in log_pipe_init (s=0xe45280, cfg=0x0) at ../../lib/logpipe.h:253
#9  0x00007fa6683dbf41 in afsocket_sd_init (s=0xaa8230) at afsocket-source.c:548
#10 0x00007fa66a5243b0 in log_pipe_init (s=0xaa8230, cfg=0x1d750e0) at logpipe.h:253
#11 0x00007fa66a526111 in cfg_tree_start (self=0x1d75248) at cfg-tree.c:1064
#12 0x00007fa66a51fec1 in cfg_init (cfg=0x1d750e0) at cfg.c:220
#13 0x00007fa66a5435ae in main_loop_reload_config_apply () at mainloop.c:501
#14 0x00007fa66a5434b8 in main_loop_io_worker_sync_call (func=0x7fa66a54355b <main_loop_reload_config_apply>) at mainloop.c:456
#15 0x00007fa66a543850 in main_loop_reload_config_initiate () at mainloop.c:574
#16 0x00007fa66a5438ae in sig_hup_handler (s=0x0) at mainloop.c:608
#17 0x00007fa66a56f8e8 in iv_signal_event (_this=0x7fa66a7a8ce0) at iv_signal.c:170
#18 0x00007fa66a56dc95 in iv_event_raw_got_event (_this=0x7fa66a7a8d20) at iv_event_raw_posix.c:89
#19 0x00007fa66a56e354 in iv_fd_poll_and_run (st=0x7b91a0, to=0x7fffa91ed630) at iv_fd.c:163
#20 0x00007fa66a56f2bb in iv_main () at iv_main_posix.c:117
#21 0x00007fa66a543c72 in main_loop_run () at mainloop.c:736
#22 0x000000000040187b in main ()

And this is one of the possible symptoms of a bug I have fixed in 3.4.5, more specifically this one:

https://bugzilla.balabit.com/show_bug.cgi?id=253

I did find the root cause of that one because of one of the valgrind runs that you made, but since it wasn't reported by you, the fix is not in your
local syslog-ng build.

That bug is a potential use-after-free and as such it may actually cause the SIGABRTs by corrupting memory.

So, it'd make sense to apply this patch too as it's definitely affecting you, but the segfault is not triggered:

$ git show 861489c4b3a895ca88f9cdbb98b351f1c5a330cf
commit 861489c4b3a895ca88f9cdbb98b351f1c5a330cf
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Sat Nov 2 20:19:12 2013 +0100

    LogProtoServer: fixed a use-after-free after reload

    self->options is a borrowed memory area, managed by the owning driver
    that constructed the given LogProtoServer instance. During reload however,
    the driver object itself is freed, while the LogProtoServer instance
    is kept alive, thereby causing the options pointer to point into the
    void.

    The solution is to update the options pointer whenever the owner is changed,
    chaining into a similar solution in the LogReader code which does exactly
    that.

    Signed-off-by: Balazs Scheidler <bazsi at balabit.hu>

diff --git a/lib/logproto-server.h b/lib/logproto-server.h
index 6188f65..969a807 100644
--- a/lib/logproto-server.h
+++ b/lib/logproto-server.h
@@ -82,6 +82,12 @@ log_proto_server_validate_options(LogProtoServer *self)
     return log_proto_server_options_validate(self->options);
 }

+static inline void
+log_proto_server_set_options(LogProtoServer *self, const LogProtoServerOptions *options)
+{
+  self->options = options;
+}
+
 static inline gboolean
 log_proto_server_prepare(LogProtoServer *s, gint *fd, GIOCondition *cond)
 {
diff --git a/lib/logreader.c b/lib/logreader.c
index 9a60345..c498ae1 100644
--- a/lib/logreader.c
+++ b/lib/logreader.c
@@ -697,6 +697,8 @@ log_reader_set_options(LogPipe *s, LogPipe *control, LogReaderOptions *options,
   self->control = control;

   self->options = options;
+  if (self->proto)
+    log_proto_server_set_options(self->proto, &self->options->proto_options.super);
 }

 /* run in the main thread in reaction to a log_reader_reopen to change


I've also reviewed the rest of the core files, but they seem to be very similar to the first crash. A different fd, but still 
the ivykis POLL mask is different from reality, which causes the abort within ivykis.


-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.