mongodb stats crash
Hi, I have encountered a fixed crash in 3.5 master in the mongodb() driver, it aborts in stats_unregister_counter() it seems that afmongodb_format_stats_instance() returns different values in init() and deinit(). This is my config: mongodb(value-pairs(pair('foo', "$STAMP")) frac-digits(3)); This is the backtrace: [Thread 0x7ffff7fca700 (LWP 27493) exited] ** ERROR:/home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/stats.c:302:stats_unregister_counter: assertion failed: (sc && (sc->live_mask & (1 << type)) && &sc->counters[type] == (*counter)) Program received signal SIGABRT, Aborted. 0x00007ffff7275037 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007ffff7275037 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007ffff7278698 in __GI_abort () at abort.c:90 #2 0x00007ffff788e3b6 in g_assertion_message () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffff788e914 in g_assertion_message_expr () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffff7b69ef8 in stats_unregister_counter (source=533, id=0x61cd10 "d_local#0", instance=0x7ffff4d26c40 <persist_name.15015> "mongodb,127.0.0.1,27017,syslog,messages", type=SC_TYPE_STORED, counter=0x6e4cd0) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/stats.c:302 #5 0x00007ffff7b5b838 in log_threaded_dest_driver_deinit_method (s=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/logthrdestdrv.c:162 #6 0x00007ffff7b49014 in log_pipe_deinit (s=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/logpipe.h:268 #7 0x00007ffff7b4afad in cfg_tree_stop (self=0x613f30) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/cfg-tree.c:1080 #8 0x00007ffff7b44908 in cfg_deinit (cfg=0x613dc0) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/cfg.c:225 #9 0x00007ffff7b607b1 in main_loop_exit_finish () at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:646 #10 0x00007ffff7b60407 in main_loop_io_worker_sync_call (func=0x7ffff7b60799 <main_loop_exit_finish>) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:470 #11 0x00007ffff7b607d0 in main_loop_exit_timer_elapsed (arg=0x0) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:653 #12 0x00007ffff7b96568 in iv_run_timers () from /home/bazsi/zwa/install/syslog-ng-ose-3.5/lib/libsyslog-ng-3.5.0beta3.so #13 0x00007ffff7b99230 in iv_main () from /home/bazsi/zwa/install/syslog-ng-ose-3.5/lib/libsyslog-ng-3.5.0beta3.so #14 0x00007ffff7b60ba7 in main_loop_run () at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:796 #15 0x0000000000401a67 in main (argc=1, argv=0x7fffffffd648) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/syslog-ng/main.c:269 (gdb) Stepping through format_stats_instance() at init time: Breakpoint 3, afmongodb_dd_format_stats_instance (d=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/modules/afmongodb/afmongodb.c:189 189 MongoDBDestDriver *self = (MongoDBDestDriver *)d; (gdb) n 192 if (self->port == MONGO_CONN_LOCAL) (gdb) p self->port $1 = 0 (gdb) n 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) l 191 192 if (self->port == MONGO_CONN_LOCAL) 193 g_snprintf(persist_name, sizeof(persist_name), 194 "mongodb,%s,%s,%s", self->address, self->db, self->coll); 195 else 196 g_snprintf(persist_name, sizeof(persist_name), 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); 198 return persist_name; 199 } 200 (gdb) n 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 198 return persist_name; (gdb) p persist_name $2 = "mongodb,(null),0,syslog,messages", '\000' <repeats 991 times> At deinit time: Breakpoint 3, afmongodb_dd_format_stats_instance (d=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/modules/afmongodb/afmongodb.c:189 189 MongoDBDestDriver *self = (MongoDBDestDriver *)d; (gdb) n 192 if (self->port == MONGO_CONN_LOCAL) (gdb) p self->port $3 = 27017 (gdb) p self->address $4 = (gchar *) 0x61d290 "127.0.0.1" (gdb) n 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 198 return persist_name; (gdb) p persist_name $5 = "mongodb,127.0.0.1,27017,syslog,messages", '\000' <repeats 984 times> This patch solves for me, however I'm not 100% certain it's the proper fix and the other LogThreadedDestDriver instances might be affected too. diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 6504a63..b625c2c 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -516,8 +516,6 @@ afmongodb_dd_init(LogPipe *s) GlobalConfig *cfg = log_pipe_get_config(s); ValuePairsTransformSet *vpts; - if (!log_threaded_dest_driver_init_method(s)) - return FALSE; log_template_options_init(&self->template_options, cfg); @@ -566,6 +564,9 @@ afmongodb_dd_init(LogPipe *s) evt_tag_str("collection", self->coll), NULL); + if (!log_threaded_dest_driver_init_method(s)) + return FALSE; + log_threaded_dest_driver_start(&self->super); return TRUE; In case we're moving this to this point, I'd say that the threaded_dest_driver_start() could be moved into the log_threaded_dest_driver_init_method() function, can't it? Can you have a look? Thanks.
Balazs Scheidler <bazsi77@gmail.com> writes:
Hi,
I have encountered a fixed crash in 3.5 master in the mongodb() driver, it aborts in stats_unregister_counter()
[...]
This patch solves for me, however I'm not 100% certain it's the proper fix and the other LogThreadedDestDriver instances might be affected too.
This is the correct fix, and all the others are affected too. Can you commit it to 3.5 master, or shall I patch up the others myself? And yes, threaded_dest_driver_start() can be moved into log_threaded_dest_driver_init_method() afterwards. I'd do that in a separate patch though. Thanks! -- |8]
Gergely Nagy <algernon@balabit.hu> writes:
Balazs Scheidler <bazsi77@gmail.com> writes:
Hi,
I have encountered a fixed crash in 3.5 master in the mongodb() driver, it aborts in stats_unregister_counter()
[...]
This patch solves for me, however I'm not 100% certain it's the proper fix and the other LogThreadedDestDriver instances might be affected too.
This is the correct fix, and all the others are affected too. Can you commit it to 3.5 master, or shall I patch up the others myself? And yes, threaded_dest_driver_start() can be moved into log_threaded_dest_driver_init_method() afterwards. I'd do that in a separate patch though.
No, it's not the correct fix as it turns out, as init_method() will override functions set earlier, if moved to later. So init_method() needs to be a little smarter too. -- |8]
Gergely Nagy <algernon@balabit.hu> writes:
No, it's not the correct fix as it turns out, as init_method() will override functions set earlier, if moved to later. So init_method() needs to be a little smarter too.
I ended up doing something else: I split the log_dest_driver_init_method() and log_threaded_dest_driver_init_method() invocations, the modules do both now. They do the firt in the beginning, the other one at the end. The _start_driver() function was merged into log_threaded_dest_driver_init_method() too. All LogThrDestDriver users in syslog-ng itself and in the Incubator have been updated accordingly. -- |8]
I don't really like that both the derived and the ancestor class has provides the same method and they.need to be called separately. I think the one provided by threaded-dest-driver should be renamed to something else. On Oct 18, 2013 5:11 PM, "Gergely Nagy" <algernon@balabit.hu> wrote:
Gergely Nagy <algernon@balabit.hu> writes:
No, it's not the correct fix as it turns out, as init_method() will override functions set earlier, if moved to later. So init_method() needs to be a little smarter too.
I ended up doing something else: I split the log_dest_driver_init_method() and log_threaded_dest_driver_init_method() invocations, the modules do both now. They do the firt in the beginning, the other one at the end. The _start_driver() function was merged into log_threaded_dest_driver_init_method() too.
All LogThrDestDriver users in syslog-ng itself and in the Incubator have been updated accordingly.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Balazs Scheidler <bazsi77@gmail.com> writes:
I don't really like that both the derived and the ancestor class has provides the same method and they.need to be called separately.
I think the one provided by threaded-dest-driver should be renamed to something else.
I'll rename it before the next beta/rc release. Any suggestions? Perhaps log_threaded_dest_driver_start() ? -- |8]
Sounds about right. On Oct 20, 2013 1:45 PM, "Gergely Nagy" <algernon@balabit.hu> wrote:
Balazs Scheidler <bazsi77@gmail.com> writes:
I don't really like that both the derived and the ancestor class has provides the same method and they.need to be called separately.
I think the one provided by threaded-dest-driver should be renamed to something else.
I'll rename it before the next beta/rc release. Any suggestions? Perhaps log_threaded_dest_driver_start() ?
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Balazs Scheidler
-
Gergely Nagy