[syslog-ng] [Bug 163] New: afmongo does not send log

[bugzilla at bugzilla.balabit.com]

https://bugzilla.balabit.com/show_bug.cgi?id=163

           Summary: afmongo does not send log
           Product: syslog-ng
           Version: 3.4.x
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: critical
          Priority: unspecified
         Component: syslog-ng
        AssignedTo: bazsi at balabit.hu
        ReportedBy: whille at 163.com
Type of the Report: bug
   Estimated Hours: 0.0


i'm using syslog-ng to store mongo, about 2k logs/sec. About half an hour, no message sent out.

trying to locate it, I set log_fifo_size(16), it's easy to reproduce the bug.

i found codes related:

afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, gpointer user_data)
{
...
  g_mutex_lock(self->queue_mutex);
  self->last_msg_stamp = cached_g_current_time_sec ();
  queue_was_empty = log_queue_get_length(self->queue) == 0;
  g_mutex_unlock(self->queue_mutex);

  log_queue_push_tail(self->queue, msg, path_options);

  g_mutex_lock(self->suspend_mutex);
  if (queue_was_empty && !self->writer_thread_suspended)
    {
      g_mutex_lock(self->queue_mutex);
      log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL);
      g_mutex_unlock(self->queue_mutex);
    }
  g_mutex_unlock(self->suspend_mutex);
}

since in afmongodb_worker_thread(){
...
 g_mutex_unlock(self->suspend_mutex);

      g_mutex_lock(self->queue_mutex);
      if (log_queue_get_length(self->queue) == 0)
        {
          g_cond_wait(self->writer_thread_wakeup_cond, self->queue_mutex);
        }
      g_mutex_unlock(self->queue_mutex);
...
}

the whole rough process i think is, worker wait fo afmongodb_dd_queue_notify(), if 0 queue. But in multi-thread race condition, queue_was_empty offend
simplicity.


Here's a situation:

afmongodb_dd_queue()
afmongodb_dd_queue()
afmongodb_dd_queue() {
                // worker thread 
                afmongodb_worker_thread{ whille(){...            // run in a while circle
                afmongodb_worker_thread{ whille(){...
                afmongodb_worker_thread{ whille(){... all queue are empty now,
                 g_cond_wait(self->writer_thread_wakeup_cond, self->queue_mutex);

log_queue_push_tail
queue_was_empty is FALSE, so afmongodb_dd_queue_notify is not called.

afmongodb_dd_queue()     // a queue has been put to tail last time, so queue_was_empty will be still FALSE.



i'm trying
to get queue length later and judge it as <=1, instead of ==0, for a queue just put to tail. wonder if it work, though a bit ugly.

afmongodb_dd_queue(){
...
if(!self->writer_thread_suspended){
    g_mutex_lock(self->queue_mutex);
    if (log_queue_get_length(self->queue)<=1){
      log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL);
    }
    g_mutex_unlock(self->queue_mutex);
  }


-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.