[syslog-ng] Logs not making it to Splunk, high scratch buffers count

[Faine, Mark R. (MSFC-IS40)[NICS]]

We've recently been seeing interruptions or delays to logs going into Splunk.  I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing.


Here are the stats:


SourceName;SourceId;SourceInstance;State;Type;Number
destination;d_spol;;a;processed;0
src.internal;s_sys#2;;a;processed;384
src.internal;s_sys#2;;a;stamp;1590690126
center;;received;a;processed;1925133
destination;d_mesg;;a;processed;384
destination;d_mail;;a;processed;2
destination;d_auth;;a;processed;3
destination;d_mlal;;a;processed;0
center;;queued;a;processed;1962257
source;pan_splunk;;a;processed;1924743
destination;d_file;;a;processed;1561759
destination;d_cron;;a;processed;1
global;scratch_buffers_count;;a;queued;150323855375
destination;#anon-destination0;;a;processed;362983
global;payload_reallocs;;a;processed;5
global;sdata_updates;;a;processed;24
global;scratch_buffers_bytes;;a;queued;1536
destination;d_boot;;a;processed;0
destination;d_kern;;a;processed;0
source;s_sys;;a;processed;390
global;msg_clones;;a;processed;4662233
destination;d_file_no_location;;a;processed;37125
global;internal_queue_length;;a;processed;0



Here is the source configuration:

source pan_splunk {
  network(port(514) transport("tcp") max-connections(100)
     flags(syslog-protocol) log-fetch-limit(100)
     log-iw-size(100000) log-msg-size(10485760)
  );
  network(port(514) transport("udp") max-connections(100)
     flags(syslog-protocol) log-fetch-limit(100)
     log-iw-size(100000) log-msg-size(10485760)
   );
  network(transport("tls") port(10514) max-connections(100)
     flags(syslog-protocol) log-iw-size(100000)
     log-fetch-limit(100) log-msg-size(10485760)
     tls(peer-verify(optional-untrusted)
       key_file("/etc/syslog-ng/cert.d/serverkey.pem")
       cert_file("/etc/syslog-ng/cert.d/servercert.pem")
       ca_dir("/etc/syslog-ng/ca.d"))
     );
};



Here is the destination configuration

destination d_file {
  file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log"
    create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750)
    owner("splunk") group("splunk") perm(0640)
    log-fifo-size(1000)
    disk-buffer(
      mem-buf-size(10000)
      disk-buf-size(2000000000)
      reliable(yes)
      dir("/var/log/remote/disk-buffer_dfile")
    )
    flags("threaded", "no-multi-line")
  );
};

This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference.  I would welcome any help you could provide.

Thanks,
-Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200528/ab09506a/attachment.html>