We've recently been seeing interruptions or delays to logs going into Splunk.  I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing. 

Here are the stats:

SourceName;SourceId;SourceInstance;State;Type;Number

destination;d_spol;;a;processed;0

src.internal;s_sys#2;;a;processed;384

src.internal;s_sys#2;;a;stamp;1590690126

center;;received;a;processed;1925133

destination;d_mesg;;a;processed;384

destination;d_mail;;a;processed;2

destination;d_auth;;a;processed;3

destination;d_mlal;;a;processed;0

center;;queued;a;processed;1962257

source;pan_splunk;;a;processed;1924743

destination;d_file;;a;processed;1561759

destination;d_cron;;a;processed;1

global;scratch_buffers_count;;a;queued;150323855375

destination;#anon-destination0;;a;processed;362983

global;payload_reallocs;;a;processed;5

global;sdata_updates;;a;processed;24

global;scratch_buffers_bytes;;a;queued;1536

destination;d_boot;;a;processed;0

destination;d_kern;;a;processed;0

source;s_sys;;a;processed;390

global;msg_clones;;a;processed;4662233

destination;d_file_no_location;;a;processed;37125

global;internal_queue_length;;a;processed;0

Here is the source configuration:

source pan_splunk {

  network(port(514) transport("tcp") max-connections(100)

     flags(syslog-protocol) log-fetch-limit(100)

     log-iw-size(100000) log-msg-size(10485760)

  );

  network(port(514) transport("udp") max-connections(100)

     flags(syslog-protocol) log-fetch-limit(100)

     log-iw-size(100000) log-msg-size(10485760)

   );

  network(transport("tls") port(10514) max-connections(100)

     flags(syslog-protocol) log-iw-size(100000)

     log-fetch-limit(100) log-msg-size(10485760)

     tls(peer-verify(optional-untrusted)

       key_file("/etc/syslog-ng/cert.d/serverkey.pem")

       cert_file("/etc/syslog-ng/cert.d/servercert.pem")

       ca_dir("/etc/syslog-ng/ca.d"))

     );

};

Here is the destination configuration

destination d_file {

  file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log"

    create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750)

    owner("splunk") group("splunk") perm(0640)

    log-fifo-size(1000)

    disk-buffer(

      mem-buf-size(10000)

      disk-buf-size(2000000000)

      reliable(yes)

      dir("/var/log/remote/disk-buffer_dfile")

    )

    flags("threaded", "no-multi-line")

  );

};

This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference.  I would welcome any help you could provide.

Thanks,

-Mark