Logs not making it to Splunk, high scratch buffers count
We've recently been seeing interruptions or delays to logs going into Splunk. I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing. Here are the stats: SourceName;SourceId;SourceInstance;State;Type;Number destination;d_spol;;a;processed;0 src.internal;s_sys#2;;a;processed;384 src.internal;s_sys#2;;a;stamp;1590690126 center;;received;a;processed;1925133 destination;d_mesg;;a;processed;384 destination;d_mail;;a;processed;2 destination;d_auth;;a;processed;3 destination;d_mlal;;a;processed;0 center;;queued;a;processed;1962257 source;pan_splunk;;a;processed;1924743 destination;d_file;;a;processed;1561759 destination;d_cron;;a;processed;1 global;scratch_buffers_count;;a;queued;150323855375 destination;#anon-destination0;;a;processed;362983 global;payload_reallocs;;a;processed;5 global;sdata_updates;;a;processed;24 global;scratch_buffers_bytes;;a;queued;1536 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;0 source;s_sys;;a;processed;390 global;msg_clones;;a;processed;4662233 destination;d_file_no_location;;a;processed;37125 global;internal_queue_length;;a;processed;0 Here is the source configuration: source pan_splunk { network(port(514) transport("tcp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(port(514) transport("udp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(transport("tls") port(10514) max-connections(100) flags(syslog-protocol) log-iw-size(100000) log-fetch-limit(100) log-msg-size(10485760) tls(peer-verify(optional-untrusted) key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; Here is the destination configuration destination d_file { file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log" create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750) owner("splunk") group("splunk") perm(0640) log-fifo-size(1000) disk-buffer( mem-buf-size(10000) disk-buf-size(2000000000) reliable(yes) dir("/var/log/remote/disk-buffer_dfile") ) flags("threaded", "no-multi-line") ); }; This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference. I would welcome any help you could provide. Thanks, -Mark
participants (1)
-
Faine, Mark R. (MSFC-IS40)[NICS]