<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">We've recently been seeing interruptions or delays to logs going into Splunk. I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Here are the stats:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">SourceName;SourceId;SourceInstance;State;Type;Number<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_spol;;a;processed;0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">src.internal;s_sys#2;;a;processed;384<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">src.internal;s_sys#2;;a;stamp;1590690126<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">center;;received;a;processed;1925133<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_mesg;;a;processed;384<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_mail;;a;processed;2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_auth;;a;processed;3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_mlal;;a;processed;0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">center;;queued;a;processed;1962257<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">source;pan_splunk;;a;processed;1924743<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_file;;a;processed;1561759<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_cron;;a;processed;1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">global;scratch_buffers_count;;a;queued;<span style="background:yellow;mso-highlight:yellow">150323855375</span><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;#anon-destination0;;a;processed;362983<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">global;payload_reallocs;;a;processed;5<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">global;sdata_updates;;a;processed;24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">global;scratch_buffers_bytes;;a;queued;1536<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_boot;;a;processed;0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_kern;;a;processed;0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">source;s_sys;;a;processed;390<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">global;msg_clones;;a;processed;4662233<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination;d_file_no_location;;a;processed;37125<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">global;internal_queue_length;;a;processed;0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Here is the source configuration:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">source pan_splunk {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> network(port(514) transport("tcp") max-connections(100)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> flags(syslog-protocol) log-fetch-limit(100)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> log-iw-size(100000) log-msg-size(10485760)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> );<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> network(port(514) transport("udp") max-connections(100)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> flags(syslog-protocol) log-fetch-limit(100)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> log-iw-size(100000) log-msg-size(10485760)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> );<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> network(transport("tls") port(10514) max-connections(100)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> flags(syslog-protocol) log-iw-size(100000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> log-fetch-limit(100) log-msg-size(10485760)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> tls(peer-verify(optional-untrusted)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> key_file("/etc/syslog-ng/cert.d/serverkey.pem")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> cert_file("/etc/syslog-ng/cert.d/servercert.pem")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> ca_dir("/etc/syslog-ng/ca.d"))<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> );<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">};</span><span style="font-size:10.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Here is the destination configuration<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">destination d_file {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> owner("splunk") group("splunk") perm(0640)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> log-fifo-size(1000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> disk-buffer(<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> mem-buf-size(10000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> disk-buf-size(2000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> reliable(yes)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> dir("/var/log/remote/disk-buffer_dfile")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> flags("threaded", "no-multi-line")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""> );<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">};<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference. I would welcome any help you could provide.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">-Mark<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>