<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Some more anecdotal details. I know
this is not really a systematic approach to tyrouble shooting
this.</div>
<div class="moz-cite-prefix">I guess I'll have to turn on the
verbose stats. We collect them regularly so that might get
overwhelmed.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The issue started again.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Number of allowed concurrent
connections reached, rejecting connection;
client='AF_INET(XXXX:50773)', local='AF_INET(YYYY:6514)',
group_name='client_network_tcp',
location='/etc/syslog-ng/syslog-ng.server.conf:61:9', max='15000'</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The established connection count was
2747</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">$ netstat -an | grep ESTABLISHED | grep
-c 6514<br>
2747<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The output destination stats were</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945402<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;memory_usage;148574728<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;written;1839783<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;truncated_count;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;truncated_bytes;0</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">repeating this statistics report a few
times with 2-3 seconds between then it shows</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945406<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945406<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945410<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
<br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945410<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
<br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1960812<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;121005<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">reload syslong-ng<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2084011<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;97926<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2084013<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;97938<br>
<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2103861<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;19842<br>
<br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2107477<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;23463<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2112825<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;28810<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">By the time I was this far
composingthis message, things seemed to have stalled again</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2909715<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;54738<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2909718<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;54738<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">eventually it started cycling through
the large queueing and flushing</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3177968<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;157790<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3187677<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;2150<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">with periods of getting stuck</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3418242<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;62294<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3418246<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;62294<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">and judging by the cpu usage, not all
of the log messages are being processed.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Verbose stats ... here I come.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2022-02-18 04:08, Laszlo Varady
(lvarady) wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SN4PR19MB5342A38C473047353C23CFB8F7379@SN4PR19MB5342.namprd19.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<br>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
A macroed file destination is unlikely to cause such an issue
if the location is writable.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
Just a tip that might help ruling out the case I mentioned:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
Setting the stats-level() to 4 results in an extremely verbose
stats output, where a counter called "free_window" can be
found for each network connection.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
This can be used to check whether a connection is suspended or
not. Note that this is a momentary value, which oscillates
between 0 and full_window, so a momentary 0 does not mean
anything bad,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
but we're looking for fixed 0 "free_window" values.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
--</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 11pt; color: rgb(0, 0, 0);">
László Várady<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
syslog-ng <a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng-bounces@lists.balabit.hu"><syslog-ng-bounces@lists.balabit.hu></a> on
behalf of Evan Rempel <a class="moz-txt-link-rfc2396E" href="mailto:erempel@uvic.ca"><erempel@uvic.ca></a><br>
<b>Sent:</b> Friday, February 18, 2022 3:22<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>
<a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu"><syslog-ng@lists.balabit.hu></a><br>
<b>Subject:</b> Re: [syslog-ng] allowed concurrent
connections - bug?</font>
<div> </div>
</div>
<div>
<div style="background-color:#FFEB9C; width:100%;
border-style:solid; border-color:#9C6500; border-width:1pt;
padding:2pt; font-size:10pt; line-height:12pt;
font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span>
This email originated from outside of the organization. Do
not follow guidance, click links, or open attachments unless
you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_moz-cite-prefix">There is only 1 destination,
although it is a file named with macros of date and hour.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">destination workstation.log {
file("/var/syslog/workstation.log.$R_YEAR$R_MONTH$R_DAY.${R_HOUR}0000"
); };<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">And that volume has never
become full.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Also, with a file based
destination I can't actually turn flow-control off since
files have soft flow-control.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Because I have 3500 real
connections that are all active (total of 10,000 messages
per second) syslog-ng cycles through reading 100 messages
from each source for 35000 messages and then writing those
to disk. it is very common for the queued messages to
fluctuate from a few hundred to 200,000 messages.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">I may have to add some new
metrics to our statistics gathering to understand more
about what is happening.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Evan Rempel.<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<br>
<div class="x_moz-cite-prefix">On 2022-02-17 13:13, Laszlo
Varady (lvarady) wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style><br>
<div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
Hi,</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
Do you have flags(flow-control) specified in your log
paths?</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
If so, a dead destination in such log paths might
cause the mentioned issue.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
When flow-control is activated, the corresponding
sources will be suspended. This suspended state does
not even allow syslog-ng to truly release connections
that have been closed by the clients.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
This is actually more of expected behavior as we don't
want to allow new connections in situations where logs
could not be delivered anyway.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
Please check the queued statistic counters of
"syslog-ng-ctl stats" to see whether this is the case.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<span class="x_VIiyi" lang="en"><span class="x_JLqJ4b
x_ChMk0b"><span>In case of anything else, I would
suspect a bug</span></span></span>.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
--</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
László Várady<br>
</div>
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> syslog-ng
<a class="x_moz-txt-link-rfc2396E"
href="mailto:syslog-ng-bounces@lists.balabit.hu"
moz-do-not-send="true">
<syslog-ng-bounces@lists.balabit.hu></a> on
behalf of Evan Rempel <a
class="x_moz-txt-link-rfc2396E"
href="mailto:erempel@uvic.ca"
moz-do-not-send="true">
<erempel@uvic.ca></a><br>
<b>Sent:</b> Thursday, February 17, 2022 19:01<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated
moz-txt-link-freetext"
href="mailto:syslog-ng@lists.balabit.hu"
moz-do-not-send="true">
syslog-ng@lists.balabit.hu</a> <a
class="x_moz-txt-link-rfc2396E"
href="mailto:syslog-ng@lists.balabit.hu"
moz-do-not-send="true">
<syslog-ng@lists.balabit.hu></a><br>
<b>Subject:</b> [syslog-ng] allowed concurrent
connections - bug?</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span
style="font-size:11pt"></span></font><br>
<font size="2"><span style="font-size:11pt">
<div class="x_PlainText">I am having an issue that
is a little difficult to reproduce so I wanted<br>
some input from others.<br>
<br>
I have a syslog-ng 3.35.1 that has a TLS source
defined with<br>
max-connections(10000)<br>
<br>
After some time the server starts logging a lot
of messages<br>
<br>
syslog-ng[12802]: Number of allowed concurrent
connections reached,<br>
rejecting connection;
client='AF_INET(XXXX:61062)',<br>
local='AF_INET(YYYY:6514)',
group_name='client_network_tcp',<br>
location='/etc/syslog-ng/syslog-ng.server.conf:61:9', max='10000'<br>
<br>
To the best of my ability I can only find about
2500 actual connections.<br>
<br>
Both lsof and netstat report around the 2500
connections.<br>
<br>
I had to restart syslog-ng to stop this
situation.<br>
<br>
Has anyone seen this behavior before?<br>
<br>
I get a lot of TLS connections without a
certificate.<br>
<br>
Error reading RFC6587 style framed data<br>
<br>
Pperhaps the counters are not decremented for
those timed out connections?<br>
<br>
--<br>
Evan Rempel</div>
</span></font></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
Evan Rempel 250.721.7691
Senior Systems Administrator <a class="moz-txt-link-abbreviated" href="mailto:erempel@uvic.ca">erempel@uvic.ca</a>
Data Centre Services, University Systems, University of Victoria
</pre>
</body>
</html>