<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
A macroed file destination is unlikely to cause such an issue if the location is writable.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Just a tip that might help ruling out the case I mentioned:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Setting the stats-level() to 4 results in an extremely verbose stats output, where a counter called "free_window" can be found for each network connection.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This can be used to check whether a connection is suspended or not. Note that this is a momentary value, which oscillates between 0 and full_window, so a momentary 0 does not mean anything bad,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
but we're looking for fixed 0 "free_window" values.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
--</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
László Várady<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Evan Rempel <erempel@uvic.ca><br>
<b>Sent:</b> Friday, February 18, 2022 3:22<br>
<b>To:</b> syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] allowed concurrent connections - bug?</font>
<div> </div>
</div>
<div>
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_moz-cite-prefix">There is only 1 destination, although it is a file named with macros of date and hour.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">destination workstation.log { file("/var/syslog/workstation.log.$R_YEAR$R_MONTH$R_DAY.${R_HOUR}0000" ); };<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">And that volume has never become full.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Also, with a file based destination I can't actually turn flow-control off since files have soft flow-control.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Because I have 3500 real connections that are all active (total of 10,000 messages per second) syslog-ng cycles through reading 100 messages from each source for 35000 messages and then writing those to disk. it is very common
for the queued messages to fluctuate from a few hundred to 200,000 messages.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">I may have to add some new metrics to our statistics gathering to understand more about what is happening.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Evan Rempel.<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<br>
<div class="x_moz-cite-prefix">On 2022-02-17 13:13, Laszlo Varady (lvarady) wrote:<br>
</div>
<blockquote type="cite"><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style><br>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
Do you have flags(flow-control) specified in your log paths?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
If so, a dead destination in such log paths might cause the mentioned issue.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
When flow-control is activated, the corresponding sources will be suspended. This suspended state does not even allow syslog-ng to truly release connections that have been closed by the clients.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
This is actually more of expected behavior as we don't want to allow new connections in situations where logs could not be delivered anyway.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
Please check the queued statistic counters of "syslog-ng-ctl stats" to see whether this is the case.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<span class="x_VIiyi" lang="en"><span class="x_JLqJ4b x_ChMk0b"><span>In case of anything else, I would suspect a bug</span></span></span>.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
--</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
László Várady<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> syslog-ng
<a class="x_moz-txt-link-rfc2396E" href="mailto:syslog-ng-bounces@lists.balabit.hu">
<syslog-ng-bounces@lists.balabit.hu></a> on behalf of Evan Rempel <a class="x_moz-txt-link-rfc2396E" href="mailto:erempel@uvic.ca">
<erempel@uvic.ca></a><br>
<b>Sent:</b> Thursday, February 17, 2022 19:01<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated" href="mailto:syslog-ng@lists.balabit.hu">
syslog-ng@lists.balabit.hu</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu">
<syslog-ng@lists.balabit.hu></a><br>
<b>Subject:</b> [syslog-ng] allowed concurrent connections - bug?</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span style="font-size:11pt"></span></font><br>
<font size="2"><span style="font-size:11pt">
<div class="x_PlainText">I am having an issue that is a little difficult to reproduce so I wanted<br>
some input from others.<br>
<br>
I have a syslog-ng 3.35.1 that has a TLS source defined with<br>
max-connections(10000)<br>
<br>
After some time the server starts logging a lot of messages<br>
<br>
syslog-ng[12802]: Number of allowed concurrent connections reached,<br>
rejecting connection; client='AF_INET(XXXX:61062)',<br>
local='AF_INET(YYYY:6514)', group_name='client_network_tcp',<br>
location='/etc/syslog-ng/syslog-ng.server.conf:61:9', max='10000'<br>
<br>
To the best of my ability I can only find about 2500 actual connections.<br>
<br>
Both lsof and netstat report around the 2500 connections.<br>
<br>
I had to restart syslog-ng to stop this situation.<br>
<br>
Has anyone seen this behavior before?<br>
<br>
I get a lot of TLS connections without a certificate.<br>
<br>
Error reading RFC6587 style framed data<br>
<br>
Pperhaps the counters are not decremented for those timed out connections?<br>
<br>
--<br>
Evan Rempel</div>
</span></font></div>
</div>
</blockquote>
</div>
</div>
</body>
</html>