<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">This was related to dns resolution.
This particular logging service has clients that are not on our
own network and as such the DNS resolution is "far away" as far as
networks are concerned.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">By turning off dns resolution in
syslog-ng the queuing became nearly 0. No longer was the queue
contents cycling from 0 through to 200,000 messages and then
flushing.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Then I added nscd using hostname
caching only, and enable the syslog-ng dns resolution again. The
great performance remained with a near zero queue size. <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Now I have the same functionality with
dns names in my log files and I do not have any slow downs for
name resolution inside of syslog-ng.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Only time will tell if this is related
to the allowed connections issue but they seem to be strongly
linked at this time.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Evan.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2022-02-18 09:45, Evan Rempel wrote:<br>
</div>
<blockquote type="cite"
cite="mid:fdae31c2-2584-f0ce-fd46-170782d4dc5d@uvic.ca">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>
<div>
<div class="moz-cite-prefix">Some more anecdotal details. I
know this is not really a systematic approach to tyrouble
shooting this.</div>
<div class="moz-cite-prefix">I guess I'll have to turn on the
verbose stats. We collect them regularly so that might get
overwhelmed.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The issue started again.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Number of allowed concurrent
connections reached, rejecting connection;
client='AF_INET(XXXX:50773)', local='AF_INET(YYYY:6514)',
group_name='client_network_tcp',
location='/etc/syslog-ng/syslog-ng.server.conf:61:9',
max='15000'</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The established connection count
was 2747</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">$ netstat -an | grep ESTABLISHED
| grep -c 6514<br>
2747<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">The output destination stats were</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945402<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;memory_usage;148574728<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;written;1839783<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;truncated_count;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;truncated_bytes;0</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">repeating this statistics report
a few times with 2-3 seconds between then it shows</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945406<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945406<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945410<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
<br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1945410<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;105619<br>
<br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;1960812<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;121005<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">reload syslong-ng<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2084011<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;97926<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2084013<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;97938<br>
<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2103861<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;19842<br>
<br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2107477<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;23463<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2112825<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;28810<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">By the time I was this far
composingthis message, things seemed to have stalled again</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2909715<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;54738<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;2909718<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;54738<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">eventually it started cycling
through the large queueing and flushing</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3177968<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;157790<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3187677<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;2150<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">with periods of getting stuck</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3418242<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;62294<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;dropped;0<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;processed;3418246<br>
dst.file;workstation.log#0;/var/syslog/workstation.log.20220218.090000;a;queued;62294<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">and judging by the cpu usage, not
all of the log messages are being processed.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Verbose stats ... here I come.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2022-02-18 04:08, Laszlo
Varady (lvarady) wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SN4PR19MB5342A38C473047353C23CFB8F7379@SN4PR19MB5342.namprd19.prod.outlook.com">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style><br>
<div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
A macroed file destination is unlikely to cause such an
issue if the location is writable.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Just a tip that might help ruling out the case I
mentioned:</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Setting the stats-level() to 4 results in an extremely
verbose stats output, where a counter called
"free_window" can be found for each network connection.</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This can be used to check whether a connection is
suspended or not. Note that this is a momentary value,
which oscillates between 0 and full_window, so a
momentary 0 does not mean anything bad,</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
but we're looking for fixed 0 "free_window" values.</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
--</div>
<div style="font-family: Calibri, Arial, Helvetica,
sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
László Várady<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> syslog-ng
<a class="moz-txt-link-rfc2396E"
href="mailto:syslog-ng-bounces@lists.balabit.hu"
moz-do-not-send="true">
<syslog-ng-bounces@lists.balabit.hu></a> on
behalf of Evan Rempel <a
class="moz-txt-link-rfc2396E"
href="mailto:erempel@uvic.ca" moz-do-not-send="true">
<erempel@uvic.ca></a><br>
<b>Sent:</b> Friday, February 18, 2022 3:22<br>
<b>To:</b> <a class="moz-txt-link-abbreviated
moz-txt-link-freetext"
href="mailto:syslog-ng@lists.balabit.hu"
moz-do-not-send="true">
syslog-ng@lists.balabit.hu</a> <a
class="moz-txt-link-rfc2396E"
href="mailto:syslog-ng@lists.balabit.hu"
moz-do-not-send="true">
<syslog-ng@lists.balabit.hu></a><br>
<b>Subject:</b> Re: [syslog-ng] allowed concurrent
connections - bug?</font>
<div> </div>
</div>
<div>
<div style="background-color:#FFEB9C; width:100%;
border-style:solid; border-color:#9C6500;
border-width:1pt; padding:2pt; font-size:10pt;
line-height:12pt; font-family:'Calibri'; color:Black;
text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span>
This email originated from outside of the
organization. Do not follow guidance, click links, or
open attachments unless you recognize the sender and
know the content is safe.</div>
<br>
<div>
<div class="x_moz-cite-prefix">There is only 1
destination, although it is a file named with macros
of date and hour.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">destination
workstation.log {
file("/var/syslog/workstation.log.$R_YEAR$R_MONTH$R_DAY.${R_HOUR}0000"
); };<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">And that volume has
never become full.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Also, with a file based
destination I can't actually turn flow-control off
since files have soft flow-control.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Because I have 3500
real connections that are all active (total of
10,000 messages per second) syslog-ng cycles through
reading 100 messages from each source for 35000
messages and then writing those to disk. it is very
common for the queued messages to fluctuate from a
few hundred to 200,000 messages.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">I may have to add some
new metrics to our statistics gathering to
understand more about what is happening.</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Evan Rempel.<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<br>
<div class="x_moz-cite-prefix">On 2022-02-17 13:13,
Laszlo Varady (lvarady) wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style><br>
<div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
Hi,</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
Do you have flags(flow-control) specified in
your log paths?</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
If so, a dead destination in such log paths
might cause the mentioned issue.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
When flow-control is activated, the
corresponding sources will be suspended. This
suspended state does not even allow syslog-ng to
truly release connections that have been closed
by the clients.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
This is actually more of expected behavior as we
don't want to allow new connections in
situations where logs could not be delivered
anyway.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
Please check the queued statistic counters of
"syslog-ng-ctl stats" to see whether this is the
case.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<span class="x_VIiyi" lang="en"><span
class="x_JLqJ4b x_ChMk0b"><span>In case of
anything else, I would suspect a bug</span></span></span>.</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
--</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:11pt; color:rgb(0,0,0)">
László Várady<br>
</div>
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri,
sans-serif" color="#000000"><b>From:</b>
syslog-ng
<a class="x_moz-txt-link-rfc2396E"
href="mailto:syslog-ng-bounces@lists.balabit.hu"
moz-do-not-send="true">
<syslog-ng-bounces@lists.balabit.hu></a>
on behalf of Evan Rempel <a
class="x_moz-txt-link-rfc2396E"
href="mailto:erempel@uvic.ca"
moz-do-not-send="true">
<erempel@uvic.ca></a><br>
<b>Sent:</b> Thursday, February 17, 2022 19:01<br>
<b>To:</b> <a
class="x_moz-txt-link-abbreviated
moz-txt-link-freetext"
href="mailto:syslog-ng@lists.balabit.hu"
moz-do-not-send="true">
syslog-ng@lists.balabit.hu</a> <a
class="x_moz-txt-link-rfc2396E"
href="mailto:syslog-ng@lists.balabit.hu"
moz-do-not-send="true">
<syslog-ng@lists.balabit.hu></a><br>
<b>Subject:</b> [syslog-ng] allowed concurrent
connections - bug?</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span
style="font-size:11pt"></span></font><br>
<font size="2"><span style="font-size:11pt">
<div class="x_PlainText">I am having an
issue that is a little difficult to
reproduce so I wanted<br>
some input from others.<br>
<br>
I have a syslog-ng 3.35.1 that has a TLS
source defined with<br>
max-connections(10000)<br>
<br>
After some time the server starts logging
a lot of messages<br>
<br>
syslog-ng[12802]: Number of allowed
concurrent connections reached,<br>
rejecting connection;
client='AF_INET(XXXX:61062)',<br>
local='AF_INET(YYYY:6514)',
group_name='client_network_tcp',<br>
location='/etc/syslog-ng/syslog-ng.server.conf:61:9', max='10000'<br>
<br>
To the best of my ability I can only find
about 2500 actual connections.<br>
<br>
Both lsof and netstat report around the
2500 connections.<br>
<br>
I had to restart syslog-ng to stop this
situation.<br>
<br>
Has anyone seen this behavior before?<br>
<br>
I get a lot of TLS connections without a
certificate.<br>
<br>
Error reading RFC6587 style framed data<br>
<br>
Pperhaps the counters are not decremented
for those timed out connections?<br>
<br>
--<br>
Evan Rempel</div>
</span></font></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</body>
</html>