[syslog-ng] allowed concurrent connections - bug?

[Evan Rempel]

There is only 1 destination, although it is a file named with macros of 
date and hour.

destination workstation.log { 
file("/var/syslog/workstation.log.$R_YEAR$R_MONTH$R_DAY.${R_HOUR}0000" ); };

And that volume has never become full.

Also, with a file based destination I can't actually turn flow-control 
off since files have soft flow-control.

Because I have 3500 real connections that are all active (total of 
10,000 messages per second) syslog-ng cycles through reading 100 
messages from each source for 35000 messages and then writing those to 
disk. it is very common for the queued messages to fluctuate from a few 
hundred to 200,000 messages.

I may have to add some new metrics to our statistics gathering to 
understand more about what is happening.

Evan Rempel.


On 2022-02-17 13:13, Laszlo Varady (lvarady) wrote:
>
> Hi,
>
> Do you have flags(flow-control) specified in your log paths?
> If so, a dead destination in such log paths might cause the mentioned 
> issue.
>
> When flow-control is activated, the corresponding sources will be 
> suspended. This suspended state does not even allow syslog-ng to truly 
> release connections that have been closed by the clients.
> This is actually more of expected behavior as we don't want to allow 
> new connections in situations where logs could not be delivered anyway.
>
> Please check the queued statistic counters of "syslog-ng-ctl stats" to 
> see whether this is the case.
>
> In case of anything else, I would suspect a bug.
>
> --
> László Várady
> ------------------------------------------------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of 
> Evan Rempel <erempel at uvic.ca>
> *Sent:* Thursday, February 17, 2022 19:01
> *To:* syslog-ng at lists.balabit.hu <syslog-ng at lists.balabit.hu>
> *Subject:* [syslog-ng] allowed concurrent connections - bug?
>
> I am having an issue that is a little difficult to reproduce so I wanted
> some input from others.
>
> I have a syslog-ng 3.35.1 that has a TLS source defined with
> max-connections(10000)
>
> After some time the server starts logging a lot of messages
>
> syslog-ng[12802]: Number of allowed concurrent connections reached,
> rejecting connection; client='AF_INET(XXXX:61062)',
> local='AF_INET(YYYY:6514)', group_name='client_network_tcp',
> location='/etc/syslog-ng/syslog-ng.server.conf:61:9', max='10000'
>
> To the best of my ability I can only find about 2500 actual connections.
>
> Both lsof and netstat report around the 2500 connections.
>
> I had to restart syslog-ng to stop this situation.
>
> Has anyone seen this behavior before?
>
> I get a lot of TLS connections without a certificate.
>
> Error reading RFC6587 style framed data
>
> Pperhaps the counters are not decremented for those timed out connections?
>
> --
> Evan Rempel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20220217/095315ae/attachment.htm>