[syslog-ng] Regular occurrences of Error writing control channel; error='Broken pipe (32)'

Thu Apr 9 14:21:59 UTC 2020

  Hello,

Normally, there should be no restriction on the poll period, unless maybe if you are executing it in a busy loop.  4 times every minute should be fine.

Technically: every syslog-ng-ctl command runs in syslog-ng main loop. Therefore it is best to do jobs that do not do much thing to avoid holding up the main loop. None of the ctl commands take long in practice, but stats is one of those that has most things to do, collecting all statistics. If you are runnig with stats-level(3)`, then there might be a lot of counters. Still it should not cause such problem.
If you want to be more gentle with syslog-ng, and you are using recent enough version (it is available from 3.18), you can use syslog-ng-ctl show-license-info, which just returns a static text. It should be better option, if you just need a health check.

So what else could go wrong? I suspect something starves the mainloop.

Just randomly guessing, but if you happen to have an older system, then you might be using an older ivykis version too. Syslog-ng uses ivykis for event loop. There was an ivykis bug, where timers registered with zero or negative timeout starved the other tasks. I believe only ivykis below 0.39 was affected, but I can look this up properly if needed.

Other than that, there were deadlock bugs with patterndb, that was fixed recently. That or similar kind of things may mess things up.

Can you share more information of your system? Syslog-ng version, ivykis version, platform? Syslog-ng config? Any information about the network load? cpu consumption?

If you can create a minimal config example, that I could use to reproduce at my side, I can look further into this problem.

Br,
  Antal

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Ahmed Soyef <Soyef.Ahmed34 at homeoffice.gov.uk>
Sent: Tuesday, March 31, 2020 19:50
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Regular occurrences of Error writing control channel; error='Broken pipe (32)'

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Bazis,

Thanks for your response. There is a Prometheus exporter that is polling the socket and we have a health check bash script running that polls the socket approximately 4 times every minute. I am trying to figure out how often the Prometheus exporter polls the socket although is there a recommended number in terms of polling the socket?

Regards,

Soyef

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Balazs Scheidler
Sent: 31 March 2020 18:47
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Regular occurrences of Error writing control channel; error='Broken pipe (32)'

Hi,

this means that the client connecting to the control socket /var/run/syslog-ng/syslog-ng.ctl (or similar, depending on your platform), so this client disconnected while syslog-ng was still writing its result.

syslog-ng-ctl uses blocking I/O to query this socket, so it shouldn't do this (unless it is killed by something on your system). is there anything else that polls the control socket?

Bazis

On Tue, Mar 31, 2020 at 3:58 PM Ahmed Soyef <Soyef.Ahmed34 at homeoffice.gov.uk<mailto:Soyef.Ahmed34 at homeoffice.gov.uk>> wrote:

Hi All,

We’ve been running syslog-ng and have recently started to see the above message in our logs. Sometimes running syslog-ng-ctl stats takes very long and results in similar error messages. Does anyone know why this may be the case? Do these errors cause any issues with log flows through the syslog-ng service? Do they cause issues with recording of stats?

Kind regards,

Soyef

**********************************************************************
This email and any files transmitted with it are private and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please return it to the address
it came from telling them it is not for you and then delete it from your system.
This email message has been swept for computer viruses.
**********************************************************************

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C4c44ed1fe757486fce6208d7d59bf609%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637212738152636925&sdata=qMFe2sP7qDyYSjwJVes3z38mVpN9PZDz7Gfp%2FC2SCeA%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C4c44ed1fe757486fce6208d7d59bf609%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637212738152646880&sdata=wjM4bWiorMsSD%2BtU877LhNS8LJ51QJl6WPjRZ4lykMA%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C4c44ed1fe757486fce6208d7d59bf609%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637212738152646880&sdata=%2BjeHBYvJyqtYICVZXXodHTunUVPTeWc1RP45TtCvYD4%3D&reserved=0>

--

Bazsi

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

**********************************************************************
This email and any files transmitted with it are private and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please return it to the address
it came from telling them it is not for you and then delete it from your system.
This email message has been swept for computer viruses.
**********************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200409/af8f5ef8/attachment.html>