Regular occurrences of Error writing control channel; error='Broken pipe (32)'
Hi All, We've been running syslog-ng and have recently started to see the above message in our logs. Sometimes running syslog-ng-ctl stats takes very long and results in similar error messages. Does anyone know why this may be the case? Do these errors cause any issues with log flows through the syslog-ng service? Do they cause issues with recording of stats? Kind regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************
Hi, this means that the client connecting to the control socket /var/run/syslog-ng/syslog-ng.ctl (or similar, depending on your platform), so this client disconnected while syslog-ng was still writing its result. syslog-ng-ctl uses blocking I/O to query this socket, so it shouldn't do this (unless it is killed by something on your system). is there anything else that polls the control socket? Bazis On Tue, Mar 31, 2020 at 3:58 PM Ahmed Soyef <Soyef.Ahmed34@homeoffice.gov.uk> wrote:
Hi All,
We’ve been running syslog-ng and have recently started to see the above message in our logs. Sometimes running syslog-ng-ctl stats takes very long and results in similar error messages. Does anyone know why this may be the case? Do these errors cause any issues with log flows through the syslog-ng service? Do they cause issues with recording of stats?
Kind regards,
Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Hi Bazis, Thanks for your response. There is a Prometheus exporter that is polling the socket and we have a health check bash script running that polls the socket approximately 4 times every minute. I am trying to figure out how often the Prometheus exporter polls the socket although is there a recommended number in terms of polling the socket? Regards, Soyef From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Balazs Scheidler Sent: 31 March 2020 18:47 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Regular occurrences of Error writing control channel; error='Broken pipe (32)' Hi, this means that the client connecting to the control socket /var/run/syslog-ng/syslog-ng.ctl (or similar, depending on your platform), so this client disconnected while syslog-ng was still writing its result. syslog-ng-ctl uses blocking I/O to query this socket, so it shouldn't do this (unless it is killed by something on your system). is there anything else that polls the control socket? Bazis On Tue, Mar 31, 2020 at 3:58 PM Ahmed Soyef <Soyef.Ahmed34@homeoffice.gov.uk<mailto:Soyef.Ahmed34@homeoffice.gov.uk>> wrote: Hi All, We’ve been running syslog-ng and have recently started to see the above message in our logs. Sometimes running syslog-ng-ctl stats takes very long and results in similar error messages. Does anyone know why this may be the case? Do these errors cause any issues with log flows through the syslog-ng service? Do they cause issues with recording of stats? Kind regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. ********************************************************************** ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -- Bazsi ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************
Hello, Normally, there should be no restriction on the poll period, unless maybe if you are executing it in a busy loop. 4 times every minute should be fine. Technically: every syslog-ng-ctl command runs in syslog-ng main loop. Therefore it is best to do jobs that do not do much thing to avoid holding up the main loop. None of the ctl commands take long in practice, but stats is one of those that has most things to do, collecting all statistics. If you are runnig with stats-level(3)`, then there might be a lot of counters. Still it should not cause such problem. If you want to be more gentle with syslog-ng, and you are using recent enough version (it is available from 3.18), you can use syslog-ng-ctl show-license-info, which just returns a static text. It should be better option, if you just need a health check. So what else could go wrong? I suspect something starves the mainloop. Just randomly guessing, but if you happen to have an older system, then you might be using an older ivykis version too. Syslog-ng uses ivykis for event loop. There was an ivykis bug, where timers registered with zero or negative timeout starved the other tasks. I believe only ivykis below 0.39 was affected, but I can look this up properly if needed. Other than that, there were deadlock bugs with patterndb, that was fixed recently. That or similar kind of things may mess things up. Can you share more information of your system? Syslog-ng version, ivykis version, platform? Syslog-ng config? Any information about the network load? cpu consumption? If you can create a minimal config example, that I could use to reproduce at my side, I can look further into this problem. Br, Antal ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Ahmed Soyef <Soyef.Ahmed34@homeoffice.gov.uk> Sent: Tuesday, March 31, 2020 19:50 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Regular occurrences of Error writing control channel; error='Broken pipe (32)' CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi Bazis, Thanks for your response. There is a Prometheus exporter that is polling the socket and we have a health check bash script running that polls the socket approximately 4 times every minute. I am trying to figure out how often the Prometheus exporter polls the socket although is there a recommended number in terms of polling the socket? Regards, Soyef From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Balazs Scheidler Sent: 31 March 2020 18:47 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Regular occurrences of Error writing control channel; error='Broken pipe (32)' Hi, this means that the client connecting to the control socket /var/run/syslog-ng/syslog-ng.ctl (or similar, depending on your platform), so this client disconnected while syslog-ng was still writing its result. syslog-ng-ctl uses blocking I/O to query this socket, so it shouldn't do this (unless it is killed by something on your system). is there anything else that polls the control socket? Bazis On Tue, Mar 31, 2020 at 3:58 PM Ahmed Soyef <Soyef.Ahmed34@homeoffice.gov.uk<mailto:Soyef.Ahmed34@homeoffice.gov.uk>> wrote: Hi All, We’ve been running syslog-ng and have recently started to see the above message in our logs. Sometimes running syslog-ng-ctl stats takes very long and results in similar error messages. Does anyone know why this may be the case? Do these errors cause any issues with log flows through the syslog-ng service? Do they cause issues with recording of stats? Kind regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. ********************************************************************** ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C4c44ed1fe757486fce6208d7d59bf609%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637212738152636925&sdata=qMFe2sP7qDyYSjwJVes3z38mVpN9PZDz7Gfp%2FC2SCeA%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C4c44ed1fe757486fce6208d7d59bf609%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637212738152646880&sdata=wjM4bWiorMsSD%2BtU877LhNS8LJ51QJl6WPjRZ4lykMA%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C4c44ed1fe757486fce6208d7d59bf609%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637212738152646880&sdata=%2BjeHBYvJyqtYICVZXXodHTunUVPTeWc1RP45TtCvYD4%3D&reserved=0> -- Bazsi ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************
participants (3)
-
Ahmed Soyef
-
Antal Nemes (anemes)
-
Balazs Scheidler