[syslog-ng] Some logs written, some are not
jbrown at boxconsulting.net
Tue Jun 16 12:21:19 UTC 2020
Just to update:
I’m getting the same exact behaviour using rsyslog as well as syslog-ng. So I’m going to attribute it to an issue with logback.
> On 12. Jun 2020, at 10:18, Jason Brown <jbrown at boxconsulting.net> wrote:
> Thanks. I’ll try disabling the internal() source.
> What’s happening is this:
> I have 3 kubernetes pods starting up (I don’t think the issue is related to kubernetes at all. I think we can treat these pods as 3 distinct servers and applications starting up).
> Each pod sends logs to the remote syslog-ng server. There are some logs sent via UDP, and others via TCP.
> The UDP logs always get created.
> The TCP logs only get created sometimes. Sometimes they’ll get created for all 3 pods. Sometimes for just 1. Rarely, none get created.
> Once a log does get created, the log continues to be written with no problems.
> Viewing a packet capture, I see TCP syslog traffic from the pods to the syslog server, even if no file is created. I’m not seeing any TCP issues overall.
>> On 12. Jun 2020, at 10:11, Fabien Wernli <wernli at in2p3.fr> wrote:
>> Hi Jason,
>> On Fri, Jun 12, 2020 at 09:58:40AM +0200, Jason Brown wrote:
>>> It looks like the cause of the “Error processing log message: <-1>” was indeed the logback configuration. Changing the priority to something valid cleans up that error.
>>> Unfortunately, it looks like that wasn’t the source of the problem. Digging in a little deeper, I’m seeing this message:
>>> Jun 12 07:37:03 s_local at syslog syslog-ng: Syslog connection accepted; fd='19', client='AF_INET(10.13.97.36:49346)', local='AF_INET(0.0.0.0:514)'
>>> Jun 12 07:37:03 s_local at syslog syslog-ng: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='Syslog connection accepted; fd=\'19\', client=\'AF_INET(10.13.97.36:49346)\', local=\'AF_INET(0.0.0.0:514)\'', first-suppressed-msg='>>>>>> filter rule evaluation begin; rule=\'f_auth\', location=\'/etc/syslog-ng/syslog-ng.conf:136:32\', msg=\'0x11c1bc0\’’
>>> So I’m assuming it’s likely a flow control issue.
>> I've already seen this message. It's probably unrelated to your issue which
>> I forgot what it was about. This message means that the internal() source,
>> which is referenced somewhere in your configuration (explicitly or
>> implicitly through an scl) which contains messages from syslog-ng itself
>> produces a feedback loop in a log path. This snowball effect is detected by
>> syslog-ng and action is taken to keep syslog-ng from exploding
>> catastrophically ;-)
>> You could disable the internal() source, but usually isolating it in its
>> separate log path is the way to go.
>> Now back to your initial issue, what's happening exactly ?
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
More information about the syslog-ng