[syslog-ng] Some logs written, some are not

Jason Brown jbrown at boxconsulting.net
Tue Jun 16 12:21:19 UTC 2020

Just to update:

I’m getting the same exact behaviour using rsyslog as well as syslog-ng.  So I’m going to attribute it to an issue with logback.


> On 12. Jun 2020, at 10:18, Jason Brown <jbrown at boxconsulting.net> wrote:
> Thanks.  I’ll try disabling the internal() source.
> What’s happening is this:
> I have 3 kubernetes pods starting up (I don’t think the issue is related to kubernetes at all.  I think we can treat these pods as 3 distinct servers and applications starting up).
> Each pod sends logs to the remote syslog-ng server.  There are some logs sent via UDP, and others via TCP.
> The UDP logs always get created.
> The TCP logs only get created sometimes.  Sometimes they’ll get created for all 3 pods.  Sometimes for just 1.  Rarely, none get created.
> Once a log does get created, the log continues to be written with no problems.
> Viewing a packet capture, I see TCP syslog traffic from the pods to the syslog server, even if no file is created.  I’m not seeing any TCP issues overall.
> Thanks,
> Jason
>> On 12. Jun 2020, at 10:11, Fabien Wernli <wernli at in2p3.fr> wrote:
>> Hi Jason,
>> On Fri, Jun 12, 2020 at 09:58:40AM +0200, Jason Brown wrote:
>>> It looks like the cause of the “Error processing log message: <-1>” was indeed the logback configuration.  Changing the priority to something valid cleans up that error.  
>>> Unfortunately, it looks like that wasn’t the source of the problem.  Digging in a little deeper, I’m seeing this message:
>>> Jun 12 07:37:03 s_local at syslog syslog-ng[28873]: Syslog connection accepted; fd='19', client='AF_INET(', local='AF_INET('
>>> Jun 12 07:37:03 s_local at syslog syslog-ng[28873]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='Syslog connection accepted; fd=\'19\', client=\'AF_INET(\', local=\'AF_INET(\'', first-suppressed-msg='>>>>>> filter rule evaluation begin; rule=\'f_auth\', location=\'/etc/syslog-ng/syslog-ng.conf:136:32\', msg=\'0x11c1bc0\’’  
>>> So I’m assuming it’s likely a flow control issue.
>> I've already seen this message. It's probably unrelated to your issue which
>> I forgot what it was about. This message means that the internal() source,
>> which is referenced somewhere in your configuration (explicitly or
>> implicitly through an scl) which contains messages from syslog-ng itself
>> produces a feedback loop in a log path. This snowball effect is detected by
>> syslog-ng and action is taken to keep syslog-ng from exploding
>> catastrophically ;-)
>> You could disable the internal() source, but usually isolating it in its
>> separate log path is the way to go.
>> Now back to your initial issue, what's happening exactly ?
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq

More information about the syslog-ng mailing list