[syslog-ng] Some logs written, some are not

[Jason Brown]

Just to update:

I’m getting the same exact behaviour using rsyslog as well as syslog-ng.  So I’m going to attribute it to an issue with logback.

Thanks,
Jason

> On 12. Jun 2020, at 10:18, Jason Brown <jbrown at boxconsulting.net> wrote:
> 
> Thanks.  I’ll try disabling the internal() source.
> 
> What’s happening is this:
> I have 3 kubernetes pods starting up (I don’t think the issue is related to kubernetes at all.  I think we can treat these pods as 3 distinct servers and applications starting up).
> Each pod sends logs to the remote syslog-ng server.  There are some logs sent via UDP, and others via TCP.
> The UDP logs always get created.
> The TCP logs only get created sometimes.  Sometimes they’ll get created for all 3 pods.  Sometimes for just 1.  Rarely, none get created.
> Once a log does get created, the log continues to be written with no problems.
> 
> Viewing a packet capture, I see TCP syslog traffic from the pods to the syslog server, even if no file is created.  I’m not seeing any TCP issues overall.
> 
> Thanks,
> Jason
> 
>> On 12. Jun 2020, at 10:11, Fabien Wernli <wernli at in2p3.fr> wrote:
>> 
>> Hi Jason,
>> 
>> On Fri, Jun 12, 2020 at 09:58:40AM +0200, Jason Brown wrote:
>>> It looks like the cause of the “Error processing log message: <-1>” was indeed the logback configuration.  Changing the priority to something valid cleans up that error.  
>>> Unfortunately, it looks like that wasn’t the source of the problem.  Digging in a little deeper, I’m seeing this message:
>>> 
>>> Jun 12 07:37:03 s_local at syslog syslog-ng[28873]: Syslog connection accepted; fd='19', client='AF_INET(10.13.97.36:49346)', local='AF_INET(0.0.0.0:514)'
>>> Jun 12 07:37:03 s_local at syslog syslog-ng[28873]: internal() messages are looping back, preventing loop by suppressing all internal messages until the current message is processed; trigger-msg='Syslog connection accepted; fd=\'19\', client=\'AF_INET(10.13.97.36:49346)\', local=\'AF_INET(0.0.0.0:514)\'', first-suppressed-msg='>>>>>> filter rule evaluation begin; rule=\'f_auth\', location=\'/etc/syslog-ng/syslog-ng.conf:136:32\', msg=\'0x11c1bc0\’’  
>>> 
>>> So I’m assuming it’s likely a flow control issue.
>> 
>> I've already seen this message. It's probably unrelated to your issue which
>> I forgot what it was about. This message means that the internal() source,
>> which is referenced somewhere in your configuration (explicitly or
>> implicitly through an scl) which contains messages from syslog-ng itself
>> produces a feedback loop in a log path. This snowball effect is detected by
>> syslog-ng and action is taken to keep syslog-ng from exploding
>> catastrophically ;-)
>> 
>> You could disable the internal() source, but usually isolating it in its
>> separate log path is the way to go.
>> 
>> Now back to your initial issue, what's happening exactly ?
>> 
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> 
>