[syslog-ng] Forwarding system startup messages

Tue Aug 25 23:09:30 CEST 2015

that seems like a good diagnosis. the dns resolution problem handling is a
pretty recent one, so this must have fallen through the cracks.

can you pls file a github ticket with your findings?

thanks

-- 
Bazsi

On Tue, Aug 25, 2015 at 8:31 PM, Saurabh Shukla <saurabh at purestorage.com>
wrote:

> Are you sure it's syslog-ng that writes /var/log/syslog ?
>>
> Yes. This is easy to verify. "file("/proc/kmsg"
> program_override("kernel"));" directive adds a "kernel:" prefix to all
> messages from the kernel and when I change this to something else, I see
> the change in /var/log/syslog.
>
> I think the issue is that the output queue for network destinations is
> created only if hostname resolution succeeds. During bootup, network
> services are not up, so hostname resolution fails and no queue is created
> for network destinations and hence syslog-ng fails to forward early startup
> messages to network destinations.
> This can be easily verified by bringing down the network, restarting
> syslog-ng and then bringing up the network. Messages logged while the
> network was down and after syslog-ng restart will not be forwarded to
> network destinations.
>
> Ideally, I would assume that syslog-ng should unconditionally create
> queues as soon as it reads destinations from configuration files. So the
> questions now are is the current behavior intentional and can it be fixed?
>
> -- Saurabh
>
>
> On Sun, Aug 23, 2015 at 11:11 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> Are you sure it's syslog-ng that writes /var/log/syslog ?
>>
>> Sometimes early startup is handled by a different logger.
>>
>> Try to disable syslog-ng from starting up, boot the system and start
>> syslog-ng manually. The kernel messages should be sitting in the dmesg
>> buffer and syslog-ng should process them as soon as it starts.
>>
>> If it shows the same symptoms try to look at syslog-ng stats counters.
>> Well you can do those even without the reboot game.
>>
>> $ syslog-ng-ctl stats
>>
>> Try to look for the processed counter for /proc/kmsg
>> On Aug 24, 2015 3:48 AM, "Saurabh Shukla" <saurabh at purestorage.com>
>> wrote:
>>
>>> I don't think clearing kernel buffers is an issue here since syslog-ng
>>> is seeing the kernel messages during system boot up and logging them to
>>> /var/log/syslog. However, it fails to forward them to the remote server. So
>>> there is some issue with buffering messages for the remote destination.
>>>
>>> -- Saurabh
>>>
>>> On Sat, Aug 22, 2015 at 12:00 PM, Scheidler, Balázs <
>>> balazs.scheidler at balabit.com> wrote:
>>>
>>>> Hmm. You don't even use /dev/kmsg or system, so this setup should work.
>>>> Don't you happen to run anything that could read /proc/kmsg or clear the
>>>> kernel ringbuffer behind the backs of syslog-ng?
>>>> On Aug 22, 2015 8:11 PM, "Saurabh Shukla" <saurabh at purestorage.com>
>>>> wrote:
>>>>
>>>>> Can you show your source declaration?
>>>>>
>>>>> I am using the syslog-ng.conf from here -
>>>>> https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf
>>>>>
>>>>> Do you use systemd journal?
>>>>>
>>>>> No.
>>>>>
>>>>>
>>>>> On Sat, Aug 22, 2015 at 3:10 AM, Scheidler, Balázs <
>>>>> balazs.scheidler at balabit.com> wrote:
>>>>>
>>>>>> Can you show your source declaration? Do you use systemd journal?
>>>>>> On Aug 22, 2015 2:56 AM, "Saurabh Shukla" <saurabh at purestorage.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am running syslog-ng 3.6.4 and I have the following destination
>>>>>>> and log path configured that forwards all messages to the destination:
>>>>>>>
>>>>>>> destination remote {
>>>>>>>     network("remote.example.com" port(514) transport(tcp)
>>>>>>> log_fifo_size(2048));
>>>>>>> };
>>>>>>> log { source(s_all); destination(remote); flags(flow-control);};
>>>>>>>
>>>>>>> When the system reboots, I see that startup messages from the kernel
>>>>>>> are logged into /var/log/syslog.
>>>>>>> syslog-ng establishes a connection to the remote destination around
>>>>>>> 10 sec after the first message was logged into /var/log/syslog. However, it
>>>>>>> fails to forward any message that was logged into /var/log/syslog during
>>>>>>> the first 10 seconds even though I have the output buffer and flow control
>>>>>>> configured.
>>>>>>>
>>>>>>> Is this a bug in syslog-ng or am I missing some configuration steps?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -- Saurabh
>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150825/810d8204/attachment-0001.htm 