[syslog-ng] Forwarding system startup messages

Wed Aug 26 00:32:41 CEST 2015

Done - https://github.com/balabit/syslog-ng/issues/659

-- Saurabh

On Tue, Aug 25, 2015 at 2:09 PM, Scheidler, Balázs <
balazs.scheidler at balabit.com> wrote:

> that seems like a good diagnosis. the dns resolution problem handling is a
> pretty recent one, so this must have fallen through the cracks.
>
> can you pls file a github ticket with your findings?
>
> thanks
>
>
> --
> Bazsi
>
> On Tue, Aug 25, 2015 at 8:31 PM, Saurabh Shukla <saurabh at purestorage.com>
> wrote:
>
>> Are you sure it's syslog-ng that writes /var/log/syslog ?
>>>
>> Yes. This is easy to verify. "file("/proc/kmsg"
>> program_override("kernel"));" directive adds a "kernel:" prefix to all
>> messages from the kernel and when I change this to something else, I see
>> the change in /var/log/syslog.
>>
>> I think the issue is that the output queue for network destinations is
>> created only if hostname resolution succeeds. During bootup, network
>> services are not up, so hostname resolution fails and no queue is created
>> for network destinations and hence syslog-ng fails to forward early startup
>> messages to network destinations.
>> This can be easily verified by bringing down the network, restarting
>> syslog-ng and then bringing up the network. Messages logged while the
>> network was down and after syslog-ng restart will not be forwarded to
>> network destinations.
>>
>> Ideally, I would assume that syslog-ng should unconditionally create
>> queues as soon as it reads destinations from configuration files. So the
>> questions now are is the current behavior intentional and can it be fixed?
>>
>> -- Saurabh
>>
>>
>> On Sun, Aug 23, 2015 at 11:11 PM, Scheidler, Balázs <
>> balazs.scheidler at balabit.com> wrote:
>>
>>> Are you sure it's syslog-ng that writes /var/log/syslog ?
>>>
>>> Sometimes early startup is handled by a different logger.
>>>
>>> Try to disable syslog-ng from starting up, boot the system and start
>>> syslog-ng manually. The kernel messages should be sitting in the dmesg
>>> buffer and syslog-ng should process them as soon as it starts.
>>>
>>> If it shows the same symptoms try to look at syslog-ng stats counters.
>>> Well you can do those even without the reboot game.
>>>
>>> $ syslog-ng-ctl stats
>>>
>>> Try to look for the processed counter for /proc/kmsg
>>> On Aug 24, 2015 3:48 AM, "Saurabh Shukla" <saurabh at purestorage.com>
>>> wrote:
>>>
>>>> I don't think clearing kernel buffers is an issue here since syslog-ng
>>>> is seeing the kernel messages during system boot up and logging them to
>>>> /var/log/syslog. However, it fails to forward them to the remote server. So
>>>> there is some issue with buffering messages for the remote destination.
>>>>
>>>> -- Saurabh
>>>>
>>>> On Sat, Aug 22, 2015 at 12:00 PM, Scheidler, Balázs <
>>>> balazs.scheidler at balabit.com> wrote:
>>>>
>>>>> Hmm. You don't even use /dev/kmsg or system, so this setup should
>>>>> work. Don't you happen to run anything that could read /proc/kmsg or clear
>>>>> the kernel ringbuffer behind the backs of syslog-ng?
>>>>> On Aug 22, 2015 8:11 PM, "Saurabh Shukla" <saurabh at purestorage.com>
>>>>> wrote:
>>>>>
>>>>>> Can you show your source declaration?
>>>>>>
>>>>>> I am using the syslog-ng.conf from here -
>>>>>> https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf
>>>>>>
>>>>>> Do you use systemd journal?
>>>>>>
>>>>>> No.
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 22, 2015 at 3:10 AM, Scheidler, Balázs <
>>>>>> balazs.scheidler at balabit.com> wrote:
>>>>>>
>>>>>>> Can you show your source declaration? Do you use systemd journal?
>>>>>>> On Aug 22, 2015 2:56 AM, "Saurabh Shukla" <saurabh at purestorage.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am running syslog-ng 3.6.4 and I have the following destination
>>>>>>>> and log path configured that forwards all messages to the destination:
>>>>>>>>
>>>>>>>> destination remote {
>>>>>>>>     network("remote.example.com" port(514) transport(tcp)
>>>>>>>> log_fifo_size(2048));
>>>>>>>> };
>>>>>>>> log { source(s_all); destination(remote); flags(flow-control);};
>>>>>>>>
>>>>>>>> When the system reboots, I see that startup messages from the
>>>>>>>> kernel are logged into /var/log/syslog.
>>>>>>>> syslog-ng establishes a connection to the remote destination around
>>>>>>>> 10 sec after the first message was logged into /var/log/syslog. However, it
>>>>>>>> fails to forward any message that was logged into /var/log/syslog during
>>>>>>>> the first 10 seconds even though I have the output buffer and flow control
>>>>>>>> configured.
>>>>>>>>
>>>>>>>> Is this a bug in syslog-ng or am I missing some configuration steps?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -- Saurabh
>>>>>>>>
>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150825/17258b15/attachment.htm 