[syslog-ng] Forwarding system startup messages

[Saurabh Shukla]

>
> Are you sure it's syslog-ng that writes /var/log/syslog ?
>
Yes. This is easy to verify. "file("/proc/kmsg" program_override("kernel"));"
directive adds a "kernel:" prefix to all messages from the kernel and when
I change this to something else, I see the change in /var/log/syslog.

I think the issue is that the output queue for network destinations is
created only if hostname resolution succeeds. During bootup, network
services are not up, so hostname resolution fails and no queue is created
for network destinations and hence syslog-ng fails to forward early startup
messages to network destinations.
This can be easily verified by bringing down the network, restarting
syslog-ng and then bringing up the network. Messages logged while the
network was down and after syslog-ng restart will not be forwarded to
network destinations.

Ideally, I would assume that syslog-ng should unconditionally create queues
as soon as it reads destinations from configuration files. So the questions
now are is the current behavior intentional and can it be fixed?

-- Saurabh


On Sun, Aug 23, 2015 at 11:11 PM, Scheidler, Balázs <
balazs.scheidler at balabit.com> wrote:

> Are you sure it's syslog-ng that writes /var/log/syslog ?
>
> Sometimes early startup is handled by a different logger.
>
> Try to disable syslog-ng from starting up, boot the system and start
> syslog-ng manually. The kernel messages should be sitting in the dmesg
> buffer and syslog-ng should process them as soon as it starts.
>
> If it shows the same symptoms try to look at syslog-ng stats counters.
> Well you can do those even without the reboot game.
>
> $ syslog-ng-ctl stats
>
> Try to look for the processed counter for /proc/kmsg
> On Aug 24, 2015 3:48 AM, "Saurabh Shukla" <saurabh at purestorage.com> wrote:
>
>> I don't think clearing kernel buffers is an issue here since syslog-ng is
>> seeing the kernel messages during system boot up and logging them to
>> /var/log/syslog. However, it fails to forward them to the remote server. So
>> there is some issue with buffering messages for the remote destination.
>>
>> -- Saurabh
>>
>> On Sat, Aug 22, 2015 at 12:00 PM, Scheidler, Balázs <
>> balazs.scheidler at balabit.com> wrote:
>>
>>> Hmm. You don't even use /dev/kmsg or system, so this setup should work.
>>> Don't you happen to run anything that could read /proc/kmsg or clear the
>>> kernel ringbuffer behind the backs of syslog-ng?
>>> On Aug 22, 2015 8:11 PM, "Saurabh Shukla" <saurabh at purestorage.com>
>>> wrote:
>>>
>>>> Can you show your source declaration?
>>>>
>>>> I am using the syslog-ng.conf from here -
>>>> https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf
>>>>
>>>> Do you use systemd journal?
>>>>
>>>> No.
>>>>
>>>>
>>>> On Sat, Aug 22, 2015 at 3:10 AM, Scheidler, Balázs <
>>>> balazs.scheidler at balabit.com> wrote:
>>>>
>>>>> Can you show your source declaration? Do you use systemd journal?
>>>>> On Aug 22, 2015 2:56 AM, "Saurabh Shukla" <saurabh at purestorage.com>
>>>>> wrote:
>>>>>
>>>>>> I am running syslog-ng 3.6.4 and I have the following destination and
>>>>>> log path configured that forwards all messages to the destination:
>>>>>>
>>>>>> destination remote {
>>>>>>     network("remote.example.com" port(514) transport(tcp)
>>>>>> log_fifo_size(2048));
>>>>>> };
>>>>>> log { source(s_all); destination(remote); flags(flow-control);};
>>>>>>
>>>>>> When the system reboots, I see that startup messages from the kernel
>>>>>> are logged into /var/log/syslog.
>>>>>> syslog-ng establishes a connection to the remote destination around
>>>>>> 10 sec after the first message was logged into /var/log/syslog. However, it
>>>>>> fails to forward any message that was logged into /var/log/syslog during
>>>>>> the first 10 seconds even though I have the output buffer and flow control
>>>>>> configured.
>>>>>>
>>>>>> Is this a bug in syslog-ng or am I missing some configuration steps?
>>>>>>
>>>>>> Thanks,
>>>>>> -- Saurabh
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150825/2df52ff0/attachment.htm