Are you sure it's syslog-ng that writes /var/log/syslog ?
Yes. This is easy to verify. "file("/proc/kmsg" program_override("kernel"));" directive adds a "kernel:" prefix to all messages from the kernel and when I change this to something else, I see the change in /var/log/syslog.

I think the issue is that the output queue for network destinations is created only if hostname resolution succeeds. During bootup, network services are not up, so hostname resolution fails and no queue is created for network destinations and hence syslog-ng fails to forward early startup messages to network destinations.
This can be easily verified by bringing down the network, restarting syslog-ng and then bringing up the network. Messages logged while the network was down and after syslog-ng restart will not be forwarded to network destinations.

Ideally, I would assume that syslog-ng should unconditionally create queues as soon as it reads destinations from configuration files. So the questions now are is the current behavior intentional and can it be fixed?

-- Saurabh


On Sun, Aug 23, 2015 at 11:11 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:

Are you sure it's syslog-ng that writes /var/log/syslog ?

Sometimes early startup is handled by a different logger.

Try to disable syslog-ng from starting up, boot the system and start syslog-ng manually. The kernel messages should be sitting in the dmesg buffer and syslog-ng should process them as soon as it starts.

If it shows the same symptoms try to look at syslog-ng stats counters. Well you can do those even without the reboot game.

$ syslog-ng-ctl stats

Try to look for the processed counter for /proc/kmsg

On Aug 24, 2015 3:48 AM, "Saurabh Shukla" <saurabh@purestorage.com> wrote:
I don't think clearing kernel buffers is an issue here since syslog-ng is seeing the kernel messages during system boot up and logging them to /var/log/syslog. However, it fails to forward them to the remote server. So there is some issue with buffering messages for the remote destination.

-- Saurabh

On Sat, Aug 22, 2015 at 12:00 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:

Hmm. You don't even use /dev/kmsg or system, so this setup should work. Don't you happen to run anything that could read /proc/kmsg or clear the kernel ringbuffer behind the backs of syslog-ng?

On Aug 22, 2015 8:11 PM, "Saurabh Shukla" <saurabh@purestorage.com> wrote:
Can you show your source declaration? 

Do you use systemd journal?
No. 


On Sat, Aug 22, 2015 at 3:10 AM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:

Can you show your source declaration? Do you use systemd journal?

On Aug 22, 2015 2:56 AM, "Saurabh Shukla" <saurabh@purestorage.com> wrote:
I am running syslog-ng 3.6.4 and I have the following destination and log path configured that forwards all messages to the destination:

destination remote {
    network("remote.example.com" port(514) transport(tcp) log_fifo_size(2048));
};
log { source(s_all); destination(remote); flags(flow-control);};

When the system reboots, I see that startup messages from the kernel are logged into /var/log/syslog.
syslog-ng establishes a connection to the remote destination around 10 sec after the first message was logged into /var/log/syslog. However, it fails to forward any message that was logged into /var/log/syslog during the first 10 seconds even though I have the output buffer and flow control configured.

Is this a bug in syslog-ng or am I missing some configuration steps?

Thanks,
-- Saurabh

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq