<div dir="ltr">Done - <a href="https://github.com/balabit/syslog-ng/issues/659">https://github.com/balabit/syslog-ng/issues/659</a><div><br></div><div>-- Saurabh</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 25, 2015 at 2:09 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>that seems like a good diagnosis. the dns resolution problem handling is a pretty recent one, so this must have fallen through the cracks. <br><br></div>can you pls file a github ticket with your findings?<br><br></div>thanks<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all"><div><div><div dir="ltr">-- <br>Bazsi<br></div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">On Tue, Aug 25, 2015 at 8:31 PM, Saurabh Shukla <span dir="ltr"><<a href="mailto:saurabh@purestorage.com" target="_blank">saurabh@purestorage.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Are you sure it's syslog-ng that writes /var/log/syslog ?<br></blockquote></span><div>Yes. This is easy to verify. "<span style="color:rgb(51,51,51);font-size:12px;line-height:16.7999992370605px;white-space:pre-wrap"><font face="monospace, monospace">file("/proc/kmsg" program_override("kernel"));</font></span>" directive adds a "<font face="monospace, monospace">kernel:</font>" prefix to all messages from the kernel and when I change this to something else, I see the change in /var/log/syslog.</div><div><br></div><div>I think the issue is that the output queue for network destinations is created only if hostname resolution succeeds. During bootup, network services are not up, so hostname resolution fails and no queue is created for network destinations and hence syslog-ng fails to forward early startup messages to network destinations.</div><div>This can be easily verified by bringing down the network, restarting syslog-ng and then bringing up the network. Messages logged while the network was down and after syslog-ng restart will not be forwarded to network destinations.</div><div><br></div><div>Ideally, I would assume that syslog-ng should unconditionally create queues as soon as it reads destinations from configuration files. So the questions now are is the current behavior intentional and can it be fixed?</div><span><font color="#888888"><div><br></div><div>-- Saurabh<br></div></font></span><div><div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Aug 23, 2015 at 11:11 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Are you sure it's syslog-ng that writes /var/log/syslog ?</p>
<p dir="ltr">Sometimes early startup is handled by a different logger.</p>
<p dir="ltr">Try to disable syslog-ng from starting up, boot the system and start syslog-ng manually. The kernel messages should be sitting in the dmesg buffer and syslog-ng should process them as soon as it starts.</p>
<p dir="ltr">If it shows the same symptoms try to look at syslog-ng stats counters. Well you can do those even without the reboot game.</p>
<p dir="ltr">$ syslog-ng-ctl stats</p>
<p dir="ltr">Try to look for the processed counter for /proc/kmsg</p><div><div>
<div class="gmail_quote">On Aug 24, 2015 3:48 AM, "Saurabh Shukla" <<a href="mailto:saurabh@purestorage.com" target="_blank">saurabh@purestorage.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I don't think clearing kernel buffers is an issue here since syslog-ng is seeing the kernel messages during system boot up and logging them to /var/log/syslog. However, it fails to forward them to the remote server. So there is some issue with buffering messages for the remote destination.<div><br></div><div>-- Saurabh<br><div><br></div><div><div class="gmail_extra"><div class="gmail_quote">On Sat, Aug 22, 2015 at 12:00 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hmm. You don't even use /dev/kmsg or system, so this setup should work. Don't you happen to run anything that could read /proc/kmsg or clear the kernel ringbuffer behind the backs of syslog-ng?</p><div><div>
<div class="gmail_quote">On Aug 22, 2015 8:11 PM, "Saurabh Shukla" <<a href="mailto:saurabh@purestorage.com" target="_blank">saurabh@purestorage.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">Can you show your source declaration? </span></blockquote><div>I am using the syslog-ng.conf from here - <a href="https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf" target="_blank">https://github.com/balabit/syslog-ng/blob/syslog-ng-3.6.4/debian/syslog-ng.conf</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">Do you use systemd journal?</span></blockquote><div>No. </div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Aug 22, 2015 at 3:10 AM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Can you show your source declaration? Do you use systemd journal?</p>
<div class="gmail_quote"><div><div>On Aug 22, 2015 2:56 AM, "Saurabh Shukla" <<a href="mailto:saurabh@purestorage.com" target="_blank">saurabh@purestorage.com</a>> wrote:<br type="attribution"></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">I am running syslog-ng 3.6.4 and I have the following destination and log path configured that forwards all messages to the destination:<div><div><br></div><div><font face="monospace, monospace">destination remote {</font></div><div><font face="monospace, monospace"> network("<a href="http://remote.example.com" target="_blank">remote.example.com</a>" port(514) transport(tcp) log_fifo_size(2048));</font></div><div><font face="monospace, monospace">};</font></div><div><font face="monospace, monospace">log { source(s_all); destination(remote); flags(flow-control);};</font></div></div><div><br></div><div>When the system reboots, I see that startup messages from the kernel are logged into /var/log/syslog.</div><div>syslog-ng establishes a connection to the remote destination around 10 sec after the first message was logged into /var/log/syslog. However, it fails to forward any message that was logged into /var/log/syslog during the first 10 seconds even though I have the output buffer and flow control configured.<br></div><div><br></div><div>Is this a bug in syslog-ng or am I missing some configuration steps?</div><div><br></div><div>Thanks,</div><div>-- Saurabh</div></div>
<br></div></div>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>
</div></div><br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>
</div></div><br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>