<div dir="ltr">So there was another error in syslog-ng's log:<div> Internal error, duplicate configuration elements refer to the same persistent config; name='affile_dd_writers<br></div><div><br></div><div style>After fixing it syslog-ng reopens logs on HUP. Whew...</div>
<div style><br></div><div style>Thanks @<span style="background-color:rgb(240,247,255);color:rgb(0,0,0);font-family:Consolas,'Lucida Console',monospace;font-size:13px">algernon (helped in IRC)</span></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jul 24, 2013 at 1:59 PM, Anton Koldaev <span dir="ltr"><<a href="mailto:koldaevav@gmail.com" target="_blank">koldaevav@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Just checked open deleted files and nothing has been written there after 23:59:59:<div><div><br></div><div># ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1 /proc/30743/fd/{} | cut -c -15 | sort | uniq -c</div>
<div>...</div></div><div><div> 2 Jul 23 23:59:48</div><div> 1 Jul 23 23:59:49</div><div> 2 Jul 23 23:59:50</div><div> 4 Jul 23 23:59:52</div><div> 3 Jul 23 23:59:53</div><div> 1 Jul 23 23:59:54</div>
<div> 3 Jul 23 23:59:56</div><div> 7 Jul 23 23:59:57</div><div> 5 Jul 23 23:59:58</div><div> 28 Jul 23 23:59:59</div></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">
On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <span dir="ltr"><<a href="mailto:koldaevav@gmail.com" target="_blank">koldaevav@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>> <span style="font-family:arial,sans-serif;font-size:13px">It is a bit hard to believe that after receiving a HUP signal syslog-ng keeps destination files open, keep-alive isn't implemented there. did you signal the supervisor process maybe?</span><div>
<span style="font-family:arial,sans-serif;font-size:13px"><br></span></div></div><div><span style="font-family:arial,sans-serif;font-size:13px"><div><b># pgrep -fl syslog-ng</b></div><div>30742 supervising syslog-ng</div>
<div>30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144</div>
<div><br></div><div><b># lsof -p 30743 | grep -c deleted</b></div><div>285</div><div><br></div><div><b># kill -HUP 30743</b></div><div><br></div><div><div><b># echo $?</b></div><div>0</div></div><div><br></div><div><b># lsof -p 30743 | grep -c deleted</b></div>
<div>290</div></span></div><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">>I'd check syslog-ng's messages.</span></div>
<div><font face="arial, sans-serif"><br></font></div></div><div><font face="arial, sans-serif">The only one message is there:</font></div><div><font face="arial, sans-serif"><b>Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload request received, reloading configuration;</b></font><br>
</div><div><div><font face="arial, sans-serif"><b><br></b></font></div><div><font face="arial, sans-serif"><b><br></b></font></div><div><span style="font-family:arial,sans-serif;font-size:13px">> BTW did you check whether the file is still being written or not?</span></div>
<div><font face="arial, sans-serif"><br></font></div></div><div><font face="arial, sans-serif">Syslog-NG started to write to the new file at 23:59:59 just as it should. I'm seeing new log lines in the new log files started at 00:00:05. So it seems to be ok.</font></div>
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">> You're using the date extracted from the incoming log messages so when a client still sends logs with the given day then syslog-ng will keep writing to that file so it won't close it - thus if another process unlinked it then lsof will show the file as deleted.</span><br>
</div><div><br></div></div><div>All the apps are configured to send logs in UTC as well as syslog-ng host is configured in UTC. Just re-checked it, the time seems to be in sync everywhere.</div><div><font face="arial, sans-serif"><b><br>
</b></font></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <span dir="ltr"><<a href="mailto:Sandor.Geller@morganstanley.com" target="_blank">Sandor.Geller@morganstanley.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It is a bit hard to believe that after receiving a HUP signal syslog-ng keeps destination files open, keep-alive isn't implemented there. did you signal the supervisor process maybe? I'd check syslog-ng's messages.<br>
<br>BTW did you check whether the file is still being written or not? You're using the date extracted from the incoming log messages so when a client still sends logs with the given day then syslog-ng will keep writing to that file so it won't close it - thus if another process unlinked it then lsof will show the file as deleted.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <span dir="ltr"><<a href="mailto:koldaevav@gmail.com" target="_blank">koldaevav@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)<div>
And I have the following destination file():</div>
<div><div> file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"</div>
<div><br></div><div>Syslog-NG switches to the new file at 23:59:59 every day just fine but for some reason it leaves files for the previous day open:</div><div><div><b># date</b></div><div>Wed Jul 24 09:04:19 UTC 2013</div>
<div><div><b># lsof | grep a/ac/account.log</b></div><div>syslog-ng 30743 root 3351w REG 252,2 72597491 66306075 /u/logs/app/0723/a/ac/account.log (deleted)</div><div>syslog-ng 30743 root 4896w REG 252,2 17017519 4572052 /u/logs/app/0724/a/ac/account.log</div>
</div><div><br></div><div>And they're being deleted by my rotating script.</div><div>Reloading syslog-ng using init script or with `kill -HUP` doesn't help - all deleted files are still open by syslog-ng.</div>
<div><div>Global option "time_reap (30);" doesn't seem to help too.</div><div><br></div><div>Any ideas?</div><span><font color="#888888"><div><br></div></font></span></div></div><span><font color="#888888"><div>
<br></div>-- <br>Best regards,<br>Koldaev Anton
</font></span></div></div>
<br></div></div>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>