So there was another error in syslog-ng's log:

  Internal error, duplicate configuration elements refer to the same persistent config; name='affile_dd_writers

After fixing it syslog-ng reopens logs on HUP. Whew...

Thanks @algernon (helped in IRC)

On Wed, Jul 24, 2013 at 1:59 PM, Anton Koldaev <koldaevav@gmail.com> wrote:

Just checked open deleted files and nothing has been written there after 23:59:59:

# ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1 /proc/30743/fd/{} | cut -c -15 | sort | uniq -c

...

      2 Jul 23 23:59:48

      1 Jul 23 23:59:49

      2 Jul 23 23:59:50

      4 Jul 23 23:59:52

      3 Jul 23 23:59:53

      1 Jul 23 23:59:54

      3 Jul 23 23:59:56

      7 Jul 23 23:59:57

      5 Jul 23 23:59:58

     28 Jul 23 23:59:59

On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <koldaevav@gmail.com> wrote:

> It is a bit hard to believe that after receiving a HUP signal syslog-ng keeps destination files open, keep-alive isn't implemented there. did you signal the supervisor process maybe?

# pgrep -fl syslog-ng

30742 supervising syslog-ng

30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144

# lsof -p 30743 | grep -c deleted

285

# kill -HUP 30743

# echo $?

0

# lsof -p 30743 | grep -c deleted

290

>I'd check syslog-ng's messages.

The only one message is there:

Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload request received, reloading configuration;

> BTW did you check whether the file is still being written or not?

Syslog-NG started to write to the new file at 23:59:59 just as it should. I'm seeing new log lines in the new log files started at 00:00:05. So it seems to be ok.

> You're using the date extracted from the incoming log messages so when a client still sends logs with the given day then syslog-ng will keep writing to that file so it won't close it - thus if another process unlinked it then lsof will show the file as deleted.

All the apps are configured to send logs in UTC as well as syslog-ng host is configured in UTC. Just re-checked it, the time seems to be in sync everywhere.

On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:

It is a bit hard to believe that after receiving a HUP signal syslog-ng keeps destination files open, keep-alive isn't implemented there. did you signal the supervisor process maybe? I'd check syslog-ng's messages.

BTW did you check whether the file is still being written or not? You're using the date extracted from the incoming log messages so when a client still sends logs with the given day then syslog-ng will keep writing to that file so it won't close it - thus if another process unlinked it then lsof will show the file as deleted.

On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <koldaevav@gmail.com> wrote:

Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)

And I have the following destination file():

    file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"

Syslog-NG switches to the new file at 23:59:59 every day just fine but for some reason it leaves files for the previous day open:

# date

Wed Jul 24 09:04:19 UTC 2013

# lsof | grep a/ac/account.log

syslog-ng 30743     root 3351w      REG              252,2    72597491   66306075 /u/logs/app/0723/a/ac/account.log (deleted)

syslog-ng 30743     root 4896w      REG              252,2    17017519    4572052 /u/logs/app/0724/a/ac/account.log

And they're being deleted by my rotating script.

Reloading syslog-ng using init script or with `kill -HUP` doesn't help - all deleted files are still open by syslog-ng.

Global option "time_reap (30);" doesn't seem to help too.

Any ideas?

--
Best regards,
Koldaev Anton

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

--
Best regards,
Koldaev Anton

--
Best regards,
Koldaev Anton

--
Best regards,
Koldaev Anton

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq