We still have the same problem.
I've tried to remove one particular file opened by syslog-ng and reload syslog-ng process manually. It works fine. Here is the strace of reload process:
https://gist.github.com/iroller/ef96983183eec5ac8d09

As I mentioned before we have logrotate that moves old files to archive destination. So every time it rotates logs we have a bunch of open deleted files. Reloading syslog-ng doesn't help this time. We have to restart it to close them.
Here is the strace of reload process that's not closing any deleted files:
https://gist.github.com/iroller/9e9cbdbda8090c9a79c7 

There are no errors or warnings in syslog-ng log at all.

Syslog-ng version and commands log:
https://gist.github.com/iroller/c0ff29ee2676196282cc

Sandor, Balazs could you let me know if there are some better ways to debug it? It happens every day on logrotate so I can do any debugging to get more info.


On Fri, Jul 26, 2013 at 10:02 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:

This only happens if you have two file destinations with the same filename as target. Just for the sake for those who read this in the archives.

On Jul 24, 2013 1:27 PM, "Anton Koldaev" <koldaevav@gmail.com> wrote:
So there was another error in syslog-ng's log:
  Internal error, duplicate configuration elements refer to the same persistent config; name='affile_dd_writers

After fixing it syslog-ng reopens logs on HUP. Whew...

Thanks @algernon (helped in IRC)


On Wed, Jul 24, 2013 at 1:59 PM, Anton Koldaev <koldaevav@gmail.com> wrote:
Just checked open deleted files and nothing has been written there after 23:59:59:

# ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1 /proc/30743/fd/{} | cut -c -15 | sort | uniq -c
...
      2 Jul 23 23:59:48
      1 Jul 23 23:59:49
      2 Jul 23 23:59:50
      4 Jul 23 23:59:52
      3 Jul 23 23:59:53
      1 Jul 23 23:59:54
      3 Jul 23 23:59:56
      7 Jul 23 23:59:57
      5 Jul 23 23:59:58
     28 Jul 23 23:59:59




On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <koldaevav@gmail.com> wrote:
> It is a bit hard to believe that after receiving a HUP signal syslog-ng keeps destination files open, keep-alive isn't implemented there. did you signal the supervisor process maybe?

# pgrep -fl syslog-ng
30742 supervising syslog-ng
30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144

# lsof -p 30743 | grep -c deleted
285

# kill -HUP 30743

# echo $?
0

# lsof -p 30743 | grep -c deleted
290

>I'd check syslog-ng's messages.

The only one message is there:
Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload request received, reloading configuration;


> BTW did you check whether the file is still being written or not?

Syslog-NG started to write to the new file at 23:59:59 just as it should. I'm seeing new log lines in the new log files started at 00:00:05. So it seems to be ok.

> You're using the date extracted from the incoming log messages so when a client still sends logs with the given day then syslog-ng will keep writing to that file so it won't close it - thus if another process unlinked it then lsof will show the file as deleted.

All the apps are configured to send logs in UTC as well as syslog-ng host is configured in UTC. Just re-checked it, the time seems to be in sync everywhere.



On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
It is a bit hard to believe that after receiving a HUP signal syslog-ng keeps destination files open, keep-alive isn't implemented there. did you signal the supervisor process maybe? I'd check syslog-ng's messages.

BTW did you check whether the file is still being written or not? You're using the date extracted from the incoming log messages so when a client still sends logs with the given day then syslog-ng will keep writing to that file so it won't close it - thus if another process unlinked it then lsof will show the file as deleted.


On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <koldaevav@gmail.com> wrote:
Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)
And I have the following destination file():
    file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"

Syslog-NG switches to the new file at 23:59:59 every day just fine but for some reason it leaves files for the previous day open:
# date
Wed Jul 24 09:04:19 UTC 2013
# lsof | grep a/ac/account.log
syslog-ng 30743     root 3351w      REG              252,2    72597491   66306075 /u/logs/app/0723/a/ac/account.log (deleted)
syslog-ng 30743     root 4896w      REG              252,2    17017519    4572052 /u/logs/app/0724/a/ac/account.log

And they're being deleted by my rotating script.
Reloading syslog-ng using init script or with `kill -HUP` doesn't help - all deleted files are still open by syslog-ng.
Global option "time_reap (30);" doesn't seem to help too.

Any ideas?


--
Best regards,
Koldaev Anton

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq





--
Best regards,
Koldaev Anton



--
Best regards,
Koldaev Anton



--
Best regards,
Koldaev Anton

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq





--
Best regards,
Koldaev Anton