[syslog-ng] syslog-ng leaves a lot of open file handles

Anton Koldaev koldaevav at gmail.com
Sat Sep 28 12:53:43 CEST 2013 

We still have the same problem.
I've tried to remove one particular file opened by syslog-ng and reload
syslog-ng process manually. It works fine. Here is the strace of reload
process:
https://gist.github.com/iroller/ef96983183eec5ac8d09

As I mentioned before we have logrotate that moves old files to archive
destination. So every time it rotates logs we have a bunch of open deleted
files. Reloading syslog-ng doesn't help this time. We have to restart it to
close them.
Here is the strace of reload process that's not closing any deleted files:
https://gist.github.com/iroller/9e9cbdbda8090c9a79c7

There are no errors or warnings in syslog-ng log at all.

Syslog-ng version and commands log:
https://gist.github.com/iroller/c0ff29ee2676196282cc

Sandor, Balazs could you let me know if there are some better ways to debug
it? It happens every day on logrotate so I can do any debugging to get more
info.


On Fri, Jul 26, 2013 at 10:02 PM, Balazs Scheidler <bazsi77 at gmail.com>wrote:

> This only happens if you have two file destinations with the same filename
> as target. Just for the sake for those who read this in the archives.
>  On Jul 24, 2013 1:27 PM, "Anton Koldaev" <koldaevav at gmail.com> wrote:
>
>> So there was another error in syslog-ng's log:
>>   Internal error, duplicate configuration elements refer to the same
>> persistent config; name='affile_dd_writers
>>
>> After fixing it syslog-ng reopens logs on HUP. Whew...
>>
>> Thanks @algernon (helped in IRC)
>>
>>
>> On Wed, Jul 24, 2013 at 1:59 PM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>
>>> Just checked open deleted files and nothing has been written there after
>>> 23:59:59:
>>>
>>> # ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1
>>> /proc/30743/fd/{} | cut -c -15 | sort | uniq -c
>>> ...
>>>       2 Jul 23 23:59:48
>>>       1 Jul 23 23:59:49
>>>       2 Jul 23 23:59:50
>>>       4 Jul 23 23:59:52
>>>       3 Jul 23 23:59:53
>>>       1 Jul 23 23:59:54
>>>       3 Jul 23 23:59:56
>>>       7 Jul 23 23:59:57
>>>       5 Jul 23 23:59:58
>>>      28 Jul 23 23:59:59
>>>
>>>
>>>
>>>
>>> On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>>
>>>> > It is a bit hard to believe that after receiving a HUP signal
>>>> syslog-ng keeps destination files open, keep-alive isn't implemented there.
>>>> did you signal the supervisor process maybe?
>>>>
>>>> *# pgrep -fl syslog-ng*
>>>> 30742 supervising syslog-ng
>>>> 30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144
>>>>
>>>> *# lsof -p 30743 | grep -c deleted*
>>>> 285
>>>>
>>>> *# kill -HUP 30743*
>>>>
>>>> *# echo $?*
>>>> 0
>>>>
>>>> *# lsof -p 30743 | grep -c deleted*
>>>> 290
>>>>
>>>> >I'd check syslog-ng's messages.
>>>>
>>>> The only one message is there:
>>>> *Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload
>>>> request received, reloading configuration;*
>>>> *
>>>> *
>>>> *
>>>> *
>>>> > BTW did you check whether the file is still being written or not?
>>>>
>>>> Syslog-NG started to write to the new file at 23:59:59 just as it
>>>> should. I'm seeing new log lines in the new log files started at 00:00:05.
>>>> So it seems to be ok.
>>>>
>>>> > You're using the date extracted from the incoming log messages so
>>>> when a client still sends logs with the given day then syslog-ng will keep
>>>> writing to that file so it won't close it - thus if another process
>>>> unlinked it then lsof will show the file as deleted.
>>>>
>>>> All the apps are configured to send logs in UTC as well as syslog-ng
>>>> host is configured in UTC. Just re-checked it, the time seems to be in sync
>>>> everywhere.
>>>> *
>>>> *
>>>>
>>>>
>>>> On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <
>>>> Sandor.Geller at morganstanley.com> wrote:
>>>>
>>>>> It is a bit hard to believe that after receiving a HUP signal
>>>>> syslog-ng keeps destination files open, keep-alive isn't implemented there.
>>>>> did you signal the supervisor process maybe? I'd check syslog-ng's messages.
>>>>>
>>>>> BTW did you check whether the file is still being written or not?
>>>>> You're using the date extracted from the incoming log messages so when a
>>>>> client still sends logs with the given day then syslog-ng will keep writing
>>>>> to that file so it won't close it - thus if another process unlinked it
>>>>> then lsof will show the file as deleted.
>>>>>
>>>>>
>>>>> On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>>>>
>>>>>> Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)
>>>>>> And I have the following destination file():
>>>>>>
>>>>>> file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"
>>>>>>
>>>>>> Syslog-NG switches to the new file at 23:59:59 every day just fine
>>>>>> but for some reason it leaves files for the previous day open:
>>>>>> *# date*
>>>>>> Wed Jul 24 09:04:19 UTC 2013
>>>>>> *# lsof | grep a/ac/account.log*
>>>>>> syslog-ng 30743     root 3351w      REG              252,2
>>>>>>  72597491   66306075 /u/logs/app/0723/a/ac/account.log (deleted)
>>>>>> syslog-ng 30743     root 4896w      REG              252,2
>>>>>>  17017519    4572052 /u/logs/app/0724/a/ac/account.log
>>>>>>
>>>>>> And they're being deleted by my rotating script.
>>>>>> Reloading syslog-ng using init script or with `kill -HUP` doesn't
>>>>>> help - all deleted files are still open by syslog-ng.
>>>>>> Global option "time_reap (30);" doesn't seem to help too.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Koldaev Anton
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Koldaev Anton
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Koldaev Anton
>>>
>>
>>
>>
>> --
>> Best regards,
>> Koldaev Anton
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>


-- 
Best regards,
Koldaev Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130928/11472f24/attachment.htm

More information about the syslog-ng mailing list