[syslog-ng] syslog-ng leaves a lot of open file handles

[Balazs Scheidler]

This only happens if you have two file destinations with the same filename
as target. Just for the sake for those who read this in the archives.
On Jul 24, 2013 1:27 PM, "Anton Koldaev" <koldaevav at gmail.com> wrote:

> So there was another error in syslog-ng's log:
>   Internal error, duplicate configuration elements refer to the same
> persistent config; name='affile_dd_writers
>
> After fixing it syslog-ng reopens logs on HUP. Whew...
>
> Thanks @algernon (helped in IRC)
>
>
> On Wed, Jul 24, 2013 at 1:59 PM, Anton Koldaev <koldaevav at gmail.com>wrote:
>
>> Just checked open deleted files and nothing has been written there after
>> 23:59:59:
>>
>> # ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1
>> /proc/30743/fd/{} | cut -c -15 | sort | uniq -c
>> ...
>>       2 Jul 23 23:59:48
>>       1 Jul 23 23:59:49
>>       2 Jul 23 23:59:50
>>       4 Jul 23 23:59:52
>>       3 Jul 23 23:59:53
>>       1 Jul 23 23:59:54
>>       3 Jul 23 23:59:56
>>       7 Jul 23 23:59:57
>>       5 Jul 23 23:59:58
>>      28 Jul 23 23:59:59
>>
>>
>>
>>
>> On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>
>>> > It is a bit hard to believe that after receiving a HUP signal
>>> syslog-ng keeps destination files open, keep-alive isn't implemented there.
>>> did you signal the supervisor process maybe?
>>>
>>> *# pgrep -fl syslog-ng*
>>> 30742 supervising syslog-ng
>>> 30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144
>>>
>>> *# lsof -p 30743 | grep -c deleted*
>>> 285
>>>
>>> *# kill -HUP 30743*
>>>
>>> *# echo $?*
>>> 0
>>>
>>> *# lsof -p 30743 | grep -c deleted*
>>> 290
>>>
>>> >I'd check syslog-ng's messages.
>>>
>>> The only one message is there:
>>> *Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload
>>> request received, reloading configuration;*
>>> *
>>> *
>>> *
>>> *
>>> > BTW did you check whether the file is still being written or not?
>>>
>>> Syslog-NG started to write to the new file at 23:59:59 just as it
>>> should. I'm seeing new log lines in the new log files started at 00:00:05.
>>> So it seems to be ok.
>>>
>>> > You're using the date extracted from the incoming log messages so when
>>> a client still sends logs with the given day then syslog-ng will keep
>>> writing to that file so it won't close it - thus if another process
>>> unlinked it then lsof will show the file as deleted.
>>>
>>> All the apps are configured to send logs in UTC as well as syslog-ng
>>> host is configured in UTC. Just re-checked it, the time seems to be in sync
>>> everywhere.
>>> *
>>> *
>>>
>>>
>>> On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <
>>> Sandor.Geller at morganstanley.com> wrote:
>>>
>>>> It is a bit hard to believe that after receiving a HUP signal syslog-ng
>>>> keeps destination files open, keep-alive isn't implemented there. did you
>>>> signal the supervisor process maybe? I'd check syslog-ng's messages.
>>>>
>>>> BTW did you check whether the file is still being written or not?
>>>> You're using the date extracted from the incoming log messages so when a
>>>> client still sends logs with the given day then syslog-ng will keep writing
>>>> to that file so it won't close it - thus if another process unlinked it
>>>> then lsof will show the file as deleted.
>>>>
>>>>
>>>> On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>>>
>>>>> Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)
>>>>> And I have the following destination file():
>>>>>
>>>>> file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"
>>>>>
>>>>> Syslog-NG switches to the new file at 23:59:59 every day just fine but
>>>>> for some reason it leaves files for the previous day open:
>>>>> *# date*
>>>>> Wed Jul 24 09:04:19 UTC 2013
>>>>> *# lsof | grep a/ac/account.log*
>>>>> syslog-ng 30743     root 3351w      REG              252,2    72597491
>>>>>   66306075 /u/logs/app/0723/a/ac/account.log (deleted)
>>>>> syslog-ng 30743     root 4896w      REG              252,2    17017519
>>>>>    4572052 /u/logs/app/0724/a/ac/account.log
>>>>>
>>>>> And they're being deleted by my rotating script.
>>>>> Reloading syslog-ng using init script or with `kill -HUP` doesn't help
>>>>> - all deleted files are still open by syslog-ng.
>>>>> Global option "time_reap (30);" doesn't seem to help too.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Koldaev Anton
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Koldaev Anton
>>>
>>
>>
>>
>> --
>> Best regards,
>> Koldaev Anton
>>
>
>
>
> --
> Best regards,
> Koldaev Anton
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130726/9e7a9c4d/attachment.htm