[syslog-ng] syslog-ng leaves a lot of open file handles

[Anton Koldaev]

So there was another error in syslog-ng's log:
  Internal error, duplicate configuration elements refer to the same
persistent config; name='affile_dd_writers

After fixing it syslog-ng reopens logs on HUP. Whew...

Thanks @algernon (helped in IRC)


On Wed, Jul 24, 2013 at 1:59 PM, Anton Koldaev <koldaevav at gmail.com> wrote:

> Just checked open deleted files and nothing has been written there after
> 23:59:59:
>
> # ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1
> /proc/30743/fd/{} | cut -c -15 | sort | uniq -c
> ...
>       2 Jul 23 23:59:48
>       1 Jul 23 23:59:49
>       2 Jul 23 23:59:50
>       4 Jul 23 23:59:52
>       3 Jul 23 23:59:53
>       1 Jul 23 23:59:54
>       3 Jul 23 23:59:56
>       7 Jul 23 23:59:57
>       5 Jul 23 23:59:58
>      28 Jul 23 23:59:59
>
>
>
>
> On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <koldaevav at gmail.com>wrote:
>
>> > It is a bit hard to believe that after receiving a HUP signal
>> syslog-ng keeps destination files open, keep-alive isn't implemented there.
>> did you signal the supervisor process maybe?
>>
>> *# pgrep -fl syslog-ng*
>> 30742 supervising syslog-ng
>> 30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144
>>
>> *# lsof -p 30743 | grep -c deleted*
>> 285
>>
>> *# kill -HUP 30743*
>>
>> *# echo $?*
>> 0
>>
>> *# lsof -p 30743 | grep -c deleted*
>> 290
>>
>> >I'd check syslog-ng's messages.
>>
>> The only one message is there:
>> *Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload
>> request received, reloading configuration;*
>> *
>> *
>> *
>> *
>> > BTW did you check whether the file is still being written or not?
>>
>> Syslog-NG started to write to the new file at 23:59:59 just as it should.
>> I'm seeing new log lines in the new log files started at 00:00:05. So it
>> seems to be ok.
>>
>> > You're using the date extracted from the incoming log messages so when
>> a client still sends logs with the given day then syslog-ng will keep
>> writing to that file so it won't close it - thus if another process
>> unlinked it then lsof will show the file as deleted.
>>
>> All the apps are configured to send logs in UTC as well as syslog-ng host
>> is configured in UTC. Just re-checked it, the time seems to be in sync
>> everywhere.
>> *
>> *
>>
>>
>> On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <
>> Sandor.Geller at morganstanley.com> wrote:
>>
>>> It is a bit hard to believe that after receiving a HUP signal syslog-ng
>>> keeps destination files open, keep-alive isn't implemented there. did you
>>> signal the supervisor process maybe? I'd check syslog-ng's messages.
>>>
>>> BTW did you check whether the file is still being written or not? You're
>>> using the date extracted from the incoming log messages so when a client
>>> still sends logs with the given day then syslog-ng will keep writing to
>>> that file so it won't close it - thus if another process unlinked it then
>>> lsof will show the file as deleted.
>>>
>>>
>>> On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>>
>>>> Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)
>>>> And I have the following destination file():
>>>>
>>>> file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"
>>>>
>>>> Syslog-NG switches to the new file at 23:59:59 every day just fine but
>>>> for some reason it leaves files for the previous day open:
>>>> *# date*
>>>> Wed Jul 24 09:04:19 UTC 2013
>>>> *# lsof | grep a/ac/account.log*
>>>> syslog-ng 30743     root 3351w      REG              252,2    72597491
>>>>   66306075 /u/logs/app/0723/a/ac/account.log (deleted)
>>>> syslog-ng 30743     root 4896w      REG              252,2    17017519
>>>>    4572052 /u/logs/app/0724/a/ac/account.log
>>>>
>>>> And they're being deleted by my rotating script.
>>>> Reloading syslog-ng using init script or with `kill -HUP` doesn't help
>>>> - all deleted files are still open by syslog-ng.
>>>> Global option "time_reap (30);" doesn't seem to help too.
>>>>
>>>> Any ideas?
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Koldaev Anton
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>> Koldaev Anton
>>
>
>
>
> --
> Best regards,
> Koldaev Anton
>



-- 
Best regards,
Koldaev Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130724/14e2f81a/attachment-0001.htm