[syslog-ng] syslog-ng leaves a lot of open file handles

[Anton Koldaev]

Just checked open deleted files and nothing has been written there after
23:59:59:

# ls -lA /proc/30743/fd | awk '/deleted/{print $8}' | xargs -I{} tail -1
/proc/30743/fd/{} | cut -c -15 | sort | uniq -c
...
      2 Jul 23 23:59:48
      1 Jul 23 23:59:49
      2 Jul 23 23:59:50
      4 Jul 23 23:59:52
      3 Jul 23 23:59:53
      1 Jul 23 23:59:54
      3 Jul 23 23:59:56
      7 Jul 23 23:59:57
      5 Jul 23 23:59:58
     28 Jul 23 23:59:59




On Wed, Jul 24, 2013 at 1:47 PM, Anton Koldaev <koldaevav at gmail.com> wrote:

> > It is a bit hard to believe that after receiving a HUP signal syslog-ng
> keeps destination files open, keep-alive isn't implemented there. did you
> signal the supervisor process maybe?
>
> *# pgrep -fl syslog-ng*
> 30742 supervising syslog-ng
> 30743 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --fd-limit 262144
>
> *# lsof -p 30743 | grep -c deleted*
> 285
>
> *# kill -HUP 30743*
>
> *# echo $?*
> 0
>
> *# lsof -p 30743 | grep -c deleted*
> 290
>
> >I'd check syslog-ng's messages.
>
> The only one message is there:
> *Jul 24 09:40:50 syslog-host syslog-ng[30743]: Configuration reload
> request received, reloading configuration;*
> *
> *
> *
> *
> > BTW did you check whether the file is still being written or not?
>
> Syslog-NG started to write to the new file at 23:59:59 just as it should.
> I'm seeing new log lines in the new log files started at 00:00:05. So it
> seems to be ok.
>
> > You're using the date extracted from the incoming log messages so when a
> client still sends logs with the given day then syslog-ng will keep writing
> to that file so it won't close it - thus if another process unlinked it
> then lsof will show the file as deleted.
>
> All the apps are configured to send logs in UTC as well as syslog-ng host
> is configured in UTC. Just re-checked it, the time seems to be in sync
> everywhere.
> *
> *
>
>
> On Wed, Jul 24, 2013 at 1:31 PM, Sandor Geller <
> Sandor.Geller at morganstanley.com> wrote:
>
>> It is a bit hard to believe that after receiving a HUP signal syslog-ng
>> keeps destination files open, keep-alive isn't implemented there. did you
>> signal the supervisor process maybe? I'd check syslog-ng's messages.
>>
>> BTW did you check whether the file is still being written or not? You're
>> using the date extracted from the incoming log messages so when a client
>> still sends logs with the given day then syslog-ng will keep writing to
>> that file so it won't close it - thus if another process unlinked it then
>> lsof will show the file as deleted.
>>
>>
>> On Wed, Jul 24, 2013 at 11:12 AM, Anton Koldaev <koldaevav at gmail.com>wrote:
>>
>>> Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid)
>>> And I have the following destination file():
>>>
>>> file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log"
>>>
>>> Syslog-NG switches to the new file at 23:59:59 every day just fine but
>>> for some reason it leaves files for the previous day open:
>>> *# date*
>>> Wed Jul 24 09:04:19 UTC 2013
>>> *# lsof | grep a/ac/account.log*
>>> syslog-ng 30743     root 3351w      REG              252,2    72597491
>>> 66306075 /u/logs/app/0723/a/ac/account.log (deleted)
>>> syslog-ng 30743     root 4896w      REG              252,2    17017519
>>>  4572052 /u/logs/app/0724/a/ac/account.log
>>>
>>> And they're being deleted by my rotating script.
>>> Reloading syslog-ng using init script or with `kill -HUP` doesn't help -
>>> all deleted files are still open by syslog-ng.
>>> Global option "time_reap (30);" doesn't seem to help too.
>>>
>>> Any ideas?
>>>
>>>
>>> --
>>> Best regards,
>>> Koldaev Anton
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> Best regards,
> Koldaev Anton
>



-- 
Best regards,
Koldaev Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130724/9ddcafe9/attachment.htm