[syslog-ng] log drop-and-delete directory

Nik Ambrosch nik at ambrosch.com
Tue Nov 6 06:34:19 UTC 2018 

i don't believe that's the case - the command syslog-ng-ctl status showed
these files as processed after they were removed even after a restart.
they only clear up after i remove the persist file and restart syslog-ng.
this is reproducible in 3.17.2, i have not tried 3.18 yet.

instead of the drop-box method i may instead just pipe output to logger and
run a few health checks (port listening, syslog-ng running) every time the
script runs.  this leaves one less mechanism for me to deal with (removing
files after they're read).


On Tue, Nov 6, 2018 at 12:37 AM Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> The files removed should be untracked automatically once the DELETED event
> is received from inotify, so they shouldn't be counted in max-files() once
> that happens.
>
> the persist file is cleaned up at restarts, quoting a comment from the
> persist-state.c file:
>  * Cleaning up:
>  * ------------
>  *
>  * It can be seen that no explicit deallocation is performed on the
>  * persistent file, in effect it could grow indefinitely. There's a
>  * simple cleanup procedure though:
>  *
>  *  - on every startup, the persist file is rewritten, entries with an
>  *    in_use bit set are copied to the new one, with the in_use bit cleared
>  *  - whenever syslog-ng looks up (e.g. uses) an entry, its in_use bit is
> set again
>  *
>  * This way unused entries in the persist file are reaped when
>  * syslog-ng restarts.
>  *
>
> stats are only in memory, and are not cleaned up, unless syslog-ng is
> restarted.
>
> In theory those cleanups could happen via syslog-ng-ctl or at reloads, but
> they don't today.
>
> On Mon, Nov 5, 2018 at 11:19 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>
>> is there a way to tell syslog-ng to stop tracking/following a file and
>> flush from syslog-ng stats?  i've noticed that when dropping a large number
>> of files into my temporary directory it's very easy to exceed max_files and
>> i'm not sure how long it takes to be removed from syslog-ng.persist
>> automatically (is it ever removed automatically?)
>>
>> when this is done i'll be fetching and deleting about five new log files
>> per minute.
>>
>>
>>
>> On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>>
>>> It seems most elegant to have syslog-ng take care of the file once it's
>>> done consuming, I don't think it would be against the role of the software
>>> to do so either.
>>>
>>> I wrote a script that seems to do the job if run periodically via cron,
>>> my biggest issue I have is a worst-case if there is an non-parsed file with
>>> a missing stamp, which would happen on a daemon reload.  I'm conflicted if
>>> the correct action is to remove the file or force a reload of the file.
>>>
>>>
>>> #!/usr/bin/env bash
>>>
>>> path='/data/tmp'
>>>
>>> # examine every file in directory
>>> for file in $(find $path -type f); do
>>>   # how many lines is in this file
>>>   lines=$(wc -l ${file} | awk '{print $1}')
>>>
>>>   # output of syslog-ng-ctl
>>>   ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
>>>
>>>   # how many lines syslog-ng has parsed
>>>   parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print
>>> $2}')
>>>
>>>   # when syslog-ng last consumed the file
>>>   stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
>>>
>>>   # debug
>>>   echo "file ${file} parsed ${parsed} of ${lines} lines"
>>>
>>>   # if file was parsed before a restart
>>>   if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then
>>>     echo "file ${file} processed before syslog-ng restart, removing"
>>>     #rm -f "${file}
>>>   fi
>>>
>>>   # if all lines in file were parsed
>>>   if [[ ${parsed} -eq ${lines} ]]; then
>>>     echo "file ${file} processed, removing"
>>>     #rm -f "${file}
>>>   fi
>>> done
>>>
>>>
>>> On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs <
>>> balazs.scheidler at oneidentity.com> wrote:
>>>
>>>> It would be possible to add an option to execute an external script
>>>> when eof is reached.
>>>> If i remember correctly the driver level has this information in the
>>>> form of a notification today. So it's only about adding the option and
>>>> calling system() on it.
>>>>
>>>> On the other hand, syslog-ng keeps statistics on every file it follows,
>>>> so the alternative is to poll syslog-ng-ctl stats and see if the counters
>>>> of the file is non-zero and delete it only in that case.
>>>>
>>>> On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik at ambrosch.com> wrote:
>>>>
>>>>> Thanks for the feedback.  The files contain predictable json data, new
>>>>> files arrive every 1-3 minutes (haven't decided yet).  There are no start
>>>>> and end markers.
>>>>>
>>>>> I'm wary of using cron to delete old files because if syslog-ng isn't
>>>>> able to consume the file (crashed, user error, upgrading package, etc) the
>>>>> non-consumed file will be deleted and contents will be lost.
>>>>>
>>>>> That same worst case applies to a separate script - if it provides the
>>>>> messages via syslog (instead of copying a file) but syslog-ng is unhealthy,
>>>>> then my messages are lost unless I build a buffer into the script and that
>>>>> starts to get complex.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> If there's a specific time for which a dropped file is specific to,
>>>>>> then just remove the file after a grace period with a simple cron job.
>>>>>>
>>>>>> On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy at oneidentity.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Nik,
>>>>>>>
>>>>>>> Syslog-ng should not be designed to delete files when it reaches
>>>>>>> EOF, it rather monitors the file for new lines if so.
>>>>>>> This would be a bit destructive behaviour even if it would be a
>>>>>>> feature with a control flag:
>>>>>>> source s_file_clearup {
>>>>>>>   wildcard-file (
>>>>>>>     base-dir("/tmp/")
>>>>>>>     filename-pattern("*")
>>>>>>>     remove-on-EOF(yes)
>>>>>>>   );
>>>>>>> };
>>>>>>> But if we are looking at from your point of view, it could be
>>>>>>> enhanced to have one-time files, or drop-off files.
>>>>>>> It could be an enhancement.
>>>>>>>
>>>>>>> With the current behaviour of syslog-ng quick ideas to solve this
>>>>>>> use case (if workaround needed):
>>>>>>> - syslog-ng closes a file after the reading is idle for time_reap
>>>>>>> seconds. This could be monitored externally and remove the given file.
>>>>>>> Example message "Destination timed out, reaping;
>>>>>>> template='input-logs', filename='input-logs"
>>>>>>> I think there is no EOF warning for files, as syslog-ng simply waits
>>>>>>> for new lines (as said above).
>>>>>>>
>>>>>>> Regards,
>>>>>>> Gabor
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik at ambrosch.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Looking to create a drop-off directory that syslog-ng handles
>>>>>>>> instead of needing to execute in a separate script.. flow would go
>>>>>>>> something like this:
>>>>>>>>
>>>>>>>> 1) mv file.log /syslog-tmp/
>>>>>>>> 2) syslog-ng reads /syslog-tmp/file.log
>>>>>>>> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
>>>>>>>>
>>>>>>>> Sounds simple but I can't seem to figure out a good way to do
>>>>>>>> this.  The other option is to read file with a script, send out with logger
>>>>>>>> (or whatever), and hope that syslog-ng is running & healthy.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181106/d8cd0bb3/attachment.html>

More information about the syslog-ng mailing list