[syslog-ng] log drop-and-delete directory

Tue Nov 6 05:37:26 UTC 2018

The files removed should be untracked automatically once the DELETED event
is received from inotify, so they shouldn't be counted in max-files() once
that happens.

the persist file is cleaned up at restarts, quoting a comment from the
persist-state.c file:
 * Cleaning up:
 * ------------
 *
 * It can be seen that no explicit deallocation is performed on the
 * persistent file, in effect it could grow indefinitely. There's a
 * simple cleanup procedure though:
 *
 *  - on every startup, the persist file is rewritten, entries with an
 *    in_use bit set are copied to the new one, with the in_use bit cleared
 *  - whenever syslog-ng looks up (e.g. uses) an entry, its in_use bit is
set again
 *
 * This way unused entries in the persist file are reaped when
 * syslog-ng restarts.
 *

stats are only in memory, and are not cleaned up, unless syslog-ng is
restarted.

In theory those cleanups could happen via syslog-ng-ctl or at reloads, but
they don't today.

On Mon, Nov 5, 2018 at 11:19 PM Nik Ambrosch <nik at ambrosch.com> wrote:

> is there a way to tell syslog-ng to stop tracking/following a file and
> flush from syslog-ng stats?  i've noticed that when dropping a large number
> of files into my temporary directory it's very easy to exceed max_files and
> i'm not sure how long it takes to be removed from syslog-ng.persist
> automatically (is it ever removed automatically?)
>
> when this is done i'll be fetching and deleting about five new log files
> per minute.
>
>
>
> On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>
>> It seems most elegant to have syslog-ng take care of the file once it's
>> done consuming, I don't think it would be against the role of the software
>> to do so either.
>>
>> I wrote a script that seems to do the job if run periodically via cron,
>> my biggest issue I have is a worst-case if there is an non-parsed file with
>> a missing stamp, which would happen on a daemon reload.  I'm conflicted if
>> the correct action is to remove the file or force a reload of the file.
>>
>>
>> #!/usr/bin/env bash
>>
>> path='/data/tmp'
>>
>> # examine every file in directory
>> for file in $(find $path -type f); do
>>   # how many lines is in this file
>>   lines=$(wc -l ${file} | awk '{print $1}')
>>
>>   # output of syslog-ng-ctl
>>   ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
>>
>>   # how many lines syslog-ng has parsed
>>   parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print
>> $2}')
>>
>>   # when syslog-ng last consumed the file
>>   stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
>>
>>   # debug
>>   echo "file ${file} parsed ${parsed} of ${lines} lines"
>>
>>   # if file was parsed before a restart
>>   if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then
>>     echo "file ${file} processed before syslog-ng restart, removing"
>>     #rm -f "${file}
>>   fi
>>
>>   # if all lines in file were parsed
>>   if [[ ${parsed} -eq ${lines} ]]; then
>>     echo "file ${file} processed, removing"
>>     #rm -f "${file}
>>   fi
>> done
>>
>>
>> On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs <
>> balazs.scheidler at oneidentity.com> wrote:
>>
>>> It would be possible to add an option to execute an external script when
>>> eof is reached.
>>> If i remember correctly the driver level has this information in the
>>> form of a notification today. So it's only about adding the option and
>>> calling system() on it.
>>>
>>> On the other hand, syslog-ng keeps statistics on every file it follows,
>>> so the alternative is to poll syslog-ng-ctl stats and see if the counters
>>> of the file is non-zero and delete it only in that case.
>>>
>>> On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik at ambrosch.com> wrote:
>>>
>>>> Thanks for the feedback.  The files contain predictable json data, new
>>>> files arrive every 1-3 minutes (haven't decided yet).  There are no start
>>>> and end markers.
>>>>
>>>> I'm wary of using cron to delete old files because if syslog-ng isn't
>>>> able to consume the file (crashed, user error, upgrading package, etc) the
>>>> non-consumed file will be deleted and contents will be lost.
>>>>
>>>> That same worst case applies to a separate script - if it provides the
>>>> messages via syslog (instead of copying a file) but syslog-ng is unhealthy,
>>>> then my messages are lost unless I build a buffer into the script and that
>>>> starts to get complex.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77 at gmail.com>
>>>> wrote:
>>>>
>>>>> If there's a specific time for which a dropped file is specific to,
>>>>> then just remove the file after a grace period with a simple cron job.
>>>>>
>>>>> On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy at oneidentity.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nik,
>>>>>>
>>>>>> Syslog-ng should not be designed to delete files when it reaches EOF,
>>>>>> it rather monitors the file for new lines if so.
>>>>>> This would be a bit destructive behaviour even if it would be a
>>>>>> feature with a control flag:
>>>>>> source s_file_clearup {
>>>>>>   wildcard-file (
>>>>>>     base-dir("/tmp/")
>>>>>>     filename-pattern("*")
>>>>>>     remove-on-EOF(yes)
>>>>>>   );
>>>>>> };
>>>>>> But if we are looking at from your point of view, it could be
>>>>>> enhanced to have one-time files, or drop-off files.
>>>>>> It could be an enhancement.
>>>>>>
>>>>>> With the current behaviour of syslog-ng quick ideas to solve this use
>>>>>> case (if workaround needed):
>>>>>> - syslog-ng closes a file after the reading is idle for time_reap
>>>>>> seconds. This could be monitored externally and remove the given file.
>>>>>> Example message "Destination timed out, reaping;
>>>>>> template='input-logs', filename='input-logs"
>>>>>> I think there is no EOF warning for files, as syslog-ng simply waits
>>>>>> for new lines (as said above).
>>>>>>
>>>>>> Regards,
>>>>>> Gabor
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik at ambrosch.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Looking to create a drop-off directory that syslog-ng handles
>>>>>>> instead of needing to execute in a separate script.. flow would go
>>>>>>> something like this:
>>>>>>>
>>>>>>> 1) mv file.log /syslog-tmp/
>>>>>>> 2) syslog-ng reads /syslog-tmp/file.log
>>>>>>> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
>>>>>>>
>>>>>>> Sounds simple but I can't seem to figure out a good way to do this.
>>>>>>> The other option is to read file with a script, send out with logger (or
>>>>>>> whatever), and hope that syslog-ng is running & healthy.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181106/7abeed0f/attachment.html>