[syslog-ng] log drop-and-delete directory

Mon Nov 5 22:19:17 UTC 2018

is there a way to tell syslog-ng to stop tracking/following a file and
flush from syslog-ng stats?  i've noticed that when dropping a large number
of files into my temporary directory it's very easy to exceed max_files and
i'm not sure how long it takes to be removed from syslog-ng.persist
automatically (is it ever removed automatically?)

when this is done i'll be fetching and deleting about five new log files
per minute.

On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik at ambrosch.com> wrote:

> It seems most elegant to have syslog-ng take care of the file once it's
> done consuming, I don't think it would be against the role of the software
> to do so either.
>
> I wrote a script that seems to do the job if run periodically via cron, my
> biggest issue I have is a worst-case if there is an non-parsed file with a
> missing stamp, which would happen on a daemon reload.  I'm conflicted if
> the correct action is to remove the file or force a reload of the file.
>
>
> #!/usr/bin/env bash
>
> path='/data/tmp'
>
> # examine every file in directory
> for file in $(find $path -type f); do
>   # how many lines is in this file
>   lines=$(wc -l ${file} | awk '{print $1}')
>
>   # output of syslog-ng-ctl
>   ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
>
>   # how many lines syslog-ng has parsed
>   parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
>
>   # when syslog-ng last consumed the file
>   stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
>
>   # debug
>   echo "file ${file} parsed ${parsed} of ${lines} lines"
>
>   # if file was parsed before a restart
>   if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then
>     echo "file ${file} processed before syslog-ng restart, removing"
>     #rm -f "${file}
>   fi
>
>   # if all lines in file were parsed
>   if [[ ${parsed} -eq ${lines} ]]; then
>     echo "file ${file} processed, removing"
>     #rm -f "${file}
>   fi
> done
>
>
> On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs <
> balazs.scheidler at oneidentity.com> wrote:
>
>> It would be possible to add an option to execute an external script when
>> eof is reached.
>> If i remember correctly the driver level has this information in the form
>> of a notification today. So it's only about adding the option and calling
>> system() on it.
>>
>> On the other hand, syslog-ng keeps statistics on every file it follows,
>> so the alternative is to poll syslog-ng-ctl stats and see if the counters
>> of the file is non-zero and delete it only in that case.
>>
>> On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik at ambrosch.com> wrote:
>>
>>> Thanks for the feedback.  The files contain predictable json data, new
>>> files arrive every 1-3 minutes (haven't decided yet).  There are no start
>>> and end markers.
>>>
>>> I'm wary of using cron to delete old files because if syslog-ng isn't
>>> able to consume the file (crashed, user error, upgrading package, etc) the
>>> non-consumed file will be deleted and contents will be lost.
>>>
>>> That same worst case applies to a separate script - if it provides the
>>> messages via syslog (instead of copying a file) but syslog-ng is unhealthy,
>>> then my messages are lost unless I build a buffer into the script and that
>>> starts to get complex.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77 at gmail.com>
>>> wrote:
>>>
>>>> If there's a specific time for which a dropped file is specific to,
>>>> then just remove the file after a grace period with a simple cron job.
>>>>
>>>> On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy at oneidentity.com>
>>>> wrote:
>>>>
>>>>> Hi Nik,
>>>>>
>>>>> Syslog-ng should not be designed to delete files when it reaches EOF,
>>>>> it rather monitors the file for new lines if so.
>>>>> This would be a bit destructive behaviour even if it would be a
>>>>> feature with a control flag:
>>>>> source s_file_clearup {
>>>>>   wildcard-file (
>>>>>     base-dir("/tmp/")
>>>>>     filename-pattern("*")
>>>>>     remove-on-EOF(yes)
>>>>>   );
>>>>> };
>>>>> But if we are looking at from your point of view, it could be enhanced
>>>>> to have one-time files, or drop-off files.
>>>>> It could be an enhancement.
>>>>>
>>>>> With the current behaviour of syslog-ng quick ideas to solve this use
>>>>> case (if workaround needed):
>>>>> - syslog-ng closes a file after the reading is idle for time_reap
>>>>> seconds. This could be monitored externally and remove the given file.
>>>>> Example message "Destination timed out, reaping;
>>>>> template='input-logs', filename='input-logs"
>>>>> I think there is no EOF warning for files, as syslog-ng simply waits
>>>>> for new lines (as said above).
>>>>>
>>>>> Regards,
>>>>> Gabor
>>>>>
>>>>>
>>>>> On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>>>>>
>>>>>> Looking to create a drop-off directory that syslog-ng handles instead
>>>>>> of needing to execute in a separate script.. flow would go something like
>>>>>> this:
>>>>>>
>>>>>> 1) mv file.log /syslog-tmp/
>>>>>> 2) syslog-ng reads /syslog-tmp/file.log
>>>>>> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
>>>>>>
>>>>>> Sounds simple but I can't seem to figure out a good way to do this.
>>>>>> The other option is to read file with a script, send out with logger (or
>>>>>> whatever), and hope that syslog-ng is running & healthy.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181105/f61610f2/attachment-0001.html>