[syslog-ng] log drop-and-delete directory

Mon Oct 15 16:36:26 UTC 2018

It seems most elegant to have syslog-ng take care of the file once it's
done consuming, I don't think it would be against the role of the software
to do so either.

I wrote a script that seems to do the job if run periodically via cron, my
biggest issue I have is a worst-case if there is an non-parsed file with a
missing stamp, which would happen on a daemon reload.  I'm conflicted if
the correct action is to remove the file or force a reload of the file.

#!/usr/bin/env bash

path='/data/tmp'

# examine every file in directory
for file in $(find $path -type f); do
  # how many lines is in this file
  lines=$(wc -l ${file} | awk '{print $1}')

  # output of syslog-ng-ctl
  ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)

  # how many lines syslog-ng has parsed
  parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')

  # when syslog-ng last consumed the file
  stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')

  # debug
  echo "file ${file} parsed ${parsed} of ${lines} lines"

  # if file was parsed before a restart
  if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then
    echo "file ${file} processed before syslog-ng restart, removing"
    #rm -f "${file}
  fi

  # if all lines in file were parsed
  if [[ ${parsed} -eq ${lines} ]]; then
    echo "file ${file} processed, removing"
    #rm -f "${file}
  fi
done

On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> It would be possible to add an option to execute an external script when
> eof is reached.
> If i remember correctly the driver level has this information in the form
> of a notification today. So it's only about adding the option and calling
> system() on it.
>
> On the other hand, syslog-ng keeps statistics on every file it follows, so
> the alternative is to poll syslog-ng-ctl stats and see if the counters of
> the file is non-zero and delete it only in that case.
>
> On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik at ambrosch.com> wrote:
>
>> Thanks for the feedback.  The files contain predictable json data, new
>> files arrive every 1-3 minutes (haven't decided yet).  There are no start
>> and end markers.
>>
>> I'm wary of using cron to delete old files because if syslog-ng isn't
>> able to consume the file (crashed, user error, upgrading package, etc) the
>> non-consumed file will be deleted and contents will be lost.
>>
>> That same worst case applies to a separate script - if it provides the
>> messages via syslog (instead of copying a file) but syslog-ng is unhealthy,
>> then my messages are lost unless I build a buffer into the script and that
>> starts to get complex.
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77 at gmail.com>
>> wrote:
>>
>>> If there's a specific time for which a dropped file is specific to, then
>>> just remove the file after a grace period with a simple cron job.
>>>
>>> On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy at oneidentity.com>
>>> wrote:
>>>
>>>> Hi Nik,
>>>>
>>>> Syslog-ng should not be designed to delete files when it reaches EOF,
>>>> it rather monitors the file for new lines if so.
>>>> This would be a bit destructive behaviour even if it would be a feature
>>>> with a control flag:
>>>> source s_file_clearup {
>>>>   wildcard-file (
>>>>     base-dir("/tmp/")
>>>>     filename-pattern("*")
>>>>     remove-on-EOF(yes)
>>>>   );
>>>> };
>>>> But if we are looking at from your point of view, it could be enhanced
>>>> to have one-time files, or drop-off files.
>>>> It could be an enhancement.
>>>>
>>>> With the current behaviour of syslog-ng quick ideas to solve this use
>>>> case (if workaround needed):
>>>> - syslog-ng closes a file after the reading is idle for time_reap
>>>> seconds. This could be monitored externally and remove the given file.
>>>> Example message "Destination timed out, reaping; template='input-logs',
>>>> filename='input-logs"
>>>> I think there is no EOF warning for files, as syslog-ng simply waits
>>>> for new lines (as said above).
>>>>
>>>> Regards,
>>>> Gabor
>>>>
>>>>
>>>> On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>>>>
>>>>> Looking to create a drop-off directory that syslog-ng handles instead
>>>>> of needing to execute in a separate script.. flow would go something like
>>>>> this:
>>>>>
>>>>> 1) mv file.log /syslog-tmp/
>>>>> 2) syslog-ng reads /syslog-tmp/file.log
>>>>> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
>>>>>
>>>>> Sounds simple but I can't seem to figure out a good way to do this.
>>>>> The other option is to read file with a script, send out with logger (or
>>>>> whatever), and hope that syslog-ng is running & healthy.
>>>>>
>>>>> Thanks.
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>> product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>> product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?
>>> product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181015/2e2879d9/attachment.html>