It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.

I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload.  I'm conflicted if the correct action is to remove the file or force a reload of the file.


#!/usr/bin/env bash

path='/data/tmp'

# examine every file in directory
for file in $(find $path -type f); do
  # how many lines is in this file
  lines=$(wc -l ${file} | awk '{print $1}')
 
  # output of syslog-ng-ctl
  ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
 
  # how many lines syslog-ng has parsed
  parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
 
  # when syslog-ng last consumed the file
  stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')

  # debug
  echo "file ${file} parsed ${parsed} of ${lines} lines"

  # if file was parsed before a restart
  if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then
    echo "file ${file} processed before syslog-ng restart, removing"
    #rm -f "${file}
  fi

  # if all lines in file were parsed
  if [[ ${parsed} -eq ${lines} ]]; then
    echo "file ${file} processed, removing"
    #rm -f "${file}
  fi
done


On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs <balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached.
If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.

On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.

On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback.  The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet).  There are no start and end markers.

I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.

That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.






On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.

On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,

Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so.
This would be a bit destructive behaviour even if it would be a feature with a control flag:
source s_file_clearup {
  wildcard-file (
    base-dir("/tmp/")
    filename-pattern("*")
    remove-on-EOF(yes)
  );
};
But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files.
It could be an enhancement.

With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed):
- syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file.
Example message "Destination timed out, reaping; template='input-logs', filename='input-logs"
I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).

Regards,
Gabor


On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:

1) mv file.log /syslog-tmp/
2) syslog-ng reads /syslog-tmp/file.log
3) syslog-ng deletes /syslog-tmp/file.log when done consuming

Sounds simple but I can't seem to figure out a good way to do this.  The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.

Thanks.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq