Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers. I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost. That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex. On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq