is there a way to tell syslog-ng to stop tracking/following a file and flush from syslog-ng stats? i've noticed that when dropping a large number of files into my temporary directory it's very easy to exceed max_files and i'm not sure how long it takes to be removed from syslog-ng.persist automatically (is it ever removed automatically?) when this is done i'll be fetching and deleting about five new log files per minute. On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik@ambrosch.com> wrote:
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.
I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file.
#!/usr/bin/env bash
path='/data/tmp'
# examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}')
# output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
# how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
# when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
# debug echo "file ${file} parsed ${parsed} of ${lines} lines"
# if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi
# if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done
On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq