log drop-and-delete directory
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this: 1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy. Thanks.
Hi Nik, Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement. With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above). Regards, Gabor On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job. On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers. I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost. That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex. On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it. On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case. On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either. I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file. #!/usr/bin/env bash path='/data/tmp' # examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}') # output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*) # how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}') # when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}') # debug echo "file ${file} parsed ${parsed} of ${lines} lines" # if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi # if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
is there a way to tell syslog-ng to stop tracking/following a file and flush from syslog-ng stats? i've noticed that when dropping a large number of files into my temporary directory it's very easy to exceed max_files and i'm not sure how long it takes to be removed from syslog-ng.persist automatically (is it ever removed automatically?) when this is done i'll be fetching and deleting about five new log files per minute. On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik@ambrosch.com> wrote:
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.
I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file.
#!/usr/bin/env bash
path='/data/tmp'
# examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}')
# output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
# how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
# when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
# debug echo "file ${file} parsed ${parsed} of ${lines} lines"
# if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi
# if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done
On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:
1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.
Thanks.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The files removed should be untracked automatically once the DELETED event is received from inotify, so they shouldn't be counted in max-files() once that happens. the persist file is cleaned up at restarts, quoting a comment from the persist-state.c file: * Cleaning up: * ------------ * * It can be seen that no explicit deallocation is performed on the * persistent file, in effect it could grow indefinitely. There's a * simple cleanup procedure though: * * - on every startup, the persist file is rewritten, entries with an * in_use bit set are copied to the new one, with the in_use bit cleared * - whenever syslog-ng looks up (e.g. uses) an entry, its in_use bit is set again * * This way unused entries in the persist file are reaped when * syslog-ng restarts. * stats are only in memory, and are not cleaned up, unless syslog-ng is restarted. In theory those cleanups could happen via syslog-ng-ctl or at reloads, but they don't today. On Mon, Nov 5, 2018 at 11:19 PM Nik Ambrosch <nik@ambrosch.com> wrote:
is there a way to tell syslog-ng to stop tracking/following a file and flush from syslog-ng stats? i've noticed that when dropping a large number of files into my temporary directory it's very easy to exceed max_files and i'm not sure how long it takes to be removed from syslog-ng.persist automatically (is it ever removed automatically?)
when this is done i'll be fetching and deleting about five new log files per minute.
On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik@ambrosch.com> wrote:
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.
I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file.
#!/usr/bin/env bash
path='/data/tmp'
# examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}')
# output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
# how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
# when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
# debug echo "file ${file} parsed ${parsed} of ${lines} lines"
# if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi
# if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done
On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
Hi Nik,
Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so. This would be a bit destructive behaviour even if it would be a feature with a control flag: source s_file_clearup { wildcard-file ( base-dir("/tmp/") filename-pattern("*") remove-on-EOF(yes) ); }; But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files. It could be an enhancement.
With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed): - syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file. Example message "Destination timed out, reaping; template='input-logs', filename='input-logs" I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).
Regards, Gabor
On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> wrote:
> Looking to create a drop-off directory that syslog-ng handles > instead of needing to execute in a separate script.. flow would go > something like this: > > 1) mv file.log /syslog-tmp/ > 2) syslog-ng reads /syslog-tmp/file.log > 3) syslog-ng deletes /syslog-tmp/file.log when done consuming > > Sounds simple but I can't seem to figure out a good way to do this. > The other option is to read file with a script, send out with logger (or > whatever), and hope that syslog-ng is running & healthy. > > Thanks. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
i don't believe that's the case - the command syslog-ng-ctl status showed these files as processed after they were removed even after a restart. they only clear up after i remove the persist file and restart syslog-ng. this is reproducible in 3.17.2, i have not tried 3.18 yet. instead of the drop-box method i may instead just pipe output to logger and run a few health checks (port listening, syslog-ng running) every time the script runs. this leaves one less mechanism for me to deal with (removing files after they're read). On Tue, Nov 6, 2018 at 12:37 AM Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
The files removed should be untracked automatically once the DELETED event is received from inotify, so they shouldn't be counted in max-files() once that happens.
the persist file is cleaned up at restarts, quoting a comment from the persist-state.c file: * Cleaning up: * ------------ * * It can be seen that no explicit deallocation is performed on the * persistent file, in effect it could grow indefinitely. There's a * simple cleanup procedure though: * * - on every startup, the persist file is rewritten, entries with an * in_use bit set are copied to the new one, with the in_use bit cleared * - whenever syslog-ng looks up (e.g. uses) an entry, its in_use bit is set again * * This way unused entries in the persist file are reaped when * syslog-ng restarts. *
stats are only in memory, and are not cleaned up, unless syslog-ng is restarted.
In theory those cleanups could happen via syslog-ng-ctl or at reloads, but they don't today.
On Mon, Nov 5, 2018 at 11:19 PM Nik Ambrosch <nik@ambrosch.com> wrote:
is there a way to tell syslog-ng to stop tracking/following a file and flush from syslog-ng stats? i've noticed that when dropping a large number of files into my temporary directory it's very easy to exceed max_files and i'm not sure how long it takes to be removed from syslog-ng.persist automatically (is it ever removed automatically?)
when this is done i'll be fetching and deleting about five new log files per minute.
On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik@ambrosch.com> wrote:
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.
I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file.
#!/usr/bin/env bash
path='/data/tmp'
# examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}')
# output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
# how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
# when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
# debug echo "file ${file} parsed ${parsed} of ${lines} lines"
# if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi
# if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done
On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
> Hi Nik, > > Syslog-ng should not be designed to delete files when it reaches > EOF, it rather monitors the file for new lines if so. > This would be a bit destructive behaviour even if it would be a > feature with a control flag: > source s_file_clearup { > wildcard-file ( > base-dir("/tmp/") > filename-pattern("*") > remove-on-EOF(yes) > ); > }; > But if we are looking at from your point of view, it could be > enhanced to have one-time files, or drop-off files. > It could be an enhancement. > > With the current behaviour of syslog-ng quick ideas to solve this > use case (if workaround needed): > - syslog-ng closes a file after the reading is idle for time_reap > seconds. This could be monitored externally and remove the given file. > Example message "Destination timed out, reaping; > template='input-logs', filename='input-logs" > I think there is no EOF warning for files, as syslog-ng simply waits > for new lines (as said above). > > Regards, > Gabor > > > On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> > wrote: > >> Looking to create a drop-off directory that syslog-ng handles >> instead of needing to execute in a separate script.. flow would go >> something like this: >> >> 1) mv file.log /syslog-tmp/ >> 2) syslog-ng reads /syslog-tmp/file.log >> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming >> >> Sounds simple but I can't seem to figure out a good way to do >> this. The other option is to read file with a script, send out with logger >> (or whatever), and hope that syslog-ng is running & healthy. >> >> Thanks. >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Note the distinction between restart and reload. are you sure you actually restarted syslog-ng? My understanding is that we are scanning the directory for files and if the file is there, we won't be able to find it, thus no means to reopen it. But i might be mistaken, it's been a while since i looked at that code. On Nov 6, 2018 07:34, "Nik Ambrosch" <nik@ambrosch.com> wrote: i don't believe that's the case - the command syslog-ng-ctl status showed these files as processed after they were removed even after a restart. they only clear up after i remove the persist file and restart syslog-ng. this is reproducible in 3.17.2, i have not tried 3.18 yet. instead of the drop-box method i may instead just pipe output to logger and run a few health checks (port listening, syslog-ng running) every time the script runs. this leaves one less mechanism for me to deal with (removing files after they're read). On Tue, Nov 6, 2018 at 12:37 AM Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
The files removed should be untracked automatically once the DELETED event is received from inotify, so they shouldn't be counted in max-files() once that happens.
the persist file is cleaned up at restarts, quoting a comment from the persist-state.c file: * Cleaning up: * ------------ * * It can be seen that no explicit deallocation is performed on the * persistent file, in effect it could grow indefinitely. There's a * simple cleanup procedure though: * * - on every startup, the persist file is rewritten, entries with an * in_use bit set are copied to the new one, with the in_use bit cleared * - whenever syslog-ng looks up (e.g. uses) an entry, its in_use bit is set again * * This way unused entries in the persist file are reaped when * syslog-ng restarts. *
stats are only in memory, and are not cleaned up, unless syslog-ng is restarted.
In theory those cleanups could happen via syslog-ng-ctl or at reloads, but they don't today.
On Mon, Nov 5, 2018 at 11:19 PM Nik Ambrosch <nik@ambrosch.com> wrote:
is there a way to tell syslog-ng to stop tracking/following a file and flush from syslog-ng stats? i've noticed that when dropping a large number of files into my temporary directory it's very easy to exceed max_files and i'm not sure how long it takes to be removed from syslog-ng.persist automatically (is it ever removed automatically?)
when this is done i'll be fetching and deleting about five new log files per minute.
On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik@ambrosch.com> wrote:
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.
I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file.
#!/usr/bin/env bash
path='/data/tmp'
# examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}')
# output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
# how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
# when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
# debug echo "file ${file} parsed ${parsed} of ${lines} lines"
# if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi
# if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done
On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
> Hi Nik, > > Syslog-ng should not be designed to delete files when it reaches > EOF, it rather monitors the file for new lines if so. > This would be a bit destructive behaviour even if it would be a > feature with a control flag: > source s_file_clearup { > wildcard-file ( > base-dir("/tmp/") > filename-pattern("*") > remove-on-EOF(yes) > ); > }; > But if we are looking at from your point of view, it could be > enhanced to have one-time files, or drop-off files. > It could be an enhancement. > > With the current behaviour of syslog-ng quick ideas to solve this > use case (if workaround needed): > - syslog-ng closes a file after the reading is idle for time_reap > seconds. This could be monitored externally and remove the given file. > Example message "Destination timed out, reaping; > template='input-logs', filename='input-logs" > I think there is no EOF warning for files, as syslog-ng simply waits > for new lines (as said above). > > Regards, > Gabor > > > On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> > wrote: > >> Looking to create a drop-off directory that syslog-ng handles >> instead of needing to execute in a separate script.. flow would go >> something like this: >> >> 1) mv file.log /syslog-tmp/ >> 2) syslog-ng reads /syslog-tmp/file.log >> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming >> >> Sounds simple but I can't seem to figure out a good way to do >> this. The other option is to read file with a script, send out with logger >> (or whatever), and hope that syslog-ng is running & healthy. >> >> Thanks. >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Another suggestion: as you seem to control the file's contents, why not add an EOF marker, add a match filter or patterndb rule that matches that marker, and feed the $FILE_NAME to a program destination that would delete the file?
participants (6)
-
Balazs Scheidler
-
Fabien Wernli
-
Nagy, Gábor
-
Nik Ambrosch
-
Scheidler, Balázs
-
Scheidler, Balázs