Note the distinction between restart and reload. are you sure you actually restarted syslog-ng? My understanding is that we are scanning the directory for files and if the file is there, we won't be able to find it, thus no means to reopen it. But i might be mistaken, it's been a while since i looked at that code. On Nov 6, 2018 07:34, "Nik Ambrosch" <nik@ambrosch.com> wrote: i don't believe that's the case - the command syslog-ng-ctl status showed these files as processed after they were removed even after a restart. they only clear up after i remove the persist file and restart syslog-ng. this is reproducible in 3.17.2, i have not tried 3.18 yet. instead of the drop-box method i may instead just pipe output to logger and run a few health checks (port listening, syslog-ng running) every time the script runs. this leaves one less mechanism for me to deal with (removing files after they're read). On Tue, Nov 6, 2018 at 12:37 AM Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
The files removed should be untracked automatically once the DELETED event is received from inotify, so they shouldn't be counted in max-files() once that happens.
the persist file is cleaned up at restarts, quoting a comment from the persist-state.c file: * Cleaning up: * ------------ * * It can be seen that no explicit deallocation is performed on the * persistent file, in effect it could grow indefinitely. There's a * simple cleanup procedure though: * * - on every startup, the persist file is rewritten, entries with an * in_use bit set are copied to the new one, with the in_use bit cleared * - whenever syslog-ng looks up (e.g. uses) an entry, its in_use bit is set again * * This way unused entries in the persist file are reaped when * syslog-ng restarts. *
stats are only in memory, and are not cleaned up, unless syslog-ng is restarted.
In theory those cleanups could happen via syslog-ng-ctl or at reloads, but they don't today.
On Mon, Nov 5, 2018 at 11:19 PM Nik Ambrosch <nik@ambrosch.com> wrote:
is there a way to tell syslog-ng to stop tracking/following a file and flush from syslog-ng stats? i've noticed that when dropping a large number of files into my temporary directory it's very easy to exceed max_files and i'm not sure how long it takes to be removed from syslog-ng.persist automatically (is it ever removed automatically?)
when this is done i'll be fetching and deleting about five new log files per minute.
On Mon, Oct 15, 2018 at 12:36 PM Nik Ambrosch <nik@ambrosch.com> wrote:
It seems most elegant to have syslog-ng take care of the file once it's done consuming, I don't think it would be against the role of the software to do so either.
I wrote a script that seems to do the job if run periodically via cron, my biggest issue I have is a worst-case if there is an non-parsed file with a missing stamp, which would happen on a daemon reload. I'm conflicted if the correct action is to remove the file or force a reload of the file.
#!/usr/bin/env bash
path='/data/tmp'
# examine every file in directory for file in $(find $path -type f); do # how many lines is in this file lines=$(wc -l ${file} | awk '{print $1}')
# output of syslog-ng-ctl ctlout=$(syslog-ng-ctl query get src.file.s_cf_file*${file}*)
# how many lines syslog-ng has parsed parsed=$(echo "${ctlout}" | grep '.processed=' | awk -F '=' '{print $2}')
# when syslog-ng last consumed the file stamp=$(echo "${ctlout}" | grep '.stamp=' | awk -F '=' '{print $2}')
# debug echo "file ${file} parsed ${parsed} of ${lines} lines"
# if file was parsed before a restart if [[ "${parsed}" = "0" && "$stamp" = "0" ]]; then echo "file ${file} processed before syslog-ng restart, removing" #rm -f "${file} fi
# if all lines in file were parsed if [[ ${parsed} -eq ${lines} ]]; then echo "file ${file} processed, removing" #rm -f "${file} fi done
On Mon, Oct 15, 2018 at 12:51 AM, Scheidler, Balázs < balazs.scheidler@oneidentity.com> wrote:
It would be possible to add an option to execute an external script when eof is reached. If i remember correctly the driver level has this information in the form of a notification today. So it's only about adding the option and calling system() on it.
On the other hand, syslog-ng keeps statistics on every file it follows, so the alternative is to poll syslog-ng-ctl stats and see if the counters of the file is non-zero and delete it only in that case.
On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik@ambrosch.com> wrote:
Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.
I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.
That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.
On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.
On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy@oneidentity.com> wrote:
> Hi Nik, > > Syslog-ng should not be designed to delete files when it reaches > EOF, it rather monitors the file for new lines if so. > This would be a bit destructive behaviour even if it would be a > feature with a control flag: > source s_file_clearup { > wildcard-file ( > base-dir("/tmp/") > filename-pattern("*") > remove-on-EOF(yes) > ); > }; > But if we are looking at from your point of view, it could be > enhanced to have one-time files, or drop-off files. > It could be an enhancement. > > With the current behaviour of syslog-ng quick ideas to solve this > use case (if workaround needed): > - syslog-ng closes a file after the reading is idle for time_reap > seconds. This could be monitored externally and remove the given file. > Example message "Destination timed out, reaping; > template='input-logs', filename='input-logs" > I think there is no EOF warning for files, as syslog-ng simply waits > for new lines (as said above). > > Regards, > Gabor > > > On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik@ambrosch.com> > wrote: > >> Looking to create a drop-off directory that syslog-ng handles >> instead of needing to execute in a separate script.. flow would go >> something like this: >> >> 1) mv file.log /syslog-tmp/ >> 2) syslog-ng reads /syslog-tmp/file.log >> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming >> >> Sounds simple but I can't seem to figure out a good way to do >> this. The other option is to read file with a script, send out with logger >> (or whatever), and hope that syslog-ng is running & healthy. >> >> Thanks. >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq