[syslog-ng] log drop-and-delete directory

[Nik Ambrosch]

Thanks for the feedback.  The files contain predictable json data, new
files arrive every 1-3 minutes (haven't decided yet).  There are no start
and end markers.

I'm wary of using cron to delete old files because if syslog-ng isn't able
to consume the file (crashed, user error, upgrading package, etc) the
non-consumed file will be deleted and contents will be lost.

That same worst case applies to a separate script - if it provides the
messages via syslog (instead of copying a file) but syslog-ng is unhealthy,
then my messages are lost unless I build a buffer into the script and that
starts to get complex.






On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77 at gmail.com> wrote:

> If there's a specific time for which a dropped file is specific to, then
> just remove the file after a grace period with a simple cron job.
>
> On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy at oneidentity.com>
> wrote:
>
>> Hi Nik,
>>
>> Syslog-ng should not be designed to delete files when it reaches EOF, it
>> rather monitors the file for new lines if so.
>> This would be a bit destructive behaviour even if it would be a feature
>> with a control flag:
>> source s_file_clearup {
>>   wildcard-file (
>>     base-dir("/tmp/")
>>     filename-pattern("*")
>>     remove-on-EOF(yes)
>>   );
>> };
>> But if we are looking at from your point of view, it could be enhanced to
>> have one-time files, or drop-off files.
>> It could be an enhancement.
>>
>> With the current behaviour of syslog-ng quick ideas to solve this use
>> case (if workaround needed):
>> - syslog-ng closes a file after the reading is idle for time_reap
>> seconds. This could be monitored externally and remove the given file.
>> Example message "Destination timed out, reaping; template='input-logs',
>> filename='input-logs"
>> I think there is no EOF warning for files, as syslog-ng simply waits for
>> new lines (as said above).
>>
>> Regards,
>> Gabor
>>
>>
>> On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>>
>>> Looking to create a drop-off directory that syslog-ng handles instead of
>>> needing to execute in a separate script.. flow would go something like this:
>>>
>>> 1) mv file.log /syslog-tmp/
>>> 2) syslog-ng reads /syslog-tmp/file.log
>>> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
>>>
>>> Sounds simple but I can't seem to figure out a good way to do this.  The
>>> other option is to read file with a script, send out with logger (or
>>> whatever), and hope that syslog-ng is running & healthy.
>>>
>>> Thanks.
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?
>>> product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181013/a75d6099/attachment-0001.html>