[syslog-ng] log drop-and-delete directory

[Scheidler, Balázs]

It would be possible to add an option to execute an external script when
eof is reached.
If i remember correctly the driver level has this information in the form
of a notification today. So it's only about adding the option and calling
system() on it.

On the other hand, syslog-ng keeps statistics on every file it follows, so
the alternative is to poll syslog-ng-ctl stats and see if the counters of
the file is non-zero and delete it only in that case.

On Sun, Oct 14, 2018, 04:16 Nik Ambrosch <nik at ambrosch.com> wrote:

> Thanks for the feedback.  The files contain predictable json data, new
> files arrive every 1-3 minutes (haven't decided yet).  There are no start
> and end markers.
>
> I'm wary of using cron to delete old files because if syslog-ng isn't able
> to consume the file (crashed, user error, upgrading package, etc) the
> non-consumed file will be deleted and contents will be lost.
>
> That same worst case applies to a separate script - if it provides the
> messages via syslog (instead of copying a file) but syslog-ng is unhealthy,
> then my messages are lost unless I build a buffer into the script and that
> starts to get complex.
>
>
>
>
>
>
> On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <bazsi77 at gmail.com>
> wrote:
>
>> If there's a specific time for which a dropped file is specific to, then
>> just remove the file after a grace period with a simple cron job.
>>
>> On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <gabor.nagy at oneidentity.com>
>> wrote:
>>
>>> Hi Nik,
>>>
>>> Syslog-ng should not be designed to delete files when it reaches EOF, it
>>> rather monitors the file for new lines if so.
>>> This would be a bit destructive behaviour even if it would be a feature
>>> with a control flag:
>>> source s_file_clearup {
>>>   wildcard-file (
>>>     base-dir("/tmp/")
>>>     filename-pattern("*")
>>>     remove-on-EOF(yes)
>>>   );
>>> };
>>> But if we are looking at from your point of view, it could be enhanced
>>> to have one-time files, or drop-off files.
>>> It could be an enhancement.
>>>
>>> With the current behaviour of syslog-ng quick ideas to solve this use
>>> case (if workaround needed):
>>> - syslog-ng closes a file after the reading is idle for time_reap
>>> seconds. This could be monitored externally and remove the given file.
>>> Example message "Destination timed out, reaping; template='input-logs',
>>> filename='input-logs"
>>> I think there is no EOF warning for files, as syslog-ng simply waits for
>>> new lines (as said above).
>>>
>>> Regards,
>>> Gabor
>>>
>>>
>>> On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <nik at ambrosch.com> wrote:
>>>
>>>> Looking to create a drop-off directory that syslog-ng handles instead
>>>> of needing to execute in a separate script.. flow would go something like
>>>> this:
>>>>
>>>> 1) mv file.log /syslog-tmp/
>>>> 2) syslog-ng reads /syslog-tmp/file.log
>>>> 3) syslog-ng deletes /syslog-tmp/file.log when done consuming
>>>>
>>>> Sounds simple but I can't seem to figure out a good way to do this.
>>>> The other option is to read file with a script, send out with logger (or
>>>> whatever), and hope that syslog-ng is running & healthy.
>>>>
>>>> Thanks.
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181015/06b87dd6/attachment.html>