<div dir="ltr"><div>Thanks for the feedback. The files contain predictable json data, new files arrive every 1-3 minutes (haven't decided yet). There are no start and end markers.</div><div><br></div><div>I'm wary of using cron to delete old files because if syslog-ng isn't able to consume the file (crashed, user error, upgrading package, etc) the non-consumed file will be deleted and contents will be lost.</div><div><br></div><div>That same worst case applies to a separate script - if it provides the messages via syslog (instead of copying a file) but syslog-ng is unhealthy, then my messages are lost unless I build a buffer into the script and that starts to get complex.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Oct 13, 2018 at 2:40 PM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">If there's a specific time for which a dropped file is specific to, then just remove the file after a grace period with a simple cron job.</div><br><div class="gmail_quote"><div dir="ltr">On Sat, Oct 13, 2018, 14:01 Nagy, Gábor <<a href="mailto:gabor.nagy@oneidentity.com" target="_blank">gabor.nagy@oneidentity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Nik,<div><br></div><div>Syslog-ng should not be designed to delete files when it reaches EOF, it rather monitors the file for new lines if so.</div><div>This would be a bit destructive behaviour even if it would be a feature with a control flag:<br></div><div>source s_file_clearup {<br></div><div> wildcard-file (</div><div> base-dir("/tmp/")</div><div> filename-pattern("*")</div><div> remove-on-EOF(yes)</div><div> );</div><div>};</div><div>But if we are looking at from your point of view, it could be enhanced to have one-time files, or drop-off files.</div><div>It could be an enhancement.</div><div><br></div><div>With the current behaviour of syslog-ng quick ideas to solve this use case (if workaround needed):</div><div>- syslog-ng closes a file after the reading is idle for time_reap seconds. This could be monitored externally and remove the given file.</div><div>Example message "Destination timed out, reaping; template='input-logs', filename='input-logs"</div><div>I think there is no EOF warning for files, as syslog-ng simply waits for new lines (as said above).</div><div><br></div><div>Regards,</div><div>Gabor</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 12, 2018 at 5:55 PM Nik Ambrosch <<a href="mailto:nik@ambrosch.com" rel="noreferrer" target="_blank">nik@ambrosch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this:</div><div><br></div><div>1) mv file.log /syslog-tmp/</div><div>2) syslog-ng reads /syslog-tmp/file.log<br></div><div>3) syslog-ng deletes /syslog-tmp/file.log when done consuming<br></div><div><br></div><div>Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy.</div><div><br></div><div>Thanks.<br></div></div>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
</blockquote></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>