<div dir="ltr">Hi, <div><br></div><div>Just a few quick notes: </div><div> * recent version of syslog-ng OSE also support disk-based buffering to avoid network and server outages. syslog-ng also has a commercial Premium Edition that can acknowledge the receiving of messages, and resend messages that got lost. </div><div> * syslog-ng can handle very long messages, so long lines shouldn't be a problem (adjust log-message-size() if needed). If a message is longer than that, then it will be truncated.</div><div> * AFAIK you cannot find inodes, but you can transfer the FILE_NAME macro that includes the path+filename</div><div> * AFAIK syslog-ng holds the source files until it is reading from them, so file truncation should not be a problem, but I'm not entirely sure about that</div><div><br></div><div>Regards, </div><div><br></div><div>Robert</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 30, 2018 at 2:26 AM, Scot <span dir="ltr"><<a href="mailto:scotrn@gmail.com" target="_blank">scotrn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- by default, there is nothing you can make with syslog-ng alone that will not lose data during a network or endpoint outage.</span></span><div>I use rsyslog on clients and relays with TCP disk buffering including relays.  Properly measured you should know when you are buffering.  </div><span class=""><div><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- transporting metadata can tell you which file the data is from, but not where in the file it's from, so you can't really tell if you have duplicate data, or missed data. (The inode number might be handy too)</span></div><div><span style="font-size:12.8px">- behaviour around input file truncation is fuzzy. That a truncation has occured might be useful metadata to send (if you're looking for people fiddling logs).</span><br></div></span><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Any mature log reader should handle those use cases, if you have no control over the rotation is it possible to load the data after rotation? Logrotated has pre and post rotation functions. </div></div><div><span class=""><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- It doesn't seem to be able to encode binary/NULs in the logs, so it cannot relay data from 'untrusted' application logs?</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- Not sure what it does with very long lines. Loses data?</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"></span>  Have not seen those cases. </div><div><br></div><div>Hope it help a little. </div><span class="HOEnZb"><font color="#888888"><div>Scot </div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 9:34 AM, Declan White <span dir="ltr"><<a href="mailto:declanw@is.bbc.co.uk" target="_blank">declanw@is.bbc.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hullo.<br>
<br>
I'm trying to fit syslog-ng around a basic problem and looking for tips.<br>
<br>
I have log files growing on one machine that I want to follow and reliably replicate to a central machine, so it's effectively a basic 'tail -f' job.<br>
It seems simple, but as I try and close out the possible error conditions it's getting hairier and hairier.<br>
<br>
e.g.<br>
- by default, there is nothing you can make with syslog-ng alone that will not lose data during a network or endpoint outage.<br>
- transporting metadata can tell you which file the data is from, but not where in the file it's from, so you can't really tell if you have duplicate data, or missed data. (The inode number might be handy too)<br>
- behaviour around input file truncation is fuzzy. That a truncation has occured might be useful metadata to send (if you're looking for people fiddling logs).<br>
- It doesn't seem to be able to encode binary/NULs in the logs, so it cannot relay data from 'untrusted' application logs?<br>
- Not sure what it does with very long lines. Loses data?<br>
<br>
I'm not necessarily looking to get syslog-ng to recreate the file exactly, just to send enough information to allow something else to work out the full order of events.<br>
Googling around to see how others solve this problem, I see people doing infinite rsync loops, or installing large Java beasties, or paying someone else to make it all go away.<br>
<br>
I tried using rsyslog, but it melted down into a screaming puddle of nondeterministic threading.<br>
<br>
Is what I'm attempting really as hard as it seems?<br>
<br>
- D<br>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>