[syslog-ng] Syslog-ng as basic realtime reliable logfile replicator. Possible?

[Fekete, Róbert]

Hi,

Just a few quick notes:
 * recent version of syslog-ng OSE also support disk-based buffering to
avoid network and server outages. syslog-ng also has a commercial Premium
Edition that can acknowledge the receiving of messages, and resend messages
that got lost.
 * syslog-ng can handle very long messages, so long lines shouldn't be a
problem (adjust log-message-size() if needed). If a message is longer than
that, then it will be truncated.
 * AFAIK you cannot find inodes, but you can transfer the FILE_NAME macro
that includes the path+filename
 * AFAIK syslog-ng holds the source files until it is reading from them, so
file truncation should not be a problem, but I'm not entirely sure about
that

Regards,

Robert


On Tue, Jan 30, 2018 at 2:26 AM, Scot <scotrn at gmail.com> wrote:

>
> - by default, there is nothing you can make with syslog-ng alone that will
> not lose data during a network or endpoint outage.
> I use rsyslog on clients and relays with TCP disk buffering including
> relays.  Properly measured you should know when you are buffering.
>
> - transporting metadata can tell you which file the data is from, but not
> where in the file it's from, so you can't really tell if you have duplicate
> data, or missed data. (The inode number might be handy too)
> - behaviour around input file truncation is fuzzy. That a truncation has
> occured might be useful metadata to send (if you're looking for people
> fiddling logs).
>
> Any mature log reader should handle those use cases, if you have no
> control over the rotation is it possible to load the data after rotation?
> Logrotated has pre and post rotation functions.
>
> - It doesn't seem to be able to encode binary/NULs in the logs, so it
> cannot relay data from 'untrusted' application logs?
> - Not sure what it does with very long lines. Loses data?
>   Have not seen those cases.
>
> Hope it help a little.
> Scot
>
> On Mon, Jan 29, 2018 at 9:34 AM, Declan White <declanw at is.bbc.co.uk>
> wrote:
>
>> Hullo.
>>
>> I'm trying to fit syslog-ng around a basic problem and looking for tips.
>>
>> I have log files growing on one machine that I want to follow and
>> reliably replicate to a central machine, so it's effectively a basic 'tail
>> -f' job.
>> It seems simple, but as I try and close out the possible error conditions
>> it's getting hairier and hairier.
>>
>> e.g.
>> - by default, there is nothing you can make with syslog-ng alone that
>> will not lose data during a network or endpoint outage.
>> - transporting metadata can tell you which file the data is from, but not
>> where in the file it's from, so you can't really tell if you have duplicate
>> data, or missed data. (The inode number might be handy too)
>> - behaviour around input file truncation is fuzzy. That a truncation has
>> occured might be useful metadata to send (if you're looking for people
>> fiddling logs).
>> - It doesn't seem to be able to encode binary/NULs in the logs, so it
>> cannot relay data from 'untrusted' application logs?
>> - Not sure what it does with very long lines. Loses data?
>>
>> I'm not necessarily looking to get syslog-ng to recreate the file
>> exactly, just to send enough information to allow something else to work
>> out the full order of events.
>> Googling around to see how others solve this problem, I see people doing
>> infinite rsync loops, or installing large Java beasties, or paying someone
>> else to make it all go away.
>>
>> I tried using rsyslog, but it melted down into a screaming puddle of
>> nondeterministic threading.
>>
>> Is what I'm attempting really as hard as it seems?
>>
>> - D
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180130/2f6a3e56/attachment-0001.html>