[syslog-ng] Syslog-ng as basic realtime reliable logfile replicator. Possible?

[Jim Hendrick]

Hmmm - a few questions:
- where are the files coming from that you want to follow?
  Is there another syslog daemon running? Or are you capturing application
logs that cannot use kernel logging?

Syslog-ng can easily use files as sources if that is all you have.
If you can use syslog-ng as the system log daemon - you can easily write
local files *and* forward to a central logger.

It's pretty easy to have the central syslog server that receives the logs
separate them by sending server - I have used HOST_FROM pretty often since
it doesn't need name resolution (better for performance) and it will deal
with non RFC logs fairly well.

As far as metadata - I typically put some of this in the filename - like
date, host_from, facility, severity, etc.

Things like file parsing, etc. can usually be dealt with using appropriate
mix of flags and parsing/rewrite rules (if necessary).

Does this help?

Jim


On Mon, Jan 29, 2018 at 9:34 AM, Declan White <declanw at is.bbc.co.uk> wrote:

> Hullo.
>
> I'm trying to fit syslog-ng around a basic problem and looking for tips.
>
> I have log files growing on one machine that I want to follow and reliably
> replicate to a central machine, so it's effectively a basic 'tail -f' job.
> It seems simple, but as I try and close out the possible error conditions
> it's getting hairier and hairier.
>
> e.g.
> - by default, there is nothing you can make with syslog-ng alone that will
> not lose data during a network or endpoint outage.
> - transporting metadata can tell you which file the data is from, but not
> where in the file it's from, so you can't really tell if you have duplicate
> data, or missed data. (The inode number might be handy too)
> - behaviour around input file truncation is fuzzy. That a truncation has
> occured might be useful metadata to send (if you're looking for people
> fiddling logs).
> - It doesn't seem to be able to encode binary/NULs in the logs, so it
> cannot relay data from 'untrusted' application logs?
> - Not sure what it does with very long lines. Loses data?
>
> I'm not necessarily looking to get syslog-ng to recreate the file exactly,
> just to send enough information to allow something else to work out the
> full order of events.
> Googling around to see how others solve this problem, I see people doing
> infinite rsync loops, or installing large Java beasties, or paying someone
> else to make it all go away.
>
> I tried using rsyslog, but it melted down into a screaming puddle of
> nondeterministic threading.
>
> Is what I'm attempting really as hard as it seems?
>
> - D
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180129/9b9b8332/attachment.html>