[syslog-ng] How to prevent loss of logs.

[Gergely Nagy]

Hi!

sagar naravane <sagar.naravane at gmail.com> writes:

> My setup is like this APACHE EDGE SERVER ---> RELAY ---> SYSLOG-NG CENTRAL
> SERVER.
>
> Now what I've learned so far (i am still a newbie) is the log collection
> happens almost real time. However I want to build a setup where if any of
> the component above goes down the logs in between that particular duration
> is not lost.

There are a couple of ways to remedy the situation, with different
drawbacks. In the end, you'd want a system where everything gets
persisted to disk, until the end point acknowledges the receipt of the
message.

This is a fairly complex problem, and I'm not entirely sure it would be
doable with syslog-ng OSE alone. syslog-ng PE has a few features that
make this easier (disk buffers & RLTP) though.

However, you *can* minimize the amount of logs lost, by upping the
log_fifo_size() setting on both the relay and the syslog-ng
central. This setting tells syslog-ng how many messages it should hold
in its internal queues: if you set it high enough so that ~2 hours of
logs would fit in, that should be enough, provided the syslog-ng
instance that is holding these messages in its queue does not go down.

That is, lets say that on the apache edge server and the relay, you have
a syslog-ng configured with log_fifo_size(100000), that will hold 100k
messages, each about log_msg_size() (default 2k, I think) in size. If
the relay goes down or becomes inaccessible, the edge will be able to
buffer 100k messages. If the relay comes back before the queue gets
filled up, these messages will be sent out, and no message will be
lost. A similar thing happens when the central becomes inaccessible.

You need to keep in mind that when the relay comes back up, the internal
queues will need to be emptied, and new messages may be piling up too,
so the queue size should be high enough to not fill up during this,
either.

The downside of this solution is that this requires quite a lot of
memory, and the queue is not persisted to disk, so if any of the
syslog-ng instances that is holding on to messages crash, or get
restarted without being able to empty its queue, those messages will be
lost.

I wrote a piece about this kind of issues a few weeks ago, I encourage
you to read it:
 http://asylum.madhouse-project.org/blog/2012/07/20/syslog-ng-and-disk-buffers/

-- 
|8]