[syslog-ng] How to prevent loss of logs.

[sagar naravane]

Gerg

On Mon, Aug 27, 2012 at 2:41 PM, Gergely Nagy <algernon at balabit.hu> wrote:

> Hi!
>
> sagar naravane <sagar.naravane at gmail.com> writes:
>
> > My setup is like this APACHE EDGE SERVER ---> RELAY ---> SYSLOG-NG
> CENTRAL
> > SERVER.
> >
> > Now what I've learned so far (i am still a newbie) is the log collection
> > happens almost real time. However I want to build a setup where if any of
> > the component above goes down the logs in between that particular
> duration
> > is not lost.
>
> There are a couple of ways to remedy the situation, with different
> drawbacks. In the end, you'd want a system where everything gets
> persisted to disk, until the end point acknowledges the receipt of the
> message.
>
> This is a fairly complex problem, and I'm not entirely sure it would be
> doable with syslog-ng OSE alone. syslog-ng PE has a few features that
> make this easier (disk buffers & RLTP) though.
>
> However, you *can* minimize the amount of logs lost, by upping the
> log_fifo_size() setting on both the relay and the syslog-ng
> central. This setting tells syslog-ng how many messages it should hold
> in its internal queues: if you set it high enough so that ~2 hours of
> logs would fit in, that should be enough, provided the syslog-ng
> instance that is holding these messages in its queue does not go down.
>
> That is, lets say that on the apache edge server and the relay, you have
> a syslog-ng configured with log_fifo_size(100000), that will hold 100k
> messages, each about log_msg_size() (default 2k, I think) in size. If
> the relay goes down or becomes inaccessible, the edge will be able to
> buffer 100k messages. If the relay comes back before the queue gets
> filled up, these messages will be sent out, and no message will be
> lost. A similar thing happens when the central becomes inaccessible.
>
> You need to keep in mind that when the relay comes back up, the internal
> queues will need to be emptied, and new messages may be piling up too,
> so the queue size should be high enough to not fill up during this,
> either.
>
> The downside of this solution is that this requires quite a lot of
> memory, and the queue is not persisted to disk, so if any of the
> syslog-ng instances that is holding on to messages crash, or get
> restarted without being able to empty its queue, those messages will be
> lost.
>
> I wrote a piece about this kind of issues a few weeks ago, I encourage
> you to read it:
>
> http://asylum.madhouse-project.org/blog/2012/07/20/syslog-ng-and-disk-buffers/
>
> --
> |8]
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>


-- 
Regards,

Sagar Naravane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120827/dd9eab7e/attachment.htm