[syslog-ng] How to prevent loss of logs.

[sagar naravane]

Ooops..i clicked send button by mistake !!

Thanks Gergely.. this is exactly what I was looking for. Let me try and
implement the pointers :)

Sagar

On Mon, Aug 27, 2012 at 2:53 PM, sagar naravane <sagar.naravane at gmail.com>wrote:

> Gerg
>
>
> On Mon, Aug 27, 2012 at 2:41 PM, Gergely Nagy <algernon at balabit.hu> wrote:
>
>> Hi!
>>
>> sagar naravane <sagar.naravane at gmail.com> writes:
>>
>> > My setup is like this APACHE EDGE SERVER ---> RELAY ---> SYSLOG-NG
>> CENTRAL
>> > SERVER.
>> >
>> > Now what I've learned so far (i am still a newbie) is the log collection
>> > happens almost real time. However I want to build a setup where if any
>> of
>> > the component above goes down the logs in between that particular
>> duration
>> > is not lost.
>>
>> There are a couple of ways to remedy the situation, with different
>> drawbacks. In the end, you'd want a system where everything gets
>> persisted to disk, until the end point acknowledges the receipt of the
>> message.
>>
>> This is a fairly complex problem, and I'm not entirely sure it would be
>> doable with syslog-ng OSE alone. syslog-ng PE has a few features that
>> make this easier (disk buffers & RLTP) though.
>>
>> However, you *can* minimize the amount of logs lost, by upping the
>> log_fifo_size() setting on both the relay and the syslog-ng
>> central. This setting tells syslog-ng how many messages it should hold
>> in its internal queues: if you set it high enough so that ~2 hours of
>> logs would fit in, that should be enough, provided the syslog-ng
>> instance that is holding these messages in its queue does not go down.
>>
>> That is, lets say that on the apache edge server and the relay, you have
>> a syslog-ng configured with log_fifo_size(100000), that will hold 100k
>> messages, each about log_msg_size() (default 2k, I think) in size. If
>> the relay goes down or becomes inaccessible, the edge will be able to
>> buffer 100k messages. If the relay comes back before the queue gets
>> filled up, these messages will be sent out, and no message will be
>> lost. A similar thing happens when the central becomes inaccessible.
>>
>> You need to keep in mind that when the relay comes back up, the internal
>> queues will need to be emptied, and new messages may be piling up too,
>> so the queue size should be high enough to not fill up during this,
>> either.
>>
>> The downside of this solution is that this requires quite a lot of
>> memory, and the queue is not persisted to disk, so if any of the
>> syslog-ng instances that is holding on to messages crash, or get
>> restarted without being able to empty its queue, those messages will be
>> lost.
>>
>> I wrote a piece about this kind of issues a few weeks ago, I encourage
>> you to read it:
>>
>> http://asylum.madhouse-project.org/blog/2012/07/20/syslog-ng-and-disk-buffers/
>>
>> --
>> |8]
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
>
> --
> Regards,
>
> Sagar Naravane
>



-- 
Regards,

Sagar Naravane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120827/fd31e8ea/attachment.htm