Gerg On Mon, Aug 27, 2012 at 2:41 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hi!
sagar naravane <sagar.naravane@gmail.com> writes:
My setup is like this APACHE EDGE SERVER ---> RELAY ---> SYSLOG-NG CENTRAL SERVER.
Now what I've learned so far (i am still a newbie) is the log collection happens almost real time. However I want to build a setup where if any of the component above goes down the logs in between that particular duration is not lost.
There are a couple of ways to remedy the situation, with different drawbacks. In the end, you'd want a system where everything gets persisted to disk, until the end point acknowledges the receipt of the message.
This is a fairly complex problem, and I'm not entirely sure it would be doable with syslog-ng OSE alone. syslog-ng PE has a few features that make this easier (disk buffers & RLTP) though.
However, you *can* minimize the amount of logs lost, by upping the log_fifo_size() setting on both the relay and the syslog-ng central. This setting tells syslog-ng how many messages it should hold in its internal queues: if you set it high enough so that ~2 hours of logs would fit in, that should be enough, provided the syslog-ng instance that is holding these messages in its queue does not go down.
That is, lets say that on the apache edge server and the relay, you have a syslog-ng configured with log_fifo_size(100000), that will hold 100k messages, each about log_msg_size() (default 2k, I think) in size. If the relay goes down or becomes inaccessible, the edge will be able to buffer 100k messages. If the relay comes back before the queue gets filled up, these messages will be sent out, and no message will be lost. A similar thing happens when the central becomes inaccessible.
You need to keep in mind that when the relay comes back up, the internal queues will need to be emptied, and new messages may be piling up too, so the queue size should be high enough to not fill up during this, either.
The downside of this solution is that this requires quite a lot of memory, and the queue is not persisted to disk, so if any of the syslog-ng instances that is holding on to messages crash, or get restarted without being able to empty its queue, those messages will be lost.
I wrote a piece about this kind of issues a few weeks ago, I encourage you to read it:
http://asylum.madhouse-project.org/blog/2012/07/20/syslog-ng-and-disk-buffer...
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Regards, Sagar Naravane