Ooops..i clicked send button by mistake !!<div><br></div><div>Thanks Gergely.. this is exactly what I was looking for. Let me try and implement the pointers :)</div><div><br></div><div>Sagar<br><br><div class="gmail_quote">
On Mon, Aug 27, 2012 at 2:53 PM, sagar naravane <span dir="ltr"><<a href="mailto:sagar.naravane@gmail.com" target="_blank">sagar.naravane@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Gerg<div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Mon, Aug 27, 2012 at 2:41 PM, Gergely Nagy <span dir="ltr"><<a href="mailto:algernon@balabit.hu" target="_blank">algernon@balabit.hu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi!<br>
<div><br>
sagar naravane <<a href="mailto:sagar.naravane@gmail.com" target="_blank">sagar.naravane@gmail.com</a>> writes:<br>
<br>
> My setup is like this APACHE EDGE SERVER ---> RELAY ---> SYSLOG-NG CENTRAL<br>
> SERVER.<br>
><br>
> Now what I've learned so far (i am still a newbie) is the log collection<br>
> happens almost real time. However I want to build a setup where if any of<br>
> the component above goes down the logs in between that particular duration<br>
> is not lost.<br>
<br>
</div>There are a couple of ways to remedy the situation, with different<br>
drawbacks. In the end, you'd want a system where everything gets<br>
persisted to disk, until the end point acknowledges the receipt of the<br>
message.<br>
<br>
This is a fairly complex problem, and I'm not entirely sure it would be<br>
doable with syslog-ng OSE alone. syslog-ng PE has a few features that<br>
make this easier (disk buffers & RLTP) though.<br>
<br>
However, you *can* minimize the amount of logs lost, by upping the<br>
log_fifo_size() setting on both the relay and the syslog-ng<br>
central. This setting tells syslog-ng how many messages it should hold<br>
in its internal queues: if you set it high enough so that ~2 hours of<br>
logs would fit in, that should be enough, provided the syslog-ng<br>
instance that is holding these messages in its queue does not go down.<br>
<br>
That is, lets say that on the apache edge server and the relay, you have<br>
a syslog-ng configured with log_fifo_size(100000), that will hold 100k<br>
messages, each about log_msg_size() (default 2k, I think) in size. If<br>
the relay goes down or becomes inaccessible, the edge will be able to<br>
buffer 100k messages. If the relay comes back before the queue gets<br>
filled up, these messages will be sent out, and no message will be<br>
lost. A similar thing happens when the central becomes inaccessible.<br>
<br>
You need to keep in mind that when the relay comes back up, the internal<br>
queues will need to be emptied, and new messages may be piling up too,<br>
so the queue size should be high enough to not fill up during this,<br>
either.<br>
<br>
The downside of this solution is that this requires quite a lot of<br>
memory, and the queue is not persisted to disk, so if any of the<br>
syslog-ng instances that is holding on to messages crash, or get<br>
restarted without being able to empty its queue, those messages will be<br>
lost.<br>
<br>
I wrote a piece about this kind of issues a few weeks ago, I encourage<br>
you to read it:<br>
<a href="http://asylum.madhouse-project.org/blog/2012/07/20/syslog-ng-and-disk-buffers/" target="_blank">http://asylum.madhouse-project.org/blog/2012/07/20/syslog-ng-and-disk-buffers/</a><br>
<span><font color="#888888"><br>
--<br>
|8]<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>Regards,<br><br>Sagar Naravane<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Regards,<br><br>Sagar Naravane<br>
</div>