I have not enabled explicit commits. That requires the latest version right? Also, where do I enable this in the config file?<br><br><div class="gmail_quote">On Thu, Mar 10, 2011 at 1:26 PM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">On Thu, 2011-03-10 at 11:23 -0500, Shawn Cannon wrote:<br>
> Thanks for all the info. The current method that our firewall<br>
> management program uses to log messages into the current database is<br>
> by multiple open connections to the database. syslog-ng is making one<br>
> connection and trying to force everything down that one connection.<br>
> So, my question is this: can syslog-ng be configured to make multiple<br>
> connections to the SQL database to insert the data? Just so you have<br>
> a comparison, our current product (which changes in the new version<br>
> and why we need a different syslog product) has182 open connections<br>
> open and that is from 8 agents. It stays up to speen by doing that.<br>
> Thanks....<br>
><br>
<br>
</div>I somehow doubt that injecting messages via multiple connections would<br>
help the message rate. Did you enable explicit-commits?<br>
<br>
An even more high performance solution is to use batched inserts that<br>
syslog-ng currently doesn't support with its sql() destination. (e.g.<br>
LOAD FROM FILE and friends).<br>
<div><div></div><div class="h5"><br>
<br>
> On Thu, Mar 10, 2011 at 11:06 AM, Martin Holste <<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>><br>
> wrote:<br>
> Feel free to contradict, but in my experience, if you have<br>
> more than<br>
> around 2k messages/second sustained, logging to any database<br>
> directly<br>
> puts you at very high risk of message drops. Flow control and<br>
> other<br>
> burst control mechanisms will not help if you have an<br>
> unsustainable<br>
> message rate.<br>
><br>
><br>
> On Thu, Mar 10, 2011 at 9:33 AM, John Kristoff <<a href="mailto:jtk@cymru.com">jtk@cymru.com</a>><br>
> wrote:<br>
> > On Thu, 10 Mar 2011 09:21:56 +0100<br>
> > Zoltán Pallagi <<a href="mailto:pzolee@balabit.hu">pzolee@balabit.hu</a>> wrote:<br>
> ><br>
> >> If you use TCP, you can use flags(flow-control) in your<br>
> server<br>
> >> configuration. If the senders are also syslog-ng, you can<br>
> use it on<br>
> >> their configurations, too.<br>
> >> flow-control will slow down (or block) receiving logs if<br>
> syslog-ng<br>
> >> cannot process (write out, forward and so on) the messages<br>
> in time.<br>
> >> It can prevent losing logs.<br>
> ><br>
> > The one caveat with this approach seems to be that if you<br>
> have multiple<br>
> > destinations, then all destinations will block until the one<br>
> stalled<br>
> > destination is free. So for instance if the SQL destination<br>
> is too<br>
> > slow, and you're also logging to a file, using flow-control<br>
> may cause<br>
> > the file-based log to lose messages as well.<br>
> ><br>
> > John<br>
> ><br>
> ______________________________________________________________________________<br>
> > Member info:<br>
> <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> > Documentation:<br>
> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> > FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
> ><br>
> ><br>
> ______________________________________________________________________________<br>
> Member info:<br>
> <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation:<br>
> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
><br>
><br>
><br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
><br>
<br>
</div></div><font color="#888888">--<br>
Bazsi<br>
</font><div><div></div><div class="h5"><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</div></div></blockquote></div><br>