On Thu, 2011-03-10 at 11:23 -0500, Shawn Cannon wrote:
Thanks for all the info. The current method that our firewall management program uses to log messages into the current database is by multiple open connections to the database. syslog-ng is making one connection and trying to force everything down that one connection. So, my question is this: can syslog-ng be configured to make multiple connections to the SQL database to insert the data? Just so you have a comparison, our current product (which changes in the new version and why we need a different syslog product) has182 open connections open and that is from 8 agents. It stays up to speen by doing that. Thanks....
I somehow doubt that injecting messages via multiple connections would help the message rate. Did you enable explicit-commits? An even more high performance solution is to use batched inserts that syslog-ng currently doesn't support with its sql() destination. (e.g. LOAD FROM FILE and friends).
On Thu, Mar 10, 2011 at 11:06 AM, Martin Holste <mcholste@gmail.com> wrote: Feel free to contradict, but in my experience, if you have more than around 2k messages/second sustained, logging to any database directly puts you at very high risk of message drops. Flow control and other burst control mechanisms will not help if you have an unsustainable message rate.
On Thu, Mar 10, 2011 at 9:33 AM, John Kristoff <jtk@cymru.com> wrote: > On Thu, 10 Mar 2011 09:21:56 +0100 > Zoltán Pallagi <pzolee@balabit.hu> wrote: > >> If you use TCP, you can use flags(flow-control) in your server >> configuration. If the senders are also syslog-ng, you can use it on >> their configurations, too. >> flow-control will slow down (or block) receiving logs if syslog-ng >> cannot process (write out, forward and so on) the messages in time. >> It can prevent losing logs. > > The one caveat with this approach seems to be that if you have multiple > destinations, then all destinations will block until the one stalled > destination is free. So for instance if the SQL destination is too > slow, and you're also logging to a file, using flow-control may cause > the file-based log to lose messages as well. > > John > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi