On Thu, 2011-03-10 at 13:33 -0500, Shawn Cannon wrote:
I have not enabled explicit commits. That requires the latest version right? Also, where do I enable this in the config file?
Well, depends what you call latest :) I'm working on releasing 3.3alpha2 right now. But yes, it does require syslog-ng 3.2. But we do have a binary of that for Linux/amd64 on our website, and a number of - recent - distributions carry that. I'd recommend you trying the binary / rolling out a box with a recent enough distribution and evaluate if it's enough for your performance requirements. Alternatively request a PE evaluation for your platform, which almost certainly supports your production environment (it supports 40 different UNIX/Linux versions x CPU combinations). If it is, then you can have a plan going forward: * if that way the performance problem is resolved, you only need to solve how you get the latest version * if it's not, then you have to work out an alternative solution, like BCP that Martin has mentioned. You can use explicit-commits this way: sql(...various-sql-options... flags(explicit-commits) flush_lines(100)); e.g. you need to tell how much messages you want to group into the same transaction with flush_lines, and explicit-commits tells syslog-ng to use explicit BEGIN/COMMIT TRANSACTION commands.
On Thu, Mar 10, 2011 at 1:26 PM, Balazs Scheidler <bazsi@balabit.hu> wrote: On Thu, 2011-03-10 at 11:23 -0500, Shawn Cannon wrote: > Thanks for all the info. The current method that our firewall > management program uses to log messages into the current database is > by multiple open connections to the database. syslog-ng is making one > connection and trying to force everything down that one connection. > So, my question is this: can syslog-ng be configured to make multiple > connections to the SQL database to insert the data? Just so you have > a comparison, our current product (which changes in the new version > and why we need a different syslog product) has182 open connections > open and that is from 8 agents. It stays up to speen by doing that. > Thanks.... >
I somehow doubt that injecting messages via multiple connections would help the message rate. Did you enable explicit-commits?
An even more high performance solution is to use batched inserts that syslog-ng currently doesn't support with its sql() destination. (e.g. LOAD FROM FILE and friends).
> On Thu, Mar 10, 2011 at 11:06 AM, Martin Holste <mcholste@gmail.com> > wrote: > Feel free to contradict, but in my experience, if you have > more than > around 2k messages/second sustained, logging to any database > directly > puts you at very high risk of message drops. Flow control and > other > burst control mechanisms will not help if you have an > unsustainable > message rate. > > > On Thu, Mar 10, 2011 at 9:33 AM, John Kristoff <jtk@cymru.com> > wrote: > > On Thu, 10 Mar 2011 09:21:56 +0100 > > Zoltán Pallagi <pzolee@balabit.hu> wrote: > > > >> If you use TCP, you can use flags(flow-control) in your > server > >> configuration. If the senders are also syslog-ng, you can > use it on > >> their configurations, too. > >> flow-control will slow down (or block) receiving logs if > syslog-ng > >> cannot process (write out, forward and so on) the messages > in time. > >> It can prevent losing logs. > > > > The one caveat with this approach seems to be that if you > have multiple > > destinations, then all destinations will block until the one > stalled > > destination is free. So for instance if the SQL destination > is too > > slow, and you're also logging to a file, using flow-control > may cause > > the file-based log to lose messages as well. > > > > John > > > ______________________________________________________________________________ > > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > ______________________________________________________________________________ > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html >
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi