[syslog-ng] Dropped messages to MSSQL?
Balazs Scheidler
bazsi at balabit.hu
Thu Mar 10 19:46:57 CET 2011
On Thu, 2011-03-10 at 13:33 -0500, Shawn Cannon wrote:
> I have not enabled explicit commits. That requires the latest version
> right? Also, where do I enable this in the config file?
Well, depends what you call latest :) I'm working on releasing 3.3alpha2
right now.
But yes, it does require syslog-ng 3.2. But we do have a binary of that
for Linux/amd64 on our website, and a number of - recent - distributions
carry that.
I'd recommend you trying the binary / rolling out a box with a recent
enough distribution and evaluate if it's enough for your performance
requirements. Alternatively request a PE evaluation for your platform,
which almost certainly supports your production environment (it supports
40 different UNIX/Linux versions x CPU combinations).
If it is, then you can have a plan going forward:
* if that way the performance problem is resolved, you only need to
solve how you get the latest version
* if it's not, then you have to work out an alternative solution, like
BCP that Martin has mentioned.
You can use explicit-commits this way:
sql(...various-sql-options... flags(explicit-commits) flush_lines(100));
e.g. you need to tell how much messages you want to group into the same
transaction with flush_lines, and explicit-commits tells syslog-ng to
use explicit BEGIN/COMMIT TRANSACTION commands.
>
> On Thu, Mar 10, 2011 at 1:26 PM, Balazs Scheidler <bazsi at balabit.hu>
> wrote:
> On Thu, 2011-03-10 at 11:23 -0500, Shawn Cannon wrote:
> > Thanks for all the info. The current method that our
> firewall
> > management program uses to log messages into the current
> database is
> > by multiple open connections to the database. syslog-ng is
> making one
> > connection and trying to force everything down that one
> connection.
> > So, my question is this: can syslog-ng be configured to
> make multiple
> > connections to the SQL database to insert the data? Just so
> you have
> > a comparison, our current product (which changes in the new
> version
> > and why we need a different syslog product) has182 open
> connections
> > open and that is from 8 agents. It stays up to speen by
> doing that.
> > Thanks....
> >
>
>
> I somehow doubt that injecting messages via multiple
> connections would
> help the message rate. Did you enable explicit-commits?
>
> An even more high performance solution is to use batched
> inserts that
> syslog-ng currently doesn't support with its sql()
> destination. (e.g.
> LOAD FROM FILE and friends).
>
>
>
> > On Thu, Mar 10, 2011 at 11:06 AM, Martin Holste
> <mcholste at gmail.com>
> > wrote:
> > Feel free to contradict, but in my experience, if
> you have
> > more than
> > around 2k messages/second sustained, logging to any
> database
> > directly
> > puts you at very high risk of message drops. Flow
> control and
> > other
> > burst control mechanisms will not help if you have
> an
> > unsustainable
> > message rate.
> >
> >
> > On Thu, Mar 10, 2011 at 9:33 AM, John Kristoff
> <jtk at cymru.com>
> > wrote:
> > > On Thu, 10 Mar 2011 09:21:56 +0100
> > > Zoltán Pallagi <pzolee at balabit.hu> wrote:
> > >
> > >> If you use TCP, you can use flags(flow-control)
> in your
> > server
> > >> configuration. If the senders are also syslog-ng,
> you can
> > use it on
> > >> their configurations, too.
> > >> flow-control will slow down (or block) receiving
> logs if
> > syslog-ng
> > >> cannot process (write out, forward and so on) the
> messages
> > in time.
> > >> It can prevent losing logs.
> > >
> > > The one caveat with this approach seems to be that
> if you
> > have multiple
> > > destinations, then all destinations will block
> until the one
> > stalled
> > > destination is free. So for instance if the SQL
> destination
> > is too
> > > slow, and you're also logging to a file, using
> flow-control
> > may cause
> > > the file-based log to lose messages as well.
> > >
> > > John
> > >
> >
> ______________________________________________________________________________
> > > Member info:
> > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > Documentation:
> >
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > > FAQ: http://www.campin.net/syslog-ng/faq.html
> > >
> > >
> >
> ______________________________________________________________________________
> > Member info:
> > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> >
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> >
> >
> >
> >
> ______________________________________________________________________________
> > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> >
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
--
Bazsi
More information about the syslog-ng
mailing list