[syslog-ng] Dropped messages to MSSQL?

[Shawn Cannon]

I have not enabled explicit commits.  That requires the latest version
right?  Also, where do I enable this in the config file?

On Thu, Mar 10, 2011 at 1:26 PM, Balazs Scheidler <bazsi at balabit.hu> wrote:

> On Thu, 2011-03-10 at 11:23 -0500, Shawn Cannon wrote:
> > Thanks for all the info.  The current method that our firewall
> > management program uses to log messages into the current database is
> > by multiple open connections to the database.  syslog-ng is making one
> > connection and trying to force everything down that one connection.
> > So, my question is this:  can syslog-ng be configured to make multiple
> > connections to the SQL database to insert the data?  Just so you have
> > a comparison, our current product (which changes in the new version
> > and why we need a different syslog product) has182 open connections
> > open and that is from 8 agents.  It stays up to speen by doing that.
> > Thanks....
> >
>
> I somehow doubt that injecting messages via multiple connections would
> help the message rate. Did you enable explicit-commits?
>
> An even more high performance solution is to use batched inserts that
> syslog-ng currently doesn't support with its sql() destination. (e.g.
> LOAD FROM FILE and friends).
>
>
> > On Thu, Mar 10, 2011 at 11:06 AM, Martin Holste <mcholste at gmail.com>
> > wrote:
> >         Feel free to contradict, but in my experience, if you have
> >         more than
> >         around 2k messages/second sustained, logging to any database
> >         directly
> >         puts you at very high risk of message drops.  Flow control and
> >         other
> >         burst control mechanisms will not help if you have an
> >         unsustainable
> >         message rate.
> >
> >
> >         On Thu, Mar 10, 2011 at 9:33 AM, John Kristoff <jtk at cymru.com>
> >         wrote:
> >         > On Thu, 10 Mar 2011 09:21:56 +0100
> >         > Zoltán Pallagi <pzolee at balabit.hu> wrote:
> >         >
> >         >> If you use TCP, you can use flags(flow-control) in your
> >         server
> >         >> configuration. If the senders are also syslog-ng, you can
> >         use it on
> >         >> their configurations, too.
> >         >> flow-control will slow down (or block) receiving logs if
> >         syslog-ng
> >         >> cannot process (write out, forward and so on) the messages
> >         in time.
> >         >> It can prevent losing logs.
> >         >
> >         > The one caveat with this approach seems to be that if you
> >         have multiple
> >         > destinations, then all destinations will block until the one
> >         stalled
> >         > destination is free. So for instance if the SQL destination
> >         is too
> >         > slow, and you're also logging to a file, using flow-control
> >         may cause
> >         > the file-based log to lose messages as well.
> >         >
> >         > John
> >         >
> >
> ______________________________________________________________________________
> >         > Member info:
> >         https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >         > Documentation:
> >         http://www.balabit.com/support/documentation/?product=syslog-ng
> >         > FAQ: http://www.campin.net/syslog-ng/faq.html
> >         >
> >         >
> >
> ______________________________________________________________________________
> >         Member info:
> >         https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >         Documentation:
> >         http://www.balabit.com/support/documentation/?product=syslog-ng
> >         FAQ: http://www.campin.net/syslog-ng/faq.html
> >
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> >
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110310/c034ab95/attachment-0001.htm