[syslog-ng] Dropped messages to MSSQL?

[Balazs Scheidler]

On Thu, 2011-03-10 at 11:23 -0500, Shawn Cannon wrote:
> Thanks for all the info.  The current method that our firewall
> management program uses to log messages into the current database is
> by multiple open connections to the database.  syslog-ng is making one
> connection and trying to force everything down that one connection.
> So, my question is this:  can syslog-ng be configured to make multiple
> connections to the SQL database to insert the data?  Just so you have
> a comparison, our current product (which changes in the new version
> and why we need a different syslog product) has182 open connections
> open and that is from 8 agents.  It stays up to speen by doing that.
> Thanks....
> 

I somehow doubt that injecting messages via multiple connections would
help the message rate. Did you enable explicit-commits?

An even more high performance solution is to use batched inserts that
syslog-ng currently doesn't support with its sql() destination. (e.g.
LOAD FROM FILE and friends).


> On Thu, Mar 10, 2011 at 11:06 AM, Martin Holste <mcholste at gmail.com>
> wrote:
>         Feel free to contradict, but in my experience, if you have
>         more than
>         around 2k messages/second sustained, logging to any database
>         directly
>         puts you at very high risk of message drops.  Flow control and
>         other
>         burst control mechanisms will not help if you have an
>         unsustainable
>         message rate.
>         
>         
>         On Thu, Mar 10, 2011 at 9:33 AM, John Kristoff <jtk at cymru.com>
>         wrote:
>         > On Thu, 10 Mar 2011 09:21:56 +0100
>         > Zoltán Pallagi <pzolee at balabit.hu> wrote:
>         >
>         >> If you use TCP, you can use flags(flow-control) in your
>         server
>         >> configuration. If the senders are also syslog-ng, you can
>         use it on
>         >> their configurations, too.
>         >> flow-control will slow down (or block) receiving logs if
>         syslog-ng
>         >> cannot process (write out, forward and so on) the messages
>         in time.
>         >> It can prevent losing logs.
>         >
>         > The one caveat with this approach seems to be that if you
>         have multiple
>         > destinations, then all destinations will block until the one
>         stalled
>         > destination is free. So for instance if the SQL destination
>         is too
>         > slow, and you're also logging to a file, using flow-control
>         may cause
>         > the file-based log to lose messages as well.
>         >
>         > John
>         >
>         ______________________________________________________________________________
>         > Member info:
>         https://lists.balabit.hu/mailman/listinfo/syslog-ng
>         > Documentation:
>         http://www.balabit.com/support/documentation/?product=syslog-ng
>         > FAQ: http://www.campin.net/syslog-ng/faq.html
>         >
>         >
>         ______________________________________________________________________________
>         Member info:
>         https://lists.balabit.hu/mailman/listinfo/syslog-ng
>         Documentation:
>         http://www.balabit.com/support/documentation/?product=syslog-ng
>         FAQ: http://www.campin.net/syslog-ng/faq.html
>         
>         
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 

-- 
Bazsi