[syslog-ng] Dropped messages to MSSQL?

[Zoltán Pallagi]

Hi,

First of all, it seems that syslog-ng receives logs faster than your 
mssql server can process them that's the reason for dropping.

However, you are using udp source for receiving logs, it's not the best 
solution if you don't want to lose logs (UDP is not a lossless protocol 
and perhaps the kernel will also drop the unprocessed messages).
You should use TCP instead of UDP.

If you use TCP, you can use flags(flow-control) in your server 
configuration. If the senders are also syslog-ng, you can use it on 
their configurations, too.
flow-control will slow down (or block) receiving logs if syslog-ng 
cannot process (write out, forward and so on) the messages in time. It 
can prevent losing logs.

For example:
log {
source(s_gms);
flags(flow-control);
destination(d_mssql);
};

For more details about flow-control: 
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html

On 2011-03-10 06:28, Shawn Cannon wrote:
> I really need help on this.  I am collecting tons of SYSLOG data from 
> over 500 firewalls.  I have these syslogs going to a Barracuda Load 
> Balancer first which will then send messages to 2 syslog-ng open 
> source servers.  Here is how I have my config file setup:
>
> @version: 3.0
> #Default configuration file for syslog-ng.
> #
> # For a description of syslog-ng configuration file directives, please 
> read
> # the syslog-ng Administrator's guide at:
> #
> # 
> http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
> #
>
> options { use_dns(no);
> };
>
> ######
> # sources
> source s_gms {
> # message generated by Syslog-NG
> # internal();
> # standard Linux log source (this is the default place for the syslog()
> # function to send logs to)
> # unix-stream("/dev/log");
> # messages from the kernel
> #file("/proc/kmsg" program_override("kernel"));
> # messages destined to udp514
> udp(ip(0.0.0.0) port(514));
> };
> source s_syslogng {
> internal();
> };
>
> ######
> # destinations
> destination d_messages { file("/var/log/messages"); };
> destination d_mssql {
> sql(type(mssql) host("rawsql.abcdefg.net <http://rawsql.abcdefg.net>") 
> port("1785")
> username("username") password("password") database("Syslog")
> table("syslogng")columns("datetime varchar(16)", "host varchar(32)",
> "program varchar(32)", "pid varchar(8)", "message varchar(4096)")
> values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
> indexes("datetime", "host", "program", "pid"));
> };
>
>
>
> log {
> source(s_gms);
> destination(d_mssql);
> };
>
> log {
> source(s_syslogng);
> destination(d_messages);
> };
>
>
> When messages are coming in, I am showing the following in the local 
> syslog-ng messages:
>
> Log statistics; processed='source(s_gms)=2155636', 
> dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net 
> <http://rawsql.abcdefg.net>,1785,Syslog)=1717472', 
> stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net 
> <http://rawsql.abcdefg.net>,1785,Syslog)=0',
>
> I assume this is telling me that I am dropping the majority of my 
> messages instead of them getting inserted into my MS SQL database?  
> The MS SQL Database runs on a very beefy server with plenty of 
> memory.  I am trying to determine why this is being dropped.  Please 
> help as I am huge newbie when it comes to syslog-ng.
>
> Thanks!
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110310/dc855957/attachment.htm