Hi,

First of all, it seems that syslog-ng receives logs faster than your mssql server can process them that's the reason for dropping.

However, you are using udp source for receiving logs, it's not the best solution if you don't want to lose logs (UDP is not a lossless protocol and perhaps the kernel will also drop the unprocessed messages).
You should use TCP instead of UDP.

If you use TCP, you can use flags(flow-control) in your server configuration. If the senders are also syslog-ng, you can use it on their configurations, too.
flow-control will slow down (or block) receiving logs if syslog-ng cannot process (write out, forward and so on) the messages in time. It can prevent losing logs.

For example:
log {
source(s_gms);
flags(flow-control);
destination(d_mssql);
};

For more details about flow-control: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html

On 2011-03-10 06:28, Shawn Cannon wrote:

I really need help on this. I am collecting tons of SYSLOG data from over 500 firewalls. I have these syslogs going to a Barracuda Load Balancer first which will then send messages to 2 syslog-ng open source servers. Here is how I have my config file setup:

@version: 3.0
#Default configuration file for syslog-ng.
#
# For a description of syslog-ng configuration file directives, please read
# the syslog-ng Administrator's guide at:
#
# http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
#

options { use_dns(no);
};

######
# sources
source s_gms {
# message generated by Syslog-NG
# internal();
# standard Linux log source (this is the default place for the syslog()
# function to send logs to)
# unix-stream("/dev/log");
# messages from the kernel
#file("/proc/kmsg" program_override("kernel"));
# messages destined to udp514
udp(ip(0.0.0.0) port(514));
};
source s_syslogng {
internal();
};

######
# destinations
destination d_messages { file("/var/log/messages"); };
destination d_mssql {
sql(type(mssql) host("rawsql.abcdefg.net") port("1785")
username("username") password("password") database("Syslog")
table("syslogng")columns("datetime varchar(16)", "host varchar(32)",
"program varchar(32)", "pid varchar(8)", "message varchar(4096)")
values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
indexes("datetime", "host", "program", "pid"));
};

log {
source(s_gms);
destination(d_mssql);
};

log {
source(s_syslogng);
destination(d_messages);
};

When messages are coming in, I am showing the following in the local syslog-ng messages:

Log statistics; processed='source(s_gms)=2155636', dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=1717472', stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=0',

I assume this is telling me that I am dropping the majority of my messages instead of them getting inserted into my MS SQL database? The MS SQL Database runs on a very beefy server with plenty of memory. I am trying to determine why this is being dropped. Please help as I am huge newbie when it comes to syslog-ng.

Thanks!
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html