<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi,<br>
<br>
First of all, it seems that syslog-ng receives logs faster than your
mssql server can process them that's the reason for dropping.<br>
<br>
However, you are using udp source for receiving logs, it's not the
best solution if you don't want to lose logs (UDP is not a lossless
protocol and perhaps the kernel will also drop the unprocessed
messages). <br>
You should use TCP instead of UDP.<br>
<br>
If you use TCP, you can use flags(flow-control) in your server
configuration. If the senders are also syslog-ng, you can use it on
their configurations, too.<br>
flow-control will slow down (or block) receiving logs if syslog-ng
cannot process (write out, forward and so on) the messages in time.
It can prevent losing logs.<br>
<br>
For example:<br>
log {<br>
source(s_gms);<br>
flags(flow-control);<br>
destination(d_mssql);<br>
};<br>
<br>
For more details about flow-control:
<a class="moz-txt-link-freetext" href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html</a><br>
<br>
On 2011-03-10 06:28, Shawn Cannon wrote:
<blockquote
cite="mid:AANLkTikOekAmhexy4Uznww8g4amezraOTX-93f_Z=mRe@mail.gmail.com"
type="cite">I really need help on this. I am collecting tons of
SYSLOG data from over 500 firewalls. I have these syslogs going
to a Barracuda Load Balancer first which will then send messages
to 2 syslog-ng open source servers. Here is how I have my config
file setup:<br>
<br>
@version: 3.0<br>
#Default configuration file for syslog-ng.<br>
#<br>
# For a description of syslog-ng configuration file directives,
please read<br>
# the syslog-ng Administrator's guide at:<br>
#<br>
# <a moz-do-not-send="true"
href="http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html">http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html</a><br>
#<br>
<br>
options { use_dns(no);<br>
};<br>
<br>
######<br>
# sources<br>
source s_gms {<br>
# message generated by Syslog-NG<br>
# internal();<br>
# standard Linux log source (this is the default place for the
syslog()<br>
# function to send logs to)<br>
# unix-stream("/dev/log");<br>
# messages from the kernel<br>
#file("/proc/kmsg" program_override("kernel"));<br>
# messages destined to udp514<br>
udp(ip(0.0.0.0) port(514));<br>
};<br>
source s_syslogng {<br>
internal();<br>
};<br>
<br>
######<br>
# destinations<br>
destination d_messages { file("/var/log/messages"); };<br>
destination d_mssql {<br>
sql(type(mssql) host("<a moz-do-not-send="true"
href="http://rawsql.abcdefg.net">rawsql.abcdefg.net</a>")
port("1785")<br>
username("username") password("password") database("Syslog")<br>
table("syslogng")columns("datetime varchar(16)", "host
varchar(32)",<br>
"program varchar(32)", "pid varchar(8)", "message varchar(4096)")<br>
values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")<br>
indexes("datetime", "host", "program", "pid"));<br>
};<br>
<br>
<br>
<br>
log {<br>
source(s_gms);<br>
destination(d_mssql);<br>
};<br>
<br>
log {<br>
source(s_syslogng);<br>
destination(d_messages);<br>
};<br>
<br>
<br>
When messages are coming in, I am showing the following in the
local syslog-ng messages:<br>
<br>
Log statistics; processed='source(s_gms)=2155636',
dropped='dst.sql(d_mssql#0,freetds,<a moz-do-not-send="true"
href="http://rawsql.abcdefg.net">rawsql.abcdefg.net</a>,1785,Syslog)=1717472',
stored='dst.sql(d_mssql#0,freetds,<a moz-do-not-send="true"
href="http://rawsql.abcdefg.net">rawsql.abcdefg.net</a>,1785,Syslog)=0',
<br>
<br>
I assume this is telling me that I am dropping the majority of my
messages instead of them getting inserted into my MS SQL
database? The MS SQL Database runs on a very beefy server with
plenty of memory. I am trying to determine why this is being
dropped. Please help as I am huge newbie when it comes to
syslog-ng.<br>
<br>
Thanks!<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
<br>
</body>
</html>