<p>Unfortunately the product we use only supports sending the syslog over UDP. I will try the latest version and the option you mentioned.</p>
<p>Shawn Cannon</p>
<div class="gmail_quote">On Mar 10, 2011 3:21 AM, "Zoltán Pallagi" <<a href="mailto:pzolee@balabit.hu">pzolee@balabit.hu</a>> wrote:<br type="attribution">> Hi,<br>> <br>> First of all, it seems that syslog-ng receives logs faster than your <br>
> mssql server can process them that's the reason for dropping.<br>> <br>> However, you are using udp source for receiving logs, it's not the best <br>> solution if you don't want to lose logs (UDP is not a lossless protocol <br>
> and perhaps the kernel will also drop the unprocessed messages).<br>> You should use TCP instead of UDP.<br>> <br>> If you use TCP, you can use flags(flow-control) in your server <br>> configuration. If the senders are also syslog-ng, you can use it on <br>
> their configurations, too.<br>> flow-control will slow down (or block) receiving logs if syslog-ng <br>> cannot process (write out, forward and so on) the messages in time. It <br>> can prevent losing logs.<br>
> <br>> For example:<br>> log {<br>> source(s_gms);<br>> flags(flow-control);<br>> destination(d_mssql);<br>> };<br>> <br>> For more details about flow-control: <br>> <a href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html</a><br>
> <br>> On 2011-03-10 06:28, Shawn Cannon wrote:<br>>> I really need help on this. I am collecting tons of SYSLOG data from <br>>> over 500 firewalls. I have these syslogs going to a Barracuda Load <br>
>> Balancer first which will then send messages to 2 syslog-ng open <br>>> source servers. Here is how I have my config file setup:<br>>><br>>> @version: 3.0<br>>> #Default configuration file for syslog-ng.<br>
>> #<br>>> # For a description of syslog-ng configuration file directives, please <br>>> read<br>>> # the syslog-ng Administrator's guide at:<br>>> #<br>>> # <br>>> <a href="http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html">http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html</a><br>
>> #<br>>><br>>> options { use_dns(no);<br>>> };<br>>><br>>> ######<br>>> # sources<br>>> source s_gms {<br>>> # message generated by Syslog-NG<br>>> # internal();<br>
>> # standard Linux log source (this is the default place for the syslog()<br>>> # function to send logs to)<br>>> # unix-stream("/dev/log");<br>>> # messages from the kernel<br>>> #file("/proc/kmsg" program_override("kernel"));<br>
>> # messages destined to udp514<br>>> udp(ip(0.0.0.0) port(514));<br>>> };<br>>> source s_syslogng {<br>>> internal();<br>>> };<br>>><br>>> ######<br>>> # destinations<br>
>> destination d_messages { file("/var/log/messages"); };<br>>> destination d_mssql {<br>>> sql(type(mssql) host("<a href="http://rawsql.abcdefg.net">rawsql.abcdefg.net</a> <<a href="http://rawsql.abcdefg.net">http://rawsql.abcdefg.net</a>>") <br>
>> port("1785")<br>>> username("username") password("password") database("Syslog")<br>>> table("syslogng")columns("datetime varchar(16)", "host varchar(32)",<br>
>> "program varchar(32)", "pid varchar(8)", "message varchar(4096)")<br>>> values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")<br>
>> indexes("datetime", "host", "program", "pid"));<br>>> };<br>>><br>>><br>>><br>>> log {<br>>> source(s_gms);<br>>> destination(d_mssql);<br>
>> };<br>>><br>>> log {<br>>> source(s_syslogng);<br>>> destination(d_messages);<br>>> };<br>>><br>>><br>>> When messages are coming in, I am showing the following in the local <br>
>> syslog-ng messages:<br>>><br>>> Log statistics; processed='source(s_gms)=2155636', <br>>> dropped='dst.sql(d_mssql#0,freetds,<a href="http://rawsql.abcdefg.net">rawsql.abcdefg.net</a> <br>
>> <<a href="http://rawsql.abcdefg.net">http://rawsql.abcdefg.net</a>>,1785,Syslog)=1717472', <br>>> stored='dst.sql(d_mssql#0,freetds,<a href="http://rawsql.abcdefg.net">rawsql.abcdefg.net</a> <br>
>> <<a href="http://rawsql.abcdefg.net">http://rawsql.abcdefg.net</a>>,1785,Syslog)=0',<br>>><br>>> I assume this is telling me that I am dropping the majority of my <br>>> messages instead of them getting inserted into my MS SQL database? <br>
>> The MS SQL Database runs on a very beefy server with plenty of <br>>> memory. I am trying to determine why this is being dropped. Please <br>>> help as I am huge newbie when it comes to syslog-ng.<br>
>><br>>> Thanks!<br>>><br>>><br>>> ______________________________________________________________________________<br>>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
>> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>>> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a><br>
>><br>> <br></div>