Please see my response on your earlier thread regarding writing to file and using bcp. On Thu, Mar 10, 2011 at 6:17 AM, Shawn Cannon <shawn@shawncannon.com> wrote:
Unfortunately the product we use only supports sending the syslog over UDP. I will try the latest version and the option you mentioned.
Shawn Cannon
On Mar 10, 2011 3:21 AM, "Zoltán Pallagi" <pzolee@balabit.hu> wrote:
Hi,
First of all, it seems that syslog-ng receives logs faster than your mssql server can process them that's the reason for dropping.
However, you are using udp source for receiving logs, it's not the best solution if you don't want to lose logs (UDP is not a lossless protocol and perhaps the kernel will also drop the unprocessed messages). You should use TCP instead of UDP.
If you use TCP, you can use flags(flow-control) in your server configuration. If the senders are also syslog-ng, you can use it on their configurations, too. flow-control will slow down (or block) receiving logs if syslog-ng cannot process (write out, forward and so on) the messages in time. It can prevent losing logs.
For example: log { source(s_gms); flags(flow-control); destination(d_mssql); };
For more details about flow-control:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...
On 2011-03-10 06:28, Shawn Cannon wrote:
I really need help on this. I am collecting tons of SYSLOG data from over 500 firewalls. I have these syslogs going to a Barracuda Load Balancer first which will then send messages to 2 syslog-ng open source servers. Here is how I have my config file setup:
@version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # #
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html #
options { use_dns(no); };
###### # sources source s_gms { # message generated by Syslog-NG # internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) # unix-stream("/dev/log"); # messages from the kernel #file("/proc/kmsg" program_override("kernel")); # messages destined to udp514 udp(ip(0.0.0.0) port(514)); }; source s_syslogng { internal(); };
###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_mssql { sql(type(mssql) host("rawsql.abcdefg.net <http://rawsql.abcdefg.net>") port("1785") username("username") password("password") database("Syslog") table("syslogng")columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid")); };
log { source(s_gms); destination(d_mssql); };
log { source(s_syslogng); destination(d_messages); };
When messages are coming in, I am showing the following in the local syslog-ng messages:
Log statistics; processed='source(s_gms)=2155636', dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net <http://rawsql.abcdefg.net>,1785,Syslog)=1717472', stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net <http://rawsql.abcdefg.net>,1785,Syslog)=0',
I assume this is telling me that I am dropping the majority of my messages instead of them getting inserted into my MS SQL database? The MS SQL Database runs on a very beefy server with plenty of memory. I am trying to determine why this is being dropped. Please help as I am huge newbie when it comes to syslog-ng.
Thanks!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html