[syslog-ng] Dropped messages to MSSQL?

[Shawn Cannon]

Unfortunately the product we use only supports sending the syslog over UDP.
I will try the latest version and the option you mentioned.

Shawn Cannon
On Mar 10, 2011 3:21 AM, "Zoltán Pallagi" <pzolee at balabit.hu> wrote:
> Hi,
>
> First of all, it seems that syslog-ng receives logs faster than your
> mssql server can process them that's the reason for dropping.
>
> However, you are using udp source for receiving logs, it's not the best
> solution if you don't want to lose logs (UDP is not a lossless protocol
> and perhaps the kernel will also drop the unprocessed messages).
> You should use TCP instead of UDP.
>
> If you use TCP, you can use flags(flow-control) in your server
> configuration. If the senders are also syslog-ng, you can use it on
> their configurations, too.
> flow-control will slow down (or block) receiving logs if syslog-ng
> cannot process (write out, forward and so on) the messages in time. It
> can prevent losing logs.
>
> For example:
> log {
> source(s_gms);
> flags(flow-control);
> destination(d_mssql);
> };
>
> For more details about flow-control:
>
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html
>
> On 2011-03-10 06:28, Shawn Cannon wrote:
>> I really need help on this. I am collecting tons of SYSLOG data from
>> over 500 firewalls. I have these syslogs going to a Barracuda Load
>> Balancer first which will then send messages to 2 syslog-ng open
>> source servers. Here is how I have my config file setup:
>>
>> @version: 3.0
>> #Default configuration file for syslog-ng.
>> #
>> # For a description of syslog-ng configuration file directives, please
>> read
>> # the syslog-ng Administrator's guide at:
>> #
>> #
>>
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
>> #
>>
>> options { use_dns(no);
>> };
>>
>> ######
>> # sources
>> source s_gms {
>> # message generated by Syslog-NG
>> # internal();
>> # standard Linux log source (this is the default place for the syslog()
>> # function to send logs to)
>> # unix-stream("/dev/log");
>> # messages from the kernel
>> #file("/proc/kmsg" program_override("kernel"));
>> # messages destined to udp514
>> udp(ip(0.0.0.0) port(514));
>> };
>> source s_syslogng {
>> internal();
>> };
>>
>> ######
>> # destinations
>> destination d_messages { file("/var/log/messages"); };
>> destination d_mssql {
>> sql(type(mssql) host("rawsql.abcdefg.net <http://rawsql.abcdefg.net>")
>> port("1785")
>> username("username") password("password") database("Syslog")
>> table("syslogng")columns("datetime varchar(16)", "host varchar(32)",
>> "program varchar(32)", "pid varchar(8)", "message varchar(4096)")
>> values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
>> indexes("datetime", "host", "program", "pid"));
>> };
>>
>>
>>
>> log {
>> source(s_gms);
>> destination(d_mssql);
>> };
>>
>> log {
>> source(s_syslogng);
>> destination(d_messages);
>> };
>>
>>
>> When messages are coming in, I am showing the following in the local
>> syslog-ng messages:
>>
>> Log statistics; processed='source(s_gms)=2155636',
>> dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net
>> <http://rawsql.abcdefg.net>,1785,Syslog)=1717472',
>> stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net
>> <http://rawsql.abcdefg.net>,1785,Syslog)=0',
>>
>> I assume this is telling me that I am dropping the majority of my
>> messages instead of them getting inserted into my MS SQL database?
>> The MS SQL Database runs on a very beefy server with plenty of
>> memory. I am trying to determine why this is being dropped. Please
>> help as I am huge newbie when it comes to syslog-ng.
>>
>> Thanks!
>>
>>
>>
______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110310/b5b16af8/attachment.htm