[syslog-ng] Dropped messages to MSSQL?

[Martin Holste]

Please see my response on your earlier thread regarding writing to
file and using bcp.

On Thu, Mar 10, 2011 at 6:17 AM, Shawn Cannon <shawn at shawncannon.com> wrote:
> Unfortunately the product we use only supports sending the syslog over UDP.
> I will try the latest version and the option you mentioned.
>
> Shawn Cannon
>
> On Mar 10, 2011 3:21 AM, "Zoltán Pallagi" <pzolee at balabit.hu> wrote:
>> Hi,
>>
>> First of all, it seems that syslog-ng receives logs faster than your
>> mssql server can process them that's the reason for dropping.
>>
>> However, you are using udp source for receiving logs, it's not the best
>> solution if you don't want to lose logs (UDP is not a lossless protocol
>> and perhaps the kernel will also drop the unprocessed messages).
>> You should use TCP instead of UDP.
>>
>> If you use TCP, you can use flags(flow-control) in your server
>> configuration. If the senders are also syslog-ng, you can use it on
>> their configurations, too.
>> flow-control will slow down (or block) receiving logs if syslog-ng
>> cannot process (write out, forward and so on) the messages in time. It
>> can prevent losing logs.
>>
>> For example:
>> log {
>> source(s_gms);
>> flags(flow-control);
>> destination(d_mssql);
>> };
>>
>> For more details about flow-control:
>>
>> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_flow_control.html
>>
>> On 2011-03-10 06:28, Shawn Cannon wrote:
>>> I really need help on this. I am collecting tons of SYSLOG data from
>>> over 500 firewalls. I have these syslogs going to a Barracuda Load
>>> Balancer first which will then send messages to 2 syslog-ng open
>>> source servers. Here is how I have my config file setup:
>>>
>>> @version: 3.0
>>> #Default configuration file for syslog-ng.
>>> #
>>> # For a description of syslog-ng configuration file directives, please
>>> read
>>> # the syslog-ng Administrator's guide at:
>>> #
>>> #
>>>
>>> http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
>>> #
>>>
>>> options { use_dns(no);
>>> };
>>>
>>> ######
>>> # sources
>>> source s_gms {
>>> # message generated by Syslog-NG
>>> # internal();
>>> # standard Linux log source (this is the default place for the syslog()
>>> # function to send logs to)
>>> # unix-stream("/dev/log");
>>> # messages from the kernel
>>> #file("/proc/kmsg" program_override("kernel"));
>>> # messages destined to udp514
>>> udp(ip(0.0.0.0) port(514));
>>> };
>>> source s_syslogng {
>>> internal();
>>> };
>>>
>>> ######
>>> # destinations
>>> destination d_messages { file("/var/log/messages"); };
>>> destination d_mssql {
>>> sql(type(mssql) host("rawsql.abcdefg.net <http://rawsql.abcdefg.net>")
>>> port("1785")
>>> username("username") password("password") database("Syslog")
>>> table("syslogng")columns("datetime varchar(16)", "host varchar(32)",
>>> "program varchar(32)", "pid varchar(8)", "message varchar(4096)")
>>> values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
>>> indexes("datetime", "host", "program", "pid"));
>>> };
>>>
>>>
>>>
>>> log {
>>> source(s_gms);
>>> destination(d_mssql);
>>> };
>>>
>>> log {
>>> source(s_syslogng);
>>> destination(d_messages);
>>> };
>>>
>>>
>>> When messages are coming in, I am showing the following in the local
>>> syslog-ng messages:
>>>
>>> Log statistics; processed='source(s_gms)=2155636',
>>> dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net
>>> <http://rawsql.abcdefg.net>,1785,Syslog)=1717472',
>>> stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net
>>> <http://rawsql.abcdefg.net>,1785,Syslog)=0',
>>>
>>> I assume this is telling me that I am dropping the majority of my
>>> messages instead of them getting inserted into my MS SQL database?
>>> The MS SQL Database runs on a very beefy server with plenty of
>>> memory. I am trying to determine why this is being dropped. Please
>>> help as I am huge newbie when it comes to syslog-ng.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>