Martin, how do I setup the config file to log to a CSV file so I can try the BCP method? I assume since BCP is a Windows program then I need to use SAMBA in Linux to create a share that BCP can read the log files from?<br>
<br><div class="gmail_quote">On Thu, Mar 10, 2011 at 10:23 AM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Yep, you're dropping most of your messages to SQL. Your dropped stats<br>
should equal 0 if you're not dropping any messages. The stats are<br>
listed by source and destination so you can see if a certain source<br>
can't read the messages fast enough and a certain destination can't<br>
write fast enough. I bet if you setup a simple file destination<br>
you'll see that it doesn't drop any at all while SQL continues to drop<br>
most messages. That's because writing to SQL is much more expensive<br>
than writing to file. One solution to this is to write to a file in<br>
CSV format and use MS-SQL bcp to load the files written out in<br>
batches. If you name the output files from syslog-ng by the minute of<br>
the timestamp, then you can have a simple script pick up the previous<br>
minute's log file and bcp it into MS-SQL.<br>
<div><div></div><div class="h5"><br>
On Wed, Mar 9, 2011 at 4:22 PM, Shawn Cannon <<a href="mailto:shawn@shawncannon.com">shawn@shawncannon.com</a>> wrote:<br>
> Can someone tell me what the numbers mean for processed and dropped? Am I<br>
> logging everything to SQL or am I dropping some?<br>
><br>
> Mar 9 17:12:03 syslog01 syslog-ng[1738]: Log statistics;<br>
> processed='source(s_gms)=2279331',<br>
> dropped='dst.sql(d_mssql#0,freetds,<a href="http://rawsql.adbcefg.net" target="_blank">rawsql.adbcefg.net</a>,1785,Syslog)=2213605',<br>
> stored='dst.sql(d_mssql#0,freetds,<a href="http://rawsql.abcdefg.net" target="_blank">rawsql.abcdefg.net</a>,1785,Syslog)=0',<br>
><br>
</div></div>> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation:<br>
> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
><br>
><br>
><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</blockquote></div><br>