Perl to the rescue! From the man page for Sybase::BCP: #!/usr/local/bin/perl use Sybase::BCP; $bcp = new Sybase::BCP $user, $passwd, $server; $bcp->config(INPUT => 'foo.bcp', OUTPUT => 'mydb.dbo.bar', SEPARATOR => '|'); $bcp->run; I recommend using tab as your separator instead of a comma, as it's less likely to naturally exist in the log you're processing. You could put in a loop around watching the directory for new files, but if it were me, I'd have syslog-ng write to program(script.pl) and have the script take care of writing to its own file handle and uploading when it feels like it. That way you don't have to worry about whether or not syslog-ng is done writing to the file or not. On Mon, Mar 14, 2011 at 6:24 AM, Shawn Cannon <shawn@shawncannon.com> wrote:
Martin, how do I setup the config file to log to a CSV file so I can try the BCP method? I assume since BCP is a Windows program then I need to use SAMBA in Linux to create a share that BCP can read the log files from?
On Thu, Mar 10, 2011 at 10:23 AM, Martin Holste <mcholste@gmail.com> wrote:
Yep, you're dropping most of your messages to SQL. Your dropped stats should equal 0 if you're not dropping any messages. The stats are listed by source and destination so you can see if a certain source can't read the messages fast enough and a certain destination can't write fast enough. I bet if you setup a simple file destination you'll see that it doesn't drop any at all while SQL continues to drop most messages. That's because writing to SQL is much more expensive than writing to file. One solution to this is to write to a file in CSV format and use MS-SQL bcp to load the files written out in batches. If you name the output files from syslog-ng by the minute of the timestamp, then you can have a simple script pick up the previous minute's log file and bcp it into MS-SQL.
On Wed, Mar 9, 2011 at 4:22 PM, Shawn Cannon <shawn@shawncannon.com> wrote:
Can someone tell me what the numbers mean for processed and dropped? Am I logging everything to SQL or am I dropping some?
Mar 9 17:12:03 syslog01 syslog-ng[1738]: Log statistics; processed='source(s_gms)=2279331',
dropped='dst.sql(d_mssql#0,freetds,rawsql.adbcefg.net,1785,Syslog)=2213605', stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=0',
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html