[syslog-ng] Reading Log file

[Martin Holste]

Perl to the rescue!  From the man page for Sybase::BCP:

#!/usr/local/bin/perl

use Sybase::BCP;

$bcp = new Sybase::BCP $user, $passwd, $server;
$bcp->config(INPUT => 'foo.bcp',
  OUTPUT => 'mydb.dbo.bar',
  SEPARATOR => '|');
$bcp->run;

I recommend using tab as your separator instead of a comma, as it's
less likely to naturally exist in the log you're processing.

You could put in a loop around watching the directory for new files,
but if it were me, I'd have syslog-ng write to program(script.pl) and
have the script take care of writing to its own file handle and
uploading when it feels like it.  That way you don't have to worry
about whether or not syslog-ng is done writing to the file or not.

On Mon, Mar 14, 2011 at 6:24 AM, Shawn Cannon <shawn at shawncannon.com> wrote:
> Martin, how do I setup the config file to log to a CSV file so I can try the
> BCP method?  I assume since BCP is a Windows program then I need to use
> SAMBA in Linux to create a share that BCP can read the log files from?
>
> On Thu, Mar 10, 2011 at 10:23 AM, Martin Holste <mcholste at gmail.com> wrote:
>>
>> Yep, you're dropping most of your messages to SQL.  Your dropped stats
>> should equal 0 if you're not dropping any messages.  The stats are
>> listed by source and destination so you can see if a certain source
>> can't read the messages fast enough and a certain destination can't
>> write fast enough.  I bet if you setup a simple file destination
>> you'll see that it doesn't drop any at all while SQL continues to drop
>> most messages.  That's because writing to SQL is much more expensive
>> than writing to file.  One solution to this is to write to a file in
>> CSV format and use MS-SQL bcp to load the files written out in
>> batches.  If you name the output files from syslog-ng by the minute of
>> the timestamp, then you can have a simple script pick up the previous
>> minute's log file and bcp it into MS-SQL.
>>
>> On Wed, Mar 9, 2011 at 4:22 PM, Shawn Cannon <shawn at shawncannon.com>
>> wrote:
>> > Can someone tell me what the numbers mean for processed and dropped?  Am
>> > I
>> > logging everything to SQL or am I dropping some?
>> >
>> > Mar  9 17:12:03 syslog01 syslog-ng[1738]: Log statistics;
>> > processed='source(s_gms)=2279331',
>> >
>> > dropped='dst.sql(d_mssql#0,freetds,rawsql.adbcefg.net,1785,Syslog)=2213605',
>> > stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=0',
>> >
>> >
>> > ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation:
>> > http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.campin.net/syslog-ng/faq.html
>> >
>> >
>> >
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>
>