[syslog-ng]using pipe results in lost messages?

[Russo, Ben]

Hi, this is my first posting to syslog-ng@lists
Syslog-NG is very nice. This is the future of network wide logging.

I am using syslog-ng-1.5.16 with no modifications as downloaded from
balabit.hu
with libol-0.3.2 also with no modifications as downloaded from balabit.hu
I am running on RedHat v7.2 with all the latest patches.
I compiled syslog-ng with "rpm -tb" to build i386.rpms.
I got it set up with a minimum of hassle.

Here is my /etc/syslog-ng/syslog-ng.conf file (ln -s /etc/syslog-ng
/usr/etc/syslog-ng)

========================================================BEGIN syslog-ng.conf
options { sync (0);
          time_reopen (10);
          log_fifo_size (100);
          long_hostnames (off);
          use_dns (yes);
          use_fqdn (yes);
          create_dirs (yes);
          keep_hostname (yes);
          time_reap(65536);
        };

source s_sys { unix-stream ("/dev/log"); internal(); };
source s_udp { udp(); };

destination d_files { file("/var/log/syslog-ng/$HOST/$FACILITY.$LEVEL"
template("$ISODATE $HOST $FACILITY $PRIORITY $PROCESS $MSG\n")); };
destination d_pipe { pipe("/var/lib/pgsql/syslog-ng.pipe" template("INSERT
INTO testtable VALUES \( '$R_ISODATE', '$S_ISODATE', '$HOST', '$FACILITY',
'$PRIORITY', '$MSG'\)\;\n") template_escape(yes)); };

log { source(s_sys); source(s_udp); destination(d_files); };
log { source(s_sys); source(s_udp); destination(d_pipe); };
==================================================================END OF
FILE

Then I have a script that runs:
  
  while true
    do
        read INSERT < /var/lib/pgsql/syslog-ng.pipe
        echo $INSERT > /tmp/outfile
    done

And I have a testfile /tmp/wholelog with 1000 lines in it.
I run "logger -p user.info -f /tmp/wholelog"

at the end of the run I have 942 messages in
/var/log/syslog-ng/testbox.tnsi.com/user.info
and I have 332 lines in /tmp/outfile

Can anyone tell me how I can tune this up so that I don't loose so many
messages?
Eventually I will have 100 hosts in my data center redirecting logs to this
box
over a 100Mb/s full duplex switched network and I will replace the script
that
does reads with one that redirects input to psql

BTW, I have already tested a script that does:

cat /var/lib/pgsql/syslog-ng.pipe | psql -U dbauser -d test

And it works as long as I don't send more than about 20 messages per second.
Although there are some "template_escape" problems I have seen that cause me
to loose a few messages here and there... I'm not that worried about it.

But losing 5% of the messages just on the localhost seems a little odd?
It should be closer to 0% right?

One other question I have (after reading as much of src/affile.c as I could
understand)
Is there a way to output the priority and facility as seperate integer
values
rather than as a bitshifted combined 2 digit hex value?

-Ben.