[syslog-ng] multiple source/destination setup

[Kevin]

On 6/28/05, William Bell <williamb at cwie.net> wrote:
> For a while now I have been using a configuration that includes 3 sources
> (remote udp, remote tcp, internal) and 3 destinations ( remote archive,
> local, loopback for an analyzer). Today I tried to move this setup that
> works on all levels to another machine. This new machine has an EXTREMELY
> large udp traffic base. About 7.5gb of syslog per day. The setup of the new
> machine is exactly the same the only variable is this large traffic volume.

Is the hardware the same? 

Can you provide details on the average and peak packets per second (PPS)
for the new host?  Perhaps you might monitor the switch port, perhaps by
using something like MRTG?


> Now, to the problem.
> I am receiving all the syslog on the remote destination archive across
> normal syslog export mechanisms. However the two log destinations under it
> are not receiving the "copies" of the logs.

So you are saying that the archive is locally logging the messages correctly,
but is failing to forward copies of the messages out?

> I was wondering if anyone else uses syslog-ng to log large pix firewall syslog
> traffic, and have seen anything similar. My theory is that syslog-ng is just not
> keeping up with the speed that its receiving data.

This is possible.  A syslog server is limited by the log software, by the OS,
and by the drives and drive controller.  The limits are usually more about
the number of messages (packets) per second than bytes per second.

Packets can be dropped in syslog-ng itself (logged via the internal source,
usually as "stats"), by the TCP/IP stack (logged in OS-specific kernel data
structures usually viewable via "netstat -p udp") or because the OS or NIC
cannot handle the data coming in over the LAN (often logged nowhere).


> Here is a copy of the conf file with only the private info adjusted.
. . .
>         log_fifo_size(0);
. . .
>         sync(1);
. . .

Ouch.  With these settings, syslog-ng isn't doing any internal buffering,
In my experience, a high-volume logger needs these two values set high,
I use the following:

        sync(256);
        log_fifo_size(4095);

With these settings, we are currently processing UDP syslog data totalling
just over 11GB/day (65 million messages/day) without significant packet loss,
there is some loss (perhaps one in fifty thousand packets) during the peak hour
of the day -- at the peak hour  the logger processes 6 million
messages totalling
just over 1GB, at that volume a few packets dropped by the OS is acceptable :)


Kevin Kadow

(P.S. Our record for log events successfully received and written in a
one second
interval is 2,100;  this is based on the message timestamp, I haven't
yet tried the
recently posted patch to count messages/second.)