[syslog-ng] Flow-control behaviour and Cached lines timestamping, plus a suspend error in 2.1beta1

[Evan Rempel]

Behal, Pavel wrote:
> Dear Bazsi,
> 
> I have found two "features" and one bug I would like to discuss a little:
> 
> 1) We use the syslog-ng 2.1alpha1 as a root system syslog. The problem is, 
 > we have had two destinations setup. First destination was the traditional write
 > of system messages to the file like "/var/log/messages" etc. The second destination
 > path was to the TCP stream to the central syslog server. And the problem is,
 > we have used the flow-control flag on the central TCP path. But, when the
 > central TCP syslog-ng becomes unavailable, after all the buffers filled up,
 > the local system syslog-ng stopped processing any system messages. It
 > did not write anything through the local file destinations until the
 > TCP communication was re-established.
> Is this a correct behaviour? We have found, that to allow smooth 
 > processing of local system sources, we can not use flow-control
> flag on any destination paths it used..

Basically, you can not use flow control for any destination that has a source
of /proc/kmsg or /dev/log

To "get around" this, we use multiple instances of syslog-ng

instance one has;
- sources of /proc/kmsg and /dev/log
- destination of a pipe (for disk)
- destination of a pipe (for network)
- very large buffers (2 million records?)

instance two has
- source of the disk pipe as #1 destination
- small buffers ( 2000 )
- destination of files
- uses flow control.

instance three has
- source of the network pip as #1 destination
- small buffers (2000)
- destination of network
- uses flow control

This keeps the OS safe in the event that the disk or network can not keep up.

This allows us to log when messages are dropped to disk destinations
because they actually get dropped to the pipe in instance #1.

We can "restart" either the disk or the network instances (#2 #3)
without having to drop the connections from applications.

We can "restart" either the disk or the network instances without
loosing the large buffer of messages stored in instance #1.
If instances #2 and #3 used even smaller buffers, then even fewer
messages would be lost.

A side effect of this is that we can use multiple CPUs to filter, process
and log messages even though syslog-ng is single threaded.


We typically deploy this structure on our syslog servers, and not on all of
our individual servers and do NOT use flow control on the individual servers that
send messages to the syslog servers. We have about 15 different pipe destinations for
a variety of programs and storage destinations used for real time log analysis,
storage and alerting.

Evan Rempel.