Hello, On Fri, May 13, 2011 at 7:43 PM, Zeek Anow <zeekstern@gmail.com> wrote:
Something really wrong with syslog-ng or my config. I'm dropping way too many packets. I will admit that my configuration is probably really a large part of the problem and would appreciate it if someone could take a look at it and offer some suggestions. There is another thread going about a similar problem on a similar platform.
We recently upgraded to Solaris 10 from Solaris 9 and I don't recall us dropping that many packets before. And we also upgraded from a very older Sylog-ng version to 3.1.2. I am basing the dropped packets on the udp stats, not syslog-ng stats. Syslog-ng stats has NO dropped packets.
This implies that syslog-ng couldn't read the messages from the UDP socket so the kernel dropped incoming logs to the floor.
UDP udpInDatagrams -4599313 udpInErrors - 0 udpOutDatagrams - 3421 udpOutErrors - 0 tcpInErrs - 0 udpNoPorts -2587612 udpInCksumErrs - 0 udpInOverflows -95806254
...
Below is a very small sampling of our syslog-ng.conf. We are filtering on about 1400 devices most of which are Firewalls and routers. The IPs in the following sample have been made up.
1400 *distinct* log sources like TCP streams? A single-threaded syslog-ng instance might not scale for such amount of log sources. An option would be running multiple syslog-ng instances listening on different ports and configure clients to use different ports sharing the load between the syslog-ng instances.
One of my questions is "Does the number of devices we are filtering on make a difference? (1400)" We have several sites and use just one version of the syslogng.conf file. It is a lot easier to maintain one copy:))
Also notice the format: ("^10\.123\.10\.133$") for the filters. All 1400 are in that format. I was hoping this would help a little but don't really know for sure:))
Could you make your filters hierarchical? Using the netmask() filter and nested log statements you could reduce the number of filter evaluations. For example filter f_tenten { netmask("10.10/16";) }; filter tentenone { netmask("10.10.1/24"); }; filter tententwo { netmask("10.10.2/24"); }; filter f_teneleven { netmask("10.11/16";) }; ... and later log { # first big network source(...); filter(f_tenten); log { filter(f_tentenone); destination(...); }; log { filter(tententwo); destination(...); }; flags(final); }; log { # second big network source(...); filter(f_teneleven); ... }; and so on, organizing filters into tree / forest hierarchy instead of evaluating all filters one by one for all messages. This way logs coming from 10.10/16 will get processed in the first log block. Logs originating from other networks will trigger the evaluation of the f_tenten filter and as it will give a false result none of the embedded log statements (including their filters) will process the log. So the logs will get processed in the next log section (with the second big network comment). Using flags(final) is very important here, it tells syslog-ng to do not process more log{} sections for the given log message, otherwise all further log sections will process the message. If you want to log a message to multiple places then just add the additional destinations to the proper log{} block. Regards, Sandor