Thanks for the reply Sandor. Much appreciated!1 I moved your comments up here to make it a little easier. Syslog-ng not being able to read the messages from UDP socket happens very fast. Probably within a minute or less of turning syslog-ng on. I have the only 2 Solaris parameters that I can set, (recv_high_water and udp_max_buf) set to the highest limit possible. I was thinking that something on the syslog-ng side might help. log_fetch_limit, log_iw_size log_fifo, flush_lines etc might help, but I don't have a clue of where to start or what to set them to. I could spend months pulling numbers out of the air and plugging them in to see what happens:)) With regards to your "1400 *distinct* log sources like TCP streams?". Up until a few weeks ago, we did have about 8 devices that were distinct log sources like TCP streams. This accounted for about 60% of our traffic. What we found was that a few devices were about 6 days behind in the log files. We changed their method of logging so it does not use syslog-ng anymore. I really like your idea of making the filters hierarchical and appreciate your example. That makes sense to me. At one time, we did do something similar to that but used a lot of regex to accomplish it. That is when we changed and went the ("^10\.123\.10\.133$") route for each IP address, thinking it would be more efficient. I will check into this asap!! Regards, Zeek On Fri, May 13, 2011 at 2:44 PM, Sandor Geller < Sandor.Geller@morganstanley.com> wrote:
Hello,
On Fri, May 13, 2011 at 7:43 PM, Zeek Anow <zeekstern@gmail.com> wrote:
Something really wrong with syslog-ng or my config. I'm dropping way too many packets. I will admit that my configuration is probably really a large part of the problem and would appreciate it if someone could take a look at it and offer some suggestions. There is another thread going about a similar problem on a similar platform.
We recently upgraded to Solaris 10 from Solaris 9 and I don't recall us dropping that many packets before. And we also upgraded from a very older Sylog-ng
version
to 3.1.2. I am basing the dropped packets on the udp stats, not syslog-ng stats. Syslog-ng stats has NO dropped packets.
This implies that syslog-ng couldn't read the messages from the UDP socket so the kernel dropped incoming logs to the floor.
UDP udpInDatagrams -4599313 udpInErrors - 0 udpOutDatagrams - 3421 udpOutErrors - 0 tcpInErrs - 0 udpNoPorts -2587612 udpInCksumErrs - 0 udpInOverflows -95806254
...
Below is a very small sampling of our syslog-ng.conf. We are filtering on about 1400 devices most of which are Firewalls and routers. The IPs in the following sample have been made up.
1400 *distinct* log sources like TCP streams? A single-threaded syslog-ng instance might not scale for such amount of log sources. An option would be running multiple syslog-ng instances listening on different ports and configure clients to use different ports sharing the load between the syslog-ng instances.
One of my questions is "Does the number of devices we are filtering on make a difference? (1400)" We have several sites and use just one version of the syslogng.conf file. It is a lot easier to maintain one copy:))
Also notice the format: ("^10\.123\.10\.133$") for the filters. All 1400 are in that format. I was hoping this would help a little but don't really know for sure:))
Could you make your filters hierarchical? Using the netmask() filter and nested log statements you could reduce the number of filter evaluations. For example
filter f_tenten { netmask("10.10/16";) };
filter tentenone { netmask("10.10.1/24"); };
filter tententwo { netmask("10.10.2/24"); };
filter f_teneleven { netmask("10.11/16";) };
...
and later
log { # first big network source(...); filter(f_tenten); log { filter(f_tentenone); destination(...); }; log { filter(tententwo); destination(...); }; flags(final); };
log { # second big network source(...); filter(f_teneleven); ... };
and so on, organizing filters into tree / forest hierarchy instead of evaluating all filters one by one for all messages. This way logs coming from 10.10/16 will get processed in the first log block. Logs originating from other networks will trigger the evaluation of the f_tenten filter and as it will give a false result none of the embedded log statements (including their filters) will process the log. So the logs will get processed in the next log section (with the second big network comment).
Using flags(final) is very important here, it tells syslog-ng to do not process more log{} sections for the given log message, otherwise all further log sections will process the message. If you want to log a message to multiple places then just add the additional destinations to the proper log{} block.
Regards,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html