Thanks for the reply Sandor. Much appreciated!1<br><br>I moved your comments up here to make it a little easier.<br><br>Syslog-ng not being able to read the messages from UDP socket happens<br>very fast. Probably within a minute or less of turning syslog-ng on. <br>
I have the only 2 Solaris parameters that I can set, (recv_high_water and<br>udp_max_buf) set to the highest limit possible. I was thinking that<br>something on the syslog-ng side might help. log_fetch_limit, log_iw_size<br>
log_fifo, flush_lines etc might help, but I don't have a clue of<br>where to start or what to set them to. I could spend months pulling<br>numbers out of the air and plugging them in to see what happens:))<br><br>With regards to your "1400 *distinct* log sources like TCP streams?".<br>
Up until a few weeks ago, we did have about 8 devices that were<br>distinct log sources like TCP streams. This accounted for about<br>60% of our traffic. What we found was that a few devices<br>were about 6 days behind in the log files. We changed their<br>
method of logging so it does not use syslog-ng anymore. <br><br>I really like your idea of making the filters hierarchical and <br>appreciate your example. That makes sense to me. At one time, <br>we did do something similar to that but used a lot of regex to<br>
accomplish it. That is when we changed and went the<br>("^10\.123\.10\.133$") route for each IP address, thinking it<br>would be more efficient. I will check into this asap!!<br><br>Regards,<br>Zeek<br><br><div class="gmail_quote">
On Fri, May 13, 2011 at 2:44 PM, Sandor Geller <span dir="ltr"><<a href="mailto:Sandor.Geller@morganstanley.com">Sandor.Geller@morganstanley.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hello,<br>
<div class="im"><br>
On Fri, May 13, 2011 at 7:43 PM, Zeek Anow <<a href="mailto:zeekstern@gmail.com">zeekstern@gmail.com</a>> wrote:<br>
><br>
> Something really wrong with syslog-ng or my config. I'm dropping way too<br>
> many packets.<br>
> I will admit that my configuration is probably really a large part of the<br>
> problem<br>
> and would appreciate it if someone could take a look at it and offer some<br>
> suggestions.<br>
> There is another thread going about a similar problem on a similar<br>
> platform.<br>
><br>
> We recently upgraded to Solaris 10 from Solaris 9 and I don't recall us<br>
> dropping that<br>
> many packets before. And we also upgraded from a very older Sylog-ng version<br>
> to 3.1.2.<br>
> I am basing the dropped packets on the udp stats, not syslog-ng stats.<br>
> Syslog-ng stats has NO dropped packets.<br>
<br>
</div>This implies that syslog-ng couldn't read the messages from the UDP<br>
socket so the kernel dropped incoming logs to the floor.<br>
<div class="im"><br>
> UDP udpInDatagrams -4599313 udpInErrors - 0<br>
> udpOutDatagrams - 3421 udpOutErrors - 0<br>
> tcpInErrs - 0 udpNoPorts -2587612<br>
> udpInCksumErrs - 0 udpInOverflows -95806254<br>
><br>
<br>
</div>...<br>
<div class="im"><br>
> Below is a very small sampling of our syslog-ng.conf. We are filtering on<br>
> about<br>
> 1400 devices most of which are Firewalls and routers. The IPs in the<br>
> following<br>
> sample have been made up.<br>
<br>
</div>1400 *distinct* log sources like TCP streams? A single-threaded<br>
syslog-ng instance might not scale for such amount of log sources. An<br>
option would be running multiple syslog-ng instances listening on<br>
different ports and configure clients to use different ports sharing<br>
the load between the syslog-ng instances.<br>
<div class="im"><br>
> One of my questions is "Does the number of devices we are filtering on make<br>
> a difference? (1400)"<br>
> We have several sites and use just one version of the syslogng.conf file. It<br>
> is<br>
> a lot easier to maintain one copy:))<br>
><br>
> Also notice the format: ("^10\.123\.10\.133$") for the filters. All 1400<br>
> are in that format.<br>
> I was hoping this would help a little but don't really know for sure:))<br>
<br>
</div>Could you make your filters hierarchical? Using the netmask() filter<br>
and nested log statements you could reduce the number of filter<br>
evaluations. For example<br>
<br>
filter f_tenten {<br>
netmask("10.10/16";)<br>
};<br>
<br>
filter tentenone {<br>
netmask("10.10.1/24");<br>
};<br>
<br>
filter tententwo {<br>
netmask("10.10.2/24");<br>
};<br>
<br>
filter f_teneleven {<br>
netmask("10.11/16";)<br>
};<br>
<br>
...<br>
<br>
and later<br>
<br>
log {<br>
# first big network<br>
source(...);<br>
filter(f_tenten);<br>
log {<br>
filter(f_tentenone);<br>
destination(...);<br>
};<br>
log {<br>
filter(tententwo);<br>
destination(...);<br>
};<br>
flags(final);<br>
};<br>
<br>
log {<br>
# second big network<br>
source(...);<br>
filter(f_teneleven);<br>
...<br>
};<br>
<br>
and so on, organizing filters into tree / forest hierarchy instead of<br>
evaluating all filters one by one for all messages. This way logs<br>
coming from 10.10/16 will get processed in the first log block. Logs<br>
originating from other networks will trigger the evaluation of the<br>
f_tenten filter and as it will give a false result none of the<br>
embedded log statements (including their filters) will process the<br>
log. So the logs will get processed in the next log section (with the<br>
second big network comment).<br>
<br>
Using flags(final) is very important here, it tells syslog-ng to do<br>
not process more log{} sections for the given log message, otherwise<br>
all further log sections will process the message. If you want to log<br>
a message to multiple places then just add the additional destinations<br>
to the proper log{} block.<br>
<br>
Regards,<br>
<br>
Sandor<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</blockquote></div><br>