That's definitely some heavy-duty regex. You'd be a good candidate for the pattern db, as the pattern matching engine is orders of magnitude faster than PCRE because it uses trie-based pattern searching. It also allows for extracting the matches and using them in the output macros, so you wouldn't have to sacrifice any functionality. I would estimate that it would drop your CPU usage down to around 25-30% while doing all of the work in a single thread.<br>
<br>@Balabit: You know what goes great with pattern matching? CUDA support with Nvidia cards for GPU-based pattern matching acceleration. They've got preliminary support for it in the Open Information Security Foundation's (OISF) Suricata IDS engine. That project is GPL, so you could port most of that code directly into the pattern db matcher for the OSE version of SyslogNG. $500 USD will buy a GPU with 480 stream processors, so you could match 480 patterns simultaneously, per card. You can link up to four cards together, so you could match 1920 patterns in parallel, offloaded from the CPU, on commodity hardware. So, a server costing under $5,000 could probably process (maybe not store) 250,000+ messages per second. Even if there wasn't much speed increase, the CPU offload alone would probably be worth it for busy log servers.<br>
<br><div class="gmail_quote">On Mon, Mar 22, 2010 at 11:50 AM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div class="h5">On Fri, 2010-03-19 at 00:56 -0400, Jan Schaumann wrote:<br>
> Martin Holste <<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>> wrote:<br>
> > How many messages per second is the system attempting to handle? I'm<br>
> > very surprised that you're seeing that level of utilization. In our<br>
> > setup we've never had a problem pushing up through 30,000 messages per<br>
> > second written to disk with Syslog-NG in production, and I've pushed<br>
> > more than 70,000 per second in development.<br>
><br>
> Could you provide your configuration for these systems (including<br>
> sysctls or kernel tunables etc.)? I've so far not been able to get my<br>
> systems to accept and process (without any regex matching) more than<br>
> approximagely 25K - 30K UDP messages/s.<br>
><br>
<br>
</div></div>udp() may be the problem here, in fact since syslog-ng is not thread<br>
based, its latency to poll for the udp socket may be a bit too long.<br>
<br>
increasing the udp socket buffer to insane values (like 256MB) could<br>
help alleviate the problem somewhat, but the latency is the root cause.<br>
<br>
adding time_sleep() to the mix increases latency even further. It only<br>
helps if you have a lot of connections (and udp is only one even if you<br>
have thousands of clients, while tcp keeps a separate connection for<br>
each client).<br>
<br>
After I get to the end of integrating the plugins branch, I intend to<br>
work on enabling multiple threads, thus decreasing the latency.<br>
<font color="#888888"><br>
<br>
--<br>
Bazsi<br>
</font><div><div></div><div class="h5"><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</div></div></blockquote></div><br>