Question on syslog-ng filtering and performance - host vs. netmask
Hi, I am using syslog-ng to do log collection and "routing" to different destinations based on the origin of the log. I used to (still do in some cases) filter like this: filter f_one_list { host("10.0.0.1") or host("10.0.2.1") or ... host("10.3.4.5"); } Since (too) many log sources break the RFC and put all kinds of strangeness in the "header", I started using netmask instead: filter f_another_list { netmask("10.0.0.1/32") or netmask("10.0.2.1/32") or ... netmask("10.3.4.5/32"); } My question is this - Is there a measurable performance difference from one to another? "intuitively" it would seem doing the netmask check should be faster than parsing the syslog header, but I thought I'd ask here. (currently I have some servers processing 4000+ messages per second with no real problem, but the CPUs are clearly busy. BTW - I am using 3.2.4 at the moment (looking at moving to the multi-threaded newer versions in the future) Thanks! Jim Hendrick
<jrhendri@maine.rr.com> writes:
I used to (still do in some cases) filter like this:
filter f_one_list { host("10.0.0.1") or host("10.0.2.1") or ... host("10.3.4.5"); }
Since (too) many log sources break the RFC and put all kinds of strangeness in the "header", I started using netmask instead:
filter f_another_list { netmask("10.0.0.1/32") or netmask("10.0.2.1/32") or ... netmask("10.3.4.5/32"); }
My question is this - Is there a measurable performance difference from one to another? "intuitively" it would seem doing the netmask check should be faster than parsing the syslog header, but I thought I'd ask here.
The header will be parsed anyway, but the netmask check is, indeed faster: instead of a string comparsion, it does only a numeric comparsion, on less data. At ~4000 messages / sec, it shouldn't make much of a difference though (but that's just my gut feeling). Nevertheless, netmask should be slightly faster, if I'm reading the code right. -- |8]
Thanks! - I'll change the filters (hadn't thought about the fact that the header is parsed anyway - Thanks for the sanity check) Jim On Fri, 2011-12-02 at 23:18 +0100, Gergely Nagy wrote:
<jrhendri@maine.rr.com> writes:
I used to (still do in some cases) filter like this:
filter f_one_list { host("10.0.0.1") or host("10.0.2.1") or ... host("10.3.4.5"); }
Since (too) many log sources break the RFC and put all kinds of strangeness in the "header", I started using netmask instead:
filter f_another_list { netmask("10.0.0.1/32") or netmask("10.0.2.1/32") or ... netmask("10.3.4.5/32"); }
My question is this - Is there a measurable performance difference from one to another? "intuitively" it would seem doing the netmask check should be faster than parsing the syslog header, but I thought I'd ask here.
The header will be parsed anyway, but the netmask check is, indeed faster: instead of a string comparsion, it does only a numeric comparsion, on less data.
At ~4000 messages / sec, it shouldn't make much of a difference though (but that's just my gut feeling). Nevertheless, netmask should be slightly faster, if I'm reading the code right.
participants (3)
-
Gergely Nagy
-
Jim
-
jrhendri＠maine.rr.com