Re: [syslog-ng] db-parser reuse for multiple logs?
I think it would depend on the filter. If it uses something that is readily available then maybe filter first. If is a match within the message, then possibly not. The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Mikkel Leth Carlsen <mlca@tdc.dk> Date: 04/10/2015 2:11 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] db-parser reuse for multiple logs? Hi Evan Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting: filter(filter_host1_or_host2); before the parser below? /Mikkel
-----Oprindelig meddelelse----- Fra: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] På vegne af Evan Rempel Sendt: 8. april 2015 17:38 Til: syslog-ng@lists.balabit.hu Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
I don't have an answer to your actual question, however, you can work around it by
log { source(src_udp); parser(myparser); log { filter(filter_host1); destination(dst_host1); flags(final); }; log { filter(filter_host2); destination(dst_host2); flags(final); }; };
Hope that helps.
Evan.
On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
Hi
Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2):
parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); };
template mytemplate { template("${A};${B};${C}\n"); }
filter filter_host1 { netmask(10.0.0.1/255.255.255.255); };
filter filter_host2 { netmask(10.0.0.2/255.255.255.255); };
destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); };
destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); };
log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); };
log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); };
This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel
_______________________________________________________________________ _______
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Jim, Mind if I ask you what the specs are on that system ? John — Sent from Mailbox On Fri, Apr 10, 2015 at 7:53 AM, Jim Hendrick <jrhendri@roadrunner.com> wrote:
I think it would depend on the filter. If it uses something that is readily available then maybe filter first. If is a match within the message, then possibly not. The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Mikkel Leth Carlsen <mlca@tdc.dk> Date: 04/10/2015 2:11 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] db-parser reuse for multiple logs? Hi Evan Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting: filter(filter_host1_or_host2); before the parser below? /Mikkel
-----Oprindelig meddelelse----- Fra: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] På vegne af Evan Rempel Sendt: 8. april 2015 17:38 Til: syslog-ng@lists.balabit.hu Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
I don't have an answer to your actual question, however, you can work around it by
log { source(src_udp); parser(myparser); log { filter(filter_host1); destination(dst_host1); flags(final); }; log { filter(filter_host2); destination(dst_host2); flags(final); }; };
Hope that helps.
Evan.
On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
Hi
Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2):
parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); };
template mytemplate { template("${A};${B};${C}\n"); }
filter filter_host1 { netmask(10.0.0.1/255.255.255.255); };
filter filter_host2 { netmask(10.0.0.2/255.255.255.255); };
destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); };
destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); };
log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); };
log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); };
This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel
_______________________________________________________________________ _______
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You guess you are right. The filter that I’m using is matching a substring within the message, so applying dbparser before the filter might not have any impact. I’ll give it a go… And yes, the parser is extremely fast. My current setup is currently processing (and parsing) around 20k events per second on a 2 CPU VMWare instance without any signs of problems… /Mikkel Fra: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] På vegne af Jim Hendrick Sendt: 10. april 2015 14:53 Til: Syslog-ng users' and developers' mailing list Emne: Re: [syslog-ng] db-parser reuse for multiple logs? I think it would depend on the filter. If it uses something that is readily available then maybe filter first. If is a match within the message, then possibly not. The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Mikkel Leth Carlsen <mlca@tdc.dk<mailto:mlca@tdc.dk>> Date: 04/10/2015 2:11 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] db-parser reuse for multiple logs? Hi Evan Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting: filter(filter_host1_or_host2); before the parser below? /Mikkel
-----Oprindelig meddelelse----- Fra: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng- bounces@lists.balabit.hu<mailto:bounces@lists.balabit.hu>] På vegne af Evan Rempel Sendt: 8. april 2015 17:38 Til: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
I don't have an answer to your actual question, however, you can work around it by
log { source(src_udp); parser(myparser); log { filter(filter_host1); destination(dst_host1); flags(final); }; log { filter(filter_host2); destination(dst_host2); flags(final); }; };
Hope that helps.
Evan.
On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
Hi
Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2):
parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); };
template mytemplate { template("${A};${B};${C}\n"); }
filter filter_host1 { netmask(10.0.0.1/255.255.255.255); };
filter filter_host2 { netmask(10.0.0.2/255.255.255.255); };
destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); };
destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); };
log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); };
log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); };
This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel
_______________________________________________________________________ _______
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Mikkel, On Mon, Apr 13, 2015 at 06:48:06AM +0000, Mikkel Leth Carlsen wrote:
And yes, the parser is extremely fast. My current setup is currently processing (and parsing) around 20k events per second on a 2 CPU VMWare instance without any signs of problems…
How many rules do you have, if I may ask?
Just to give a performance example for the patterndb parser. We process approximately 5,000 msg/sec and use two patterndb parsers. One with 20,000 patterns and another with 4.300 patterns. All messages go through both parsers. This consumes about 5% of a 2 socket, Intel X5675 (6 core), 3GHz cpu system. Evan. On 04/12/2015 11:51 PM, Fabien Wernli wrote:
Hi Mikkel,
On Mon, Apr 13, 2015 at 06:48:06AM +0000, Mikkel Leth Carlsen wrote:
And yes, the parser is extremely fast. My current setup is currently processing (and parsing) around 20k events per second on a 2 CPU VMWare instance without any signs of problems… How many rules do you have, if I may ask?
On Mon, Apr 13, 2015 at 08:37:56AM -0700, Evan Rempel wrote:
We process approximately 5,000 msg/sec and use two patterndb parsers. One with 20,000 patterns and another with 4.300 patterns. All messages go through both parsers.
25k patterns… …you're lying!
I know you are being facetious, but... We generate our patterns programmatically from our inventory/asset tracking database. We use the syslog-ng patterndb to apply metadata about our hosts and applications. The metadata consists of: 1. Which unit in our organization is responsible for an host/application pair, so in the event that a log message should generate an alert/incident, our alerting layer knows which group to direct the alert to. 2. The role that the host plays (development, pre-production, production, BCP or Disaster recovery). That information is used to decide if alerts should go to pagers/cell phones, if they should go to e-mail or ticketing systems or if they should be ignored completely. Essentially a severity based on the role of the host. What this all means is that we use syslog-ng patterdb against a template of "$HOST:$PROGRAM" and have patterns for all hosts/program combinations (not all combinations since there is a catchall pattern) that we have in our asset tracking system. It is this large combination that makes the 20,000 pattern database. We have the other 4,300 patterns that match the actual log lines and define what kind of event each log line is. We are currently matching > 98% of our log volume and classifying the messages as one of safe, alert, timer, heartbeat, collection, rate threshold or command line tool execution. Just thought I would explain a little about how useful the patterndb is at our site. Evan. On 04/14/2015 02:08 AM, Fabien Wernli wrote:
On Mon, Apr 13, 2015 at 08:37:56AM -0700, Evan Rempel wrote:
We process approximately 5,000 msg/sec and use two patterndb parsers. One with 20,000 patterns and another with 4.300 patterns. All messages go through both parsers. 25k patterns… …you're lying!
Hi Evan, Thanks for clarifying this in so much detail. Really impressive figures you have there! I was already confident that syslog-ng would scale very well. Now I'm absolutely certain. Cheers
participants (5)
-
Evan Rempel
-
Fabien Wernli
-
Jim Hendrick
-
John Dyer
-
Mikkel Leth Carlsen