Re: [syslog-ng] db-parser reuse for multiple logs?

13 Apr 2015

      You guess you are right. The filter that I’m using is matching a substring within the message, so applying dbparser before the filter might not have any impact. I’ll give it a go…

And yes, the parser is extremely fast. My current setup is currently processing (and parsing) around 20k events per second on a 2 CPU VMWare instance without any signs of problems…

/Mikkel

Fra: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] På vegne af Jim Hendrick
Sendt: 10. april 2015 14:53
Til: Syslog-ng users' and developers' mailing list
Emne: Re: [syslog-ng] db-parser reuse for multiple logs?

 I think it would depend on the filter. If it uses something that is readily available then maybe filter first.  If is a match within the message, then possibly not.

The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress.

Jim

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message --------
From: Mikkel Leth Carlsen <mlca@tdc.dk<mailto:mlca@tdc.dk>>
Date: 04/10/2015 2:11 AM (GMT-05:00)
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>
Subject: Re: [syslog-ng] db-parser reuse for multiple logs?

Hi Evan

Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting:

filter(filter_host1_or_host2);

before the parser below?

/Mikkel
...
-----Oprindelig meddelelse-----
Fra: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-
bounces@lists.balabit.hu<mailto:bounces@lists.balabit.hu>] På vegne af Evan Rempel
Sendt: 8. april 2015 17:38
Til: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>
Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
I don't have an answer to your actual question, however, you can work
around it by
log {
         source(src_udp);
         parser(myparser);
         log {
                 filter(filter_host1);
                 destination(dst_host1);
                 flags(final);
         };
         log {
                 filter(filter_host2);
                 destination(dst_host2);
                 flags(final);
         };
};
Hope that helps.
Evan.
On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
...
Hi
Are db-parsers defined in syslog-ng configurations not reusable for
multiple logs?  A simplified example (syslog 3.6.2):
parser myparser {
         db_parser(
                 file("/usr/local/etc/patterndb.d/myparser.xml")
         );
};
template mytemplate {
          template("${A};${B};${C}\n");
}
filter filter_host1 {
         netmask(10.0.0.1/255.255.255.255);
};
filter filter_host2 {
         netmask(10.0.0.2/255.255.255.255);
};
destination dst_host1 {
         file("host1.log" perm(0644) template(mytemplate));
};
destination dst_host2 {
         file("host2.log" perm(0644) template(mytemplate));
};
log {
         source(src_udp);
         filter(filter_host1);
         parser(myparser);
         destination(dst_host1);
         flags(final);
};
log {
         source(src_udp);
         filter(filter_host2);
         parser(myparser);
         destination(dst_host2);
         flags(final);
};
This seems to work as expected and 'syslog-ng -s' does not report any
problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same
persistent config; name='db-
parser(/usr/local/etc/patterndb.d/myparser.xml)'
Internal error, duplicate configuration elements refer to the same
persistent config; name='db-
parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel
_______________________________________________________________________
_______
...
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
_______________________________________________________________________
_______
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq