[syslog-ng] db-parser reuse for multiple logs?

[Jim Hendrick]

 I think it would depend on the filter. If it uses something that is readily available then maybe filter first.  If is a match within the message, then possibly not. 
The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. 
Jim


Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message --------
From: Mikkel Leth Carlsen <mlca at tdc.dk> 
Date: 04/10/2015  2:11 AM  (GMT-05:00) 
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu> 
Subject: Re: [syslog-ng] db-parser reuse for multiple logs? 

Hi Evan

Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting:

filter(filter_host1_or_host2);

before the parser below? 

/Mikkel

> -----Oprindelig meddelelse-----
> Fra: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-
> bounces at lists.balabit.hu] På vegne af Evan Rempel
> Sendt: 8. april 2015 17:38
> Til: syslog-ng at lists.balabit.hu
> Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
> 
> I don't have an answer to your actual question, however, you can work
> around it by
> 
> log {
>          source(src_udp);
>          parser(myparser);
>          log {
>                  filter(filter_host1);
>                  destination(dst_host1);
>                  flags(final);
>          };
>          log {
>                  filter(filter_host2);
>                  destination(dst_host2);
>                  flags(final);
>          };
> };
> 
> Hope that helps.
> 
> Evan.
> 
> On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
> > Hi
> >
> > Are db-parsers defined in syslog-ng configurations not reusable for
> multiple logs?  A simplified example (syslog 3.6.2):
> >
> > parser myparser {
> >          db_parser(
> >                  file("/usr/local/etc/patterndb.d/myparser.xml")
> >          );
> > };
> >
> > template mytemplate {
> >           template("${A};${B};${C}\n");
> > }
> >
> > filter filter_host1 {
> >          netmask(10.0.0.1/255.255.255.255);
> > };
> >
> > filter filter_host2 {
> >          netmask(10.0.0.2/255.255.255.255);
> > };
> >
> > destination dst_host1 {
> >          file("host1.log" perm(0644) template(mytemplate));
> > };
> >
> >
> > destination dst_host2 {
> >          file("host2.log" perm(0644) template(mytemplate));
> > };
> >
> > log {
> >          source(src_udp);
> >          filter(filter_host1);
> >          parser(myparser);
> >          destination(dst_host1);
> >          flags(final);
> > };
> >
> > log {
> >          source(src_udp);
> >          filter(filter_host2);
> >          parser(myparser);
> >          destination(dst_host2);
> >          flags(final);
> > };
> >
> > This seems to work as expected and 'syslog-ng -s' does not report any
> problems, but I see the following in the syslog-ng internal log:
> >
> > Internal error, duplicate configuration elements refer to the same
> persistent config; name='db-
> parser(/usr/local/etc/patterndb.d/myparser.xml)'
> > Internal error, duplicate configuration elements refer to the same
> persistent config; name='db-
> parser(/usr/local/etc/patterndb.d/myparser.xml)'
> >
> > /Mikkel
> >
> _______________________________________________________________________
> _______
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> _______________________________________________________________________
> _______
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150410/801037c5/attachment.htm