[syslog-ng] db-parser reuse for multiple logs?

[Mikkel Leth Carlsen]

You guess you are right. The filter that I’m using is matching a substring within the message, so applying dbparser before the filter might not have any impact. I’ll give it a go…

And yes, the parser is extremely fast. My current setup is currently processing (and parsing) around 20k events per second on a 2 CPU VMWare instance without any signs of problems…

/Mikkel

Fra: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] På vegne af Jim Hendrick
Sendt: 10. april 2015 14:53
Til: Syslog-ng users' and developers' mailing list
Emne: Re: [syslog-ng] db-parser reuse for multiple logs?

 I think it would depend on the filter. If it uses something that is readily available then maybe filter first.  If is a match within the message, then possibly not.

The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress.

Jim



Sent from my Verizon Wireless 4G LTE smartphone


-------- Original message --------
From: Mikkel Leth Carlsen <mlca at tdc.dk<mailto:mlca at tdc.dk>>
Date: 04/10/2015 2:11 AM (GMT-05:00)
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] db-parser reuse for multiple logs?

Hi Evan

Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting:

filter(filter_host1_or_host2);

before the parser below?

/Mikkel

> -----Oprindelig meddelelse-----
> Fra: syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu> [mailto:syslog-ng-
> bounces at lists.balabit.hu<mailto:bounces at lists.balabit.hu>] På vegne af Evan Rempel
> Sendt: 8. april 2015 17:38
> Til: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
> Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
>
> I don't have an answer to your actual question, however, you can work
> around it by
>
> log {
>          source(src_udp);
>          parser(myparser);
>          log {
>                  filter(filter_host1);
>                  destination(dst_host1);
>                  flags(final);
>          };
>          log {
>                  filter(filter_host2);
>                  destination(dst_host2);
>                  flags(final);
>          };
> };
>
> Hope that helps.
>
> Evan.
>
> On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
> > Hi
> >
> > Are db-parsers defined in syslog-ng configurations not reusable for
> multiple logs?  A simplified example (syslog 3.6.2):
> >
> > parser myparser {
> >          db_parser(
> >                  file("/usr/local/etc/patterndb.d/myparser.xml")
> >          );
> > };
> >
> > template mytemplate {
> >           template("${A};${B};${C}\n");
> > }
> >
> > filter filter_host1 {
> >          netmask(10.0.0.1/255.255.255.255);
> > };
> >
> > filter filter_host2 {
> >          netmask(10.0.0.2/255.255.255.255);
> > };
> >
> > destination dst_host1 {
> >          file("host1.log" perm(0644) template(mytemplate));
> > };
> >
> >
> > destination dst_host2 {
> >          file("host2.log" perm(0644) template(mytemplate));
> > };
> >
> > log {
> >          source(src_udp);
> >          filter(filter_host1);
> >          parser(myparser);
> >          destination(dst_host1);
> >          flags(final);
> > };
> >
> > log {
> >          source(src_udp);
> >          filter(filter_host2);
> >          parser(myparser);
> >          destination(dst_host2);
> >          flags(final);
> > };
> >
> > This seems to work as expected and 'syslog-ng -s' does not report any
> problems, but I see the following in the syslog-ng internal log:
> >
> > Internal error, duplicate configuration elements refer to the same
> persistent config; name='db-
> parser(/usr/local/etc/patterndb.d/myparser.xml)'
> > Internal error, duplicate configuration elements refer to the same
> persistent config; name='db-
> parser(/usr/local/etc/patterndb.d/myparser.xml)'
> >
> > /Mikkel
> >
> _______________________________________________________________________
> _______
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> _______________________________________________________________________
> _______
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150413/1a9942e4/attachment.htm