db-parser reuse for multiple logs?
Hi Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2): parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); }; template mytemplate { template("${A};${B};${C}\n"); } filter filter_host1 { netmask(10.0.0.1/255.255.255.255); }; filter filter_host2 { netmask(10.0.0.2/255.255.255.255); }; destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); }; destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); }; log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); }; log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); }; This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log: Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' /Mikkel
I don't have an answer to your actual question, however, you can work around it by log { source(src_udp); parser(myparser); log { filter(filter_host1); destination(dst_host1); flags(final); }; log { filter(filter_host2); destination(dst_host2); flags(final); }; }; Hope that helps. Evan. On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
Hi
Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2):
parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); };
template mytemplate { template("${A};${B};${C}\n"); }
filter filter_host1 { netmask(10.0.0.1/255.255.255.255); };
filter filter_host2 { netmask(10.0.0.2/255.255.255.255); };
destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); };
destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); };
log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); };
log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); };
This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Evan Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting: filter(filter_host1_or_host2); before the parser below? /Mikkel
-----Oprindelig meddelelse----- Fra: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng- bounces@lists.balabit.hu] På vegne af Evan Rempel Sendt: 8. april 2015 17:38 Til: syslog-ng@lists.balabit.hu Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
I don't have an answer to your actual question, however, you can work around it by
log { source(src_udp); parser(myparser); log { filter(filter_host1); destination(dst_host1); flags(final); }; log { filter(filter_host2); destination(dst_host2); flags(final); }; };
Hope that helps.
Evan.
On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
Hi
Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2):
parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); };
template mytemplate { template("${A};${B};${C}\n"); }
filter filter_host1 { netmask(10.0.0.1/255.255.255.255); };
filter filter_host2 { netmask(10.0.0.2/255.255.255.255); };
destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); };
destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); };
log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); };
log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); };
This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db- parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel
_______________________________________________________________________ _______
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Mikkel, On Fri, Apr 10, 2015 at 06:11:44AM +0000, Mikkel Leth Carlsen wrote:
Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting:
I wouldn't worry too much about patterndb's performance. That being said, feel free to try out the two approaches and report back :) Cheers
Hmm. I dont see what dbparser is trying to persist there. Now as I think of it, its probably the correlation state table. This seems to be a genuine bug. On Apr 8, 2015 1:31 PM, "Mikkel Leth Carlsen" <mlca@tdc.dk> wrote:
Hi
Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2):
parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); };
template mytemplate { template("${A};${B};${C}\n"); }
filter filter_host1 { netmask(10.0.0.1/255.255.255.255); };
filter filter_host2 { netmask(10.0.0.2/255.255.255.255); };
destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); };
destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); };
log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); };
log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); };
This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log:
Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)'
/Mikkel
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Evan Rempel
-
Fabien Wernli
-
Mikkel Leth Carlsen
-
Scheidler, Balázs