[syslog-ng] db-parser reuse for multiple logs?

[John Dyer]

Jim,




  Mind if I ask you what the specs are on that system ?




John






—
Sent from Mailbox

On Fri, Apr 10, 2015 at 7:53 AM, Jim Hendrick <jrhendri at roadrunner.com>
wrote:

>     
>  I think it would depend on the filter. If it uses something that is readily available then maybe filter first.  If is a match within the message, then possibly not. 
> The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. 
> Jim
> Sent from my Verizon Wireless 4G LTE smartphone
> -------- Original message --------
> From: Mikkel Leth Carlsen <mlca at tdc.dk> 
> Date: 04/10/2015  2:11 AM  (GMT-05:00) 
> To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu> 
> Subject: Re: [syslog-ng] db-parser reuse for multiple logs? 
> Hi Evan
> Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting:
> filter(filter_host1_or_host2);
> before the parser below? 
> /Mikkel
>> -----Oprindelig meddelelse-----
>> Fra: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-
>> bounces at lists.balabit.hu] På vegne af Evan Rempel
>> Sendt: 8. april 2015 17:38
>> Til: syslog-ng at lists.balabit.hu
>> Emne: Re: [syslog-ng] db-parser reuse for multiple logs?
>> 
>> I don't have an answer to your actual question, however, you can work
>> around it by
>> 
>> log {
>>          source(src_udp);
>>          parser(myparser);
>>          log {
>>                  filter(filter_host1);
>>                  destination(dst_host1);
>>                  flags(final);
>>          };
>>          log {
>>                  filter(filter_host2);
>>                  destination(dst_host2);
>>                  flags(final);
>>          };
>> };
>> 
>> Hope that helps.
>> 
>> Evan.
>> 
>> On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote:
>> > Hi
>> >
>> > Are db-parsers defined in syslog-ng configurations not reusable for
>> multiple logs?  A simplified example (syslog 3.6.2):
>> >
>> > parser myparser {
>> >          db_parser(
>> >                  file("/usr/local/etc/patterndb.d/myparser.xml")
>> >          );
>> > };
>> >
>> > template mytemplate {
>> >           template("${A};${B};${C}\n");
>> > }
>> >
>> > filter filter_host1 {
>> >          netmask(10.0.0.1/255.255.255.255);
>> > };
>> >
>> > filter filter_host2 {
>> >          netmask(10.0.0.2/255.255.255.255);
>> > };
>> >
>> > destination dst_host1 {
>> >          file("host1.log" perm(0644) template(mytemplate));
>> > };
>> >
>> >
>> > destination dst_host2 {
>> >          file("host2.log" perm(0644) template(mytemplate));
>> > };
>> >
>> > log {
>> >          source(src_udp);
>> >          filter(filter_host1);
>> >          parser(myparser);
>> >          destination(dst_host1);
>> >          flags(final);
>> > };
>> >
>> > log {
>> >          source(src_udp);
>> >          filter(filter_host2);
>> >          parser(myparser);
>> >          destination(dst_host2);
>> >          flags(final);
>> > };
>> >
>> > This seems to work as expected and 'syslog-ng -s' does not report any
>> problems, but I see the following in the syslog-ng internal log:
>> >
>> > Internal error, duplicate configuration elements refer to the same
>> persistent config; name='db-
>> parser(/usr/local/etc/patterndb.d/myparser.xml)'
>> > Internal error, duplicate configuration elements refer to the same
>> persistent config; name='db-
>> parser(/usr/local/etc/patterndb.d/myparser.xml)'
>> >
>> > /Mikkel
>> >
>> _______________________________________________________________________
>> _______
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >
>> _______________________________________________________________________
>> _______
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150410/e0523135/attachment-0001.htm