[syslog-ng] PatternDB for a10?

[Daniel Neubacher]

Thanks for the answer. I’m a fan of rock stable and fast software – logstash isn’t both. It’s a really neat logshipper and I’m still using it to get the syslog-ng json into our elasticsearch cluster but when grok has to process 25k messages and more per second the cpu usage goes up quite a bit. In my tests the server needed around 20-30% cpu to parse some logs and not even all of them completely. With the pattern db’s I found around the web the logs are perfectly parsed and I don’t even notice a rise in the cpu usage. Maybe 1-2% but this isn’t comparable to grok. Forging the json was part of logstash too and now it just takes thesyslog-ng json as input and put it into elasticsearch. For many people this isn’t relevant because grok is fine for a few hundred messages but beyond that it’s simply a hassle.
I really think you guys should make it more approachable for everyone because it’s awesome. I had to do a lot of digging, searching around and trial and error to get it running as a logstash competitor. Some sample cfg’s, a new commit in the patterndb git and a more detailed documentation could show many guys that no fancy cool new software is needed to get the job done better and faster on smaller hardware.

And I’m feeling really stupid about it but I don’t see how patternize can help me creating patterns. I don’t have a problem creating the patterns myself, it just takes some time :P
For example I have this log msg:
[HMON]<6> SLB server xas-1.app.fra1 (256.256.256.256) TCP port 80 of group app-xas is up (HTTP Expected Response Received).

So I made this pattern:
<pattern>[HMON]&lt;6&gt; SLB server @ESTRING:slb.server: @(@ESTRING:slb.server.ip:)@ TCP port @ESTRING:slb.service-group.port: @of group @ESTRING:slb.service-group: @is @ESTRING:slb.service-group-status: @(@ESTRING:slb.message:)@</pattern>
        </patterns>

How would patternize help me with this rule? I tried it and it’s nice to get the basic xml but changing the variables and editing the errors took me longer than writing it from the scratch. I guess I’m just not seeing it at the right angle.

Daniel

Von: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] Im Auftrag von Scheidler, Balázs
Gesendet: Donnerstag, 19. Februar 2015 13:35
An: Syslog-ng users' and developers' mailing list
Betreff: Re: [syslog-ng] PatternDB for a10?

Hi,
Happy to hear that you like patterndb. Can you pls tell me what makes it better than grok for you?
On the a10 front, you might want to try pdbtool patternize if you have a lot of logs, it can help you to define the patterns automatically. It is in fact an implementation of http://ristov.users.sourceforge.net/slct/.
Bazsi

--
Bazsi

On Wed, Feb 18, 2015 at 8:01 AM, Daniel Neubacher <daniel.neubacher at xing.com<mailto:daniel.neubacher at xing.com>> wrote:
Hey there,
I’ve really become a fan of patterndb and I’m migrating almost all of my grok filters to it. My next log devices to migrate are our a10 ax loadbalancer so I wanted to ask if anyone done this work before me :P

Daniel

--
Daniel Neubacher,  Senior Network Engineer
daniel.neubacher at xing.com<mailto:daniel.neubacher at xing.com>

XING AG
Dammtorstraße 30, 20354 Hamburg, Germany
Tel. +49 40 419131-28<tel:%2B49%2040%20419131-28>, Fax +49 40 419131-11<tel:%2B49%2040%20419131-11>

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807
Exec. Board (Vorstand): Dr. Thomas Vollmoeller (Vorsitzender), Ingo Chu, Jens Pape, Timm Richter
Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Stefan Winners

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150220/98009f91/attachment-0001.htm