[syslog-ng] Pattern extraction

[majid as]

Hi 
Thanks lot. your email was very usefull. 
I have also general problem(not only syslog-ng), if you know, I want classify and extract log fields, lags can be syslog, snmp trap , ... Then normalize logs in IDMEF standard format.
I dont know how can i extract pattern form logs, I must check every log type separately?, using pattern recognition methods? or using pattern database (if exist for all aplication and device)? 
 
Thanks
regards 
 
Majid
 

--- On Fri, 13/8/10, Robert Fekete <frobert at balabit.com> wrote:


From: Robert Fekete <frobert at balabit.com>
Subject: Re: [syslog-ng] Pattern extraction
To: "Syslog-ng users' and developers' mailing list" <syslog-ng at lists.balabit.hu>
Date: Friday, 13 August, 2010, 1:14 PM


Hi,

The syslog-ng pattern database is capable of extracting fields and classify log 
messages, and with well-structured name-value pairs you can achieve log 
normalization as well. However, currently there are not many well-written and 
tagged patterns available, so probably you'll have to create your own patterns.

You can find some sample patterns and a preliminary schema at the following git 
repository: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=summary
and some other, less-detailed patterns at 
http://www.balabit.com/downloads/files/patterndb-snapshot/

You might also want to check Bazsi's blog (http://bazsi.blogs.balabit.com), it 
has a number of interesting posts about patterndb, and of course the syslog-ng 
adminguide, in particular: 
http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/concepts_pattern_databases.html 
  and
http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/reference_parsers_pattern_databases.html

Correlation has to be done with an external application based on the tags/fields 
you assign to your log messages - maybe others already using patterndb can help 
you with the details.

Regards,

Robert

majid as wrote:

> Hi
> Thanks for replying and file.
> I work on network management project(Correlation of logs), my big problem is log classification and extract log field(normalization of logs). Do you have any idea for it? 
> 
> --- On Thu, 12/8/10, Robert Fekete <frobert at balabit.com> wrote:
> 
> 
> From: Robert Fekete <frobert at balabit.com>
> Subject: Re: [syslog-ng] Pattern extraction
> To: "Syslog-ng users' and developers' mailing list" <syslog-ng at lists.balabit.hu>
> Date: Thursday, 12 August, 2010, 4:19 PM
> 
> 
> majid as wrote:
> 
>> Hi
>> I have problem with pattern extraction from syslog messages. can anyone help me how extract patterns?
> 
> 
> Hi,
> I assume you are trying to use the pattern database (db_parser()). My collegue, 
> Peter Holtzl has written a tutorial about it that you might find useful: 
> http://www.balabit.com/dl/white_papers/syslog-ng-v3.1-whitepaper-message-classification-en.pdf
> 
> Otherwise, please let us know exactly what you are trying to do, how, and what 
> the problem is so we can help you.
> 
> Regards,
> 
> Robert
> 
>>   
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100813/73c399a4/attachment.htm