<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;"><DIV>Thanks Mr Holste, your mail was very usefull.</DIV>
<DIV>I am tenderfoot in log parsing. I must extract several field for one message such as IP (or hostname), user, port, date, protocol, and other fileds if can extract. then fields must be normalized in IDMEF format. it must be done for every different syslog message types. </DIV>
<DIV>So, is the syslog-ng suitable tool for this task? and how?</DIV>
<DIV><BR><BR>--- On <B>Sat, 14/8/10, Martin Holste <I><mcholste@gmail.com></I></B> wrote:<BR></DIV>
<BLOCKQUOTE style="BORDER-LEFT: rgb(16,16,255) 2px solid; PADDING-LEFT: 5px; MARGIN-LEFT: 5px"><BR>From: Martin Holste <mcholste@gmail.com><BR>Subject: Re: [syslog-ng] Pattern extraction<BR>To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu><BR>Date: Saturday, 14 August, 2010, 7:32 PM<BR><BR>
<DIV class=plainMail>If you're looking to do never-wrong, full normalization, then yes,<BR>you're looking at thousands of signatures. However, if you're looking<BR>to extract some common fields, it's actually not that much work to<BR>grab things like IP addresses using regexp. Since regexp is slow, I'm<BR>thinking about writing some generic patterns that would match on IP's<BR>using the fast pattern matcher. I don't know if it'll work, but it<BR>would look like "@ANYSTRING@@IPv4@@ANYSTRING@" and then maybe another<BR>one to grep out two IP's, then another for three, etc. I have no idea<BR>if that will work; we'll see how it goes.<BR><BR>I think that the pursuit of perfection in this field will be<BR>discouraging, and may stifle efforts before they begin. I urge you to<BR>take it one pattern at a time. Sure, we may need thousands of<BR>patterns, but there are hundreds if not thousands on this mailing<BR>list.
Everybody take two patterns ;) And don't forget that the<BR>patternize tool may be able to help by heuristically identifying<BR>fields in messages. Then it just comes down to a human naming the<BR>fields instead of painstakingly writing the patterns themselves.<BR><BR>Something else to consider: Even if you're only extracting the RFC<BR>headers of the syslog but you have full-text search abilities of the<BR>log messages, you can make some OLAP-style basic dimensional analysis<BR>happen. So, let's say you're going through router logs looking for an<BR>OSPF adjacency change. You search for "LOADING to FULL" and then<BR>group by host. You've just magically discovered all of the routers<BR>that flapped during whatever incident caused the adjacency change.<BR>Obviously this is very basic, but don't underestimate the immediate<BR>value of being able to quickly pinpoint which hosts had which events<BR>occur. I would
say that 70% of the total value you'd get from having<BR>all messages perfectly parsed is already attained just by being able<BR>to do free text searches and group by host.<BR><BR>Lastly, not all logs are created equal! I wrote parsers for Cisco<BR>firewall connection teardowns and firewall denies, and now more than<BR>half of my logs are neatly parsed. That's because the vast majority<BR>of Cisco logs at notification level are build/teardown messages.<BR>(Something like four logs per flow per device). Now if I'm looking<BR>for something weird, I can easily take the majority of the hay out of<BR>the haystack by excluding the already classified logs in my search.<BR>It even helps with reporting, because a big jump in the number of<BR>unclassified messages shows up on the radar.<BR><BR>So to sum up, the benefit of creating log patterns is exponential.<BR>Not having a pattern for every possible log isn't really a big deal,<BR>but having
patterns for certain logs is.<BR><BR>On Fri, Aug 13, 2010 at 8:00 PM, Anton Chuvakin <<A href="http://aa.mc631.mail.yahoo.com/mc/compose?to=anton@chuvakin.org" ymailto="mailto:anton@chuvakin.org">anton@chuvakin.org</A>> wrote:<BR>>> So, I must extract hundreds of pattern manually. :(<BR>><BR>> Not really hundreds, try tens of thousands. If you sit and watch a<BR>> busy syslog server for, say, 5 years, some say you'd see a few<BR>> thousand or more of unique messages. Personally, I have not tried it,<BR>> but I trust the source.<BR>><BR>><BR>>> Regards<BR>>><BR>>> --- On Fri, 13/8/10, Anton Chuvakin <<A href="http://aa.mc631.mail.yahoo.com/mc/compose?to=anton@chuvakin.org" ymailto="mailto:anton@chuvakin.org">anton@chuvakin.org</A>> wrote:<BR>>><BR>>> From: Anton Chuvakin <<A href="http://aa.mc631.mail.yahoo.com/mc/compose?to=anton@chuvakin.org"
ymailto="mailto:anton@chuvakin.org">anton@chuvakin.org</A>><BR>>> Subject: Re: [syslog-ng] Pattern extraction<BR>>> To: "Syslog-ng users' and developers' mailing list" <<A href="http://aa.mc631.mail.yahoo.com/mc/compose?to=syslog-ng@lists.balabit.hu" ymailto="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</A>><BR>>> Date: Friday, 13 August, 2010, 7:18 PM<BR>>><BR>>> > I dont know how can i extract pattern form logs, I must check every log type separately?, using pattern recognition methods? or using<BR>>> >pattern database (if exist for all aplication and device)?<BR>>><BR>>> Well, this is not just you - it is "you and the rest of the world."<BR>>> The standard way is pretty much to manually (or with tools - but still<BR>>> mostly manually) write regular expressions for every distinct log<BR>>> message type.<BR>>><BR>>>
--<BR>>> Dr. Anton Chuvakin<BR>>> Site: <A href="http://www.chuvakin.org/" target=_blank>http://www.chuvakin.org</A><BR>>> Blog: <A href="http://www.securitywarrior.org/" target=_blank>http://www.securitywarrior.org</A><BR>>> LinkedIn: <A href="http://www.linkedin.com/in/chuvakin" target=_blank>http://www.linkedin.com/in/chuvakin</A><BR>>> Consulting: <A href="http://www.securitywarriorconsulting.com/" target=_blank>http://www.securitywarriorconsulting.com</A><BR>>> Twitter: @anton_chuvakin<BR>>> Google Voice: +1-510-771-7106<BR>>> ______________________________________________________________________________<BR>>> Member info: <A href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target=_blank>https://lists.balabit.hu/mailman/listinfo/syslog-ng</A><BR>>> Documentation: <A href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target=_blank>http://www.balabit.com/support/documentation/?product=syslog-ng</A><BR>>> FAQ: <A href="http://www.campin.net/syslog-ng/faq.html" target=_blank>http://www.campin.net/syslog-ng/faq.html</A><BR>>><BR>>><BR>>><BR>>> ______________________________________________________________________________<BR>>> Member info: <A href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target=_blank>https://lists.balabit.hu/mailman/listinfo/syslog-ng</A><BR>>> Documentation: <A href="http://www.balabit.com/support/documentation/?product=syslog-ng" target=_blank>http://www.balabit.com/support/documentation/?product=syslog-ng</A><BR>>> FAQ: <A href="http://www.campin.net/syslog-ng/faq.html" target=_blank>http://www.campin.net/syslog-ng/faq.html</A><BR>>><BR>>><BR>><BR>><BR>><BR>> --<BR>> Dr. Anton Chuvakin<BR>> Site: <A href="http://www.chuvakin.org/"
target=_blank>http://www.chuvakin.org</A><BR>> Blog: <A href="http://www.securitywarrior.org/" target=_blank>http://www.securitywarrior.org</A><BR>> LinkedIn: <A href="http://www.linkedin.com/in/chuvakin" target=_blank>http://www.linkedin.com/in/chuvakin</A><BR>> Consulting: <A href="http://www.securitywarriorconsulting.com/" target=_blank>http://www.securitywarriorconsulting.com</A><BR>> Twitter: @anton_chuvakin<BR>> Google Voice: +1-510-771-7106<BR>> ______________________________________________________________________________<BR>> Member info: <A href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target=_blank>https://lists.balabit.hu/mailman/listinfo/syslog-ng</A><BR>> Documentation: <A href="http://www.balabit.com/support/documentation/?product=syslog-ng" target=_blank>http://www.balabit.com/support/documentation/?product=syslog-ng</A><BR>> FAQ: <A href="http://www.campin.net/syslog-ng/faq.html"
target=_blank>http://www.campin.net/syslog-ng/faq.html</A><BR>><BR>><BR>______________________________________________________________________________<BR>Member info: <A href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target=_blank>https://lists.balabit.hu/mailman/listinfo/syslog-ng</A><BR>Documentation: <A href="http://www.balabit.com/support/documentation/?product=syslog-ng" target=_blank>http://www.balabit.com/support/documentation/?product=syslog-ng</A><BR>FAQ: <A href="http://www.campin.net/syslog-ng/faq.html" target=_blank>http://www.campin.net/syslog-ng/faq.html</A><BR><BR></DIV></BLOCKQUOTE></td></tr></table><br>