few questions about patterndb
Hi folks, i'm currently testing the patterndb with syslog-ng. After a few test runs,there are some open questions. Will there be some future releases of the patterndb by balabit? Last Snapshot has a timestamp from sep. 2009. Or is there a community somewhere out there, that i haven't seen yet?:-) I tested the classification into different logfiles (system,violation,unknown) with 1.5M lines of logfiles, and there was just one rule matching 'em all: 734adfc8-8a4d-40ad-923d-9d3fbabda34b, which is something about a newly connected usb mass storage device. This behaviour can't be correct? The patternfile "system-apache2.xml" still containts just plain messages, no XML anywhere - i thougt this should have been fixed yet? After some testing with logging messages to file like "messages-system/messages-violation/messages-unknown" i'm just asking myself about the profit i get when using patterndb. With a standard syslog-ng i usually can log things into files divided by the priority of the messages - that's all i want it to do, just have a logfile where all errors und failures are logged, just to get the pure number of lines down. When controlling a logfile i usually don't really care about a cronjob to be successful. What is the profit of using patterndb right there? Thanks in advance, i.A. Hendrik Pahl System Engineering team! datentechnik GmbH & Co.KG Werner von Siemens Straße 12a 49124 Georgsmarienhuette Tel.: +49 (0)5401-8226-50 Fax : +49 (0)5401-8226-55 E-Mail: pahl@team-datentechnik.de Internet: www.team-datentechnik.de HRA 110397, Amtsgericht Osnabrück Geschäftsführung: Reemt Lükenga Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Vielen Dank. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Currently, there's not a community repository for pattern-db, but I believe Balabit is hoping to have one. That said, it does not sound like you need to use it for what you're trying to do. The value of pattern-db is in fine-tuned classification and also parsing fields out of the bodies of messages. For instance, here's a pattern I use to parse the basic fields in a Snort IDS log message: <ruleset name="snort" id='8'> <pattern>snort</pattern> <rules> <rule provider="local" class='8' id='8'> <patterns> <pattern>@QSTRING:sig_sid:[]@ @ESTRING:sig_msg:[@Classification:@QSTRING:sig_classification: ]@ [Priority: @NUMBER:sig_priority:@]: @QSTRING:proto:{}@ @IPv4:srcip:@:@NUMBER:srcport:@ -> @IPv4:dstip:@:@NUMBER:dstport:@</pattern> </patterns> </rule> </rules> </ruleset> This will take effect on messages with a program matching "snort", assign some meta data (class=8 and id=8) and then extract fields from the message text. I can then put this directly into a database with each field as a column for reporting, etc. This is a pretty drastic rule. Most of the example rules provided by Balabit are demonstrating that messages can be classified based on the program name and assigned tags, etc. which is basically providing another way of attaching unlimited meta data as the logs come in. I'm using them for parsing because pattern-db is much more efficient than using regular expressions, so it is necessary for high-volume parsing. --Martin On Wed, Jun 23, 2010 at 1:46 AM, Hendrik Pahl <pahl@team-datentechnik.de> wrote:
Hi folks,
i'm currently testing the patterndb with syslog-ng. After a few test runs,there are some open questions.
Will there be some future releases of the patterndb by balabit? Last Snapshot has a timestamp from sep. 2009. Or is there a community somewhere out there, that i haven't seen yet?:-)
I tested the classification into different logfiles (system,violation,unknown) with 1.5M lines of logfiles, and there was just one rule matching 'em all: 734adfc8-8a4d-40ad-923d-9d3fbabda34b, which is something about a newly connected usb mass storage device. This behaviour can't be correct?
The patternfile "system-apache2.xml" still containts just plain messages, no XML anywhere - i thougt this should have been fixed yet?
After some testing with logging messages to file like "messages-system/messages-violation/messages-unknown" i'm just asking myself about the profit i get when using patterndb. With a standard syslog-ng i usually can log things into files divided by the priority of the messages - that's all i want it to do, just have a logfile where all errors und failures are logged, just to get the pure number of lines down. When controlling a logfile i usually don't really care about a cronjob to be successful. What is the profit of using patterndb right there?
Thanks in advance,
i.A. Hendrik Pahl System Engineering
team! datentechnik GmbH & Co.KG Werner von Siemens Straße 12a 49124 Georgsmarienhuette Tel.: +49 (0)5401-8226-50 Fax : +49 (0)5401-8226-55
E-Mail: pahl@team-datentechnik.de Internet: www.team-datentechnik.de HRA 110397, Amtsgericht Osnabrück Geschäftsführung: Reemt Lükenga
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Vielen Dank.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi, ...
That said, it does not soundlike you need to use it for what you're trying to do.
Okay, i already had the feeling patterndb was not the one really giving me a solution. I simply need something to bring down the relevant loglines, since 1.5M lines/month in a logfile/different logfiles are simply much to much to monitor/read. Grepping after "error" or "warning" or "failure" is just one approach, but never will be the only one, since this might kick out things i wanna definitely see. currently i'm looking at logfiles and size down the amount of lines by piping the cat output into sed, which kicks out the informational and overhead lines. this ia an iterative apporach, since i refine the sed expression time to time. How are others managing this issue? i.A. Hendrik Pahl System Engineering team! datentechnik GmbH & Co.KG Werner von Siemens Straße 12a 49124 Georgsmarienhuette Tel.: +49 (0)5401-8226-50 Fax : +49 (0)5401-8226-55 E-Mail: pahl@team-datentechnik.de Internet: www.team-datentechnik.de HRA 110397, Amtsgericht Osnabrück Geschäftsführung: Reemt Lükenga Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Vielen Dank. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Sounds like a Job for LogZilla :-) (shameless plug - my tool) http://nms.gdd.net/index.php/LogZilla You can also use SEC (simple event correllator) to decide which messages to log. ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Thu, Jun 24, 2010 at 2:34 AM, Hendrik Pahl <pahl@team-datentechnik.de> wrote:
Hi,
...
That said, it does not soundlike you need to use it for what you're trying to do.
Okay, i already had the feeling patterndb was not the one really giving me a solution. I simply need something to bring down the relevant loglines, since 1.5M lines/month in a logfile/different logfiles are simply much to much to monitor/read.
Grepping after "error" or "warning" or "failure" is just one approach, but never will be the only one, since this might kick out things i wanna definitely see.
currently i'm looking at logfiles and size down the amount of lines by piping the cat output into sed, which kicks out the informational and overhead lines. this ia an iterative apporach, since i refine the sed expression time to time.
How are others managing this issue?
i.A. Hendrik Pahl System Engineering
team! datentechnik GmbH & Co.KG Werner von Siemens Straße 12a 49124 Georgsmarienhuette Tel.: +49 (0)5401-8226-50 Fax : +49 (0)5401-8226-55
E-Mail: pahl@team-datentechnik.de Internet: www.team-datentechnik.de HRA 110397, Amtsgericht Osnabrück Geschäftsführung: Reemt Lükenga
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Vielen Dank.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you.
Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, Jun 23, 2010 at 11:34 PM, Hendrik Pahl <pahl@team-datentechnik.de>wrote:
Hi,
...
That said, it does not soundlike you need to use it for what you're trying to do.
Okay, i already had the feeling patterndb was not the one really giving me a solution. I simply need something to bring down the relevant loglines, since 1.5M lines/month in a logfile/different logfiles are simply much to much to monitor/read.
Grepping after "error" or "warning" or "failure" is just one approach, but never will be the only one, since this might kick out things i wanna definitely see.
currently i'm looking at logfiles and size down the amount of lines by piping the cat output into sed, which kicks out the informational and overhead lines. this ia an iterative apporach, since i refine the sed expression time to time.
How are others managing this issue?
Hi, So by the sounds of it, you're looking to match certain lines based on regex, and then you want to read the surrounding lines of your match. Is this accurate? This is *always* going to be tedious without the help of some software. Syslog-ng will get you to the point where you can match whatever you want in the log line contents and then write that out to a file, ignoring everything that doesn't get matched. You can do this with either filters{} or patterndb. If you need alerts based on matching, syslog-ng can do this do if you output matches to alerting scripts; however, a log file analyzer may be better suited. Simple Event Correlator (SEC) is a popular tool, but there are plenty of log security tools that you can use that are easy to write matches for. OSSEC is a very good tool which is easy to configure, it's just noisy until you customize it. As for your problem where you are wanting to see the log lines above and below the line(s) you are grepping/regexing for, you're going to have to look into flexible log viewing applications such as Splunk, or if your log volume isn't insane, some sort of syslog frontend like LogZilla. The problem with Splunk though? Money. Splunk is so expensive because it's the only thing out there that does everything log related well - lots of people have the same problem as you except in some cases need searchable access to gigabytes or terabytes worth of logs. You can look for free alternatives, but they will all be using an sql-based database to store logs and you will most likely end up finding too many tools that are trying to be security event alerting packages rather than log viewing apps. You should also consider why you are needing to parse through so many logs manually, what is generating that volume, and whether you can make the log messages the application outputs more relevant. If things are constantly needing your attention to the extent that you cannot make enough matching rules to alert on, or the error-severity messages aren't including the actual problem which is forcing you to read what happened before it, then they are broken to begin with and should be fixed at the application. Regards, -- Lance Laursen Demonware Systems Engineer
On Wed, Jun 23, 2010 at 11:34 PM, Hendrik Pahl <pahl@team-datentechnik.de>wrote:
Hi,
...
That said, it does not soundlike you need to use it for what you're trying to do.
Okay, i already had the feeling patterndb was not the one really giving me a solution. I simply need something to bring down the relevant loglines, since 1.5M lines/month in a logfile/different logfiles are simply much to much to monitor/read.
Grepping after "error" or "warning" or "failure" is just one approach, but never will be the only one, since this might kick out things i wanna definitely see.
currently i'm looking at logfiles and size down the amount of lines by piping the cat output into sed, which kicks out the informational and overhead lines. this ia an iterative apporach, since i refine the sed expression time to time.
How are others managing this issue?
Also, aside from the essay I just wrote :), take a look at http://crunchtools.com/software/petit/ . It should be very useful for any manual log parsing. -- Lance Laursen Demonware Systems Engineer
Hey Lance, I'll have you know the new version of LogZilla can process 20k messages per second :-) Which isn't syslog-ng good, but pretty good nonetheless ;-) ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Thu, Jun 24, 2010 at 3:34 PM, Lance Laursen <lance@demonware.net> wrote:
On Wed, Jun 23, 2010 at 11:34 PM, Hendrik Pahl <pahl@team-datentechnik.de> wrote:
Hi,
...
That said, it does not soundlike you need to use it for what you're trying to do.
Okay, i already had the feeling patterndb was not the one really giving me a solution. I simply need something to bring down the relevant loglines, since 1.5M lines/month in a logfile/different logfiles are simply much to much to monitor/read.
Grepping after "error" or "warning" or "failure" is just one approach, but never will be the only one, since this might kick out things i wanna definitely see.
currently i'm looking at logfiles and size down the amount of lines by piping the cat output into sed, which kicks out the informational and overhead lines. this ia an iterative apporach, since i refine the sed expression time to time.
How are others managing this issue?
Also, aside from the essay I just wrote :), take a look at http://crunchtools.com/software/petit/ . It should be very useful for any manual log parsing.
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, Jun 24, 2010 at 12:43 PM, Clayton Dukes <cdukes@gmail.com> wrote:
Hey Lance, I'll have you know the new version of LogZilla can process 20k messages per second :-)
Which isn't syslog-ng good, but pretty good nonetheless ;-)
You bet, I didn't mean offense :). LogZilla is definitely one of the best free options out there -- Lance Laursen Demonware Systems Engineer
Hi, On Wed, 2010-06-23 at 18:14 -0500, Martin Holste wrote:
Currently, there's not a community repository for pattern-db, but I believe Balabit is hoping to have one. That said, it does not sound like you need to use it for what you're trying to do. The value of pattern-db is in fine-tuned classification and also parsing fields out of the bodies of messages. For instance, here's a pattern I use to parse the basic fields in a Snort IDS log message:
There's progress on our patterndb front, but that seems to be slow, as we originally planned to come forward with a shiny web interface. However, I'm starting to think that simply creating a "best-practice" policy document and putting user-contributed patterns into a version controlled directory would give us tremendous value, even without the web interface. So this is what I'm going to do: * draft this patterndb policy document * create a git repository * create a daily snapshot of the set of "verified" patterns * ask anyone who has patterns to contribute their patterns (we do too) The policy document would be an important part of that, since a consistent naming policy would be very important to create a maintainable database. -- Bazsi
participants (5)
-
Balazs Scheidler
-
Clayton Dukes
-
Hendrik Pahl
-
Lance Laursen
-
Martin Holste