destination based on custom field question
Greetings, I'm new to syslog-ng, and I'm trying to key off of a custom field in a log entry, and put the message in a particular directory named for the field. a sample log entry would look like this: Dec 16 14:08:51 u910-05 testapp: \ 00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: \ | PartitionInfo [0x275f720] '\' denotes line continuation. The field I want to key off in this line is: 00000000000000a7 I created a filter to only get stuff from 'testapp', but now I want to make the destination be tied to the field. The field is positional, but can have any value, and cannot be known a priori. Not sure how to go about it. Any links to examples or where to start would be very much appreciated. -- Thanks, -Christopher
On Thu, Jan 06, 2011 at 06:01:13PM -0500, Christopher Barry wrote:
Greetings,
I'm new to syslog-ng, and I'm trying to key off of a custom field in a log entry, and put the message in a particular directory named for the field.
a sample log entry would look like this: Dec 16 14:08:51 u910-05 testapp: \ 00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: \ | PartitionInfo [0x275f720]
'\' denotes line continuation.
The field I want to key off in this line is: 00000000000000a7
I created a filter to only get stuff from 'testapp', but now I want to make the destination be tied to the field. The field is positional, but can have any value, and cannot be known a priori.
Not sure how to go about it. Any links to examples or where to start would be very much appreciated.
-- Thanks, -Christopher
This is certainly doable. What you need is the patterndb feature described here: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... You will create a series of rules which match your messages to the level of granularity desired, then you can capture variables from the message with names you select. Then you can use the variables anywhere 'downstream' from where you applied the patterndb matching, such as the output file template or rewrite rules, etc. You could also do this with PCRE capturing, but it would be less efficient. How many messages/sec. are you seeing and planning to support? Matthew.
Give this a shot for a patterndb.xml: <patterndb version="3"> <ruleset name="testapp"> <pattern>testapp</pattern> <rules> <rule id="x" class="x"> <patterns> <pattern>@ESTRING:field_name::@@ESTRING:field_name_2::@@ESTRING:level: @:@ESTRING:file_name: @</pattern> </patterns> <examples> <example> <test_message program="testapp">00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: PartitionInfo [0x275f720]</test_message> <test_values> <test_value name="field_name">00000000000000a7</test_value> <test_value name="field_name_2">00007f62d170a910</test_value> <test_value name="level">DEBUG</test_value> <test_value name="file_name">part.cpp</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> On Thu, Jan 6, 2011 at 6:52 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
On Thu, Jan 06, 2011 at 06:01:13PM -0500, Christopher Barry wrote:
Greetings,
I'm new to syslog-ng, and I'm trying to key off of a custom field in a log entry, and put the message in a particular directory named for the field.
a sample log entry would look like this: Dec 16 14:08:51 u910-05 testapp: \ 00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: \ | PartitionInfo [0x275f720]
'\' denotes line continuation.
The field I want to key off in this line is: 00000000000000a7
I created a filter to only get stuff from 'testapp', but now I want to make the destination be tied to the field. The field is positional, but can have any value, and cannot be known a priori.
Not sure how to go about it. Any links to examples or where to start would be very much appreciated.
-- Thanks, -Christopher
This is certainly doable.
What you need is the patterndb feature described here:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...
You will create a series of rules which match your messages to the level of granularity desired, then you can capture variables from the message with names you select. Then you can use the variables anywhere 'downstream' from where you applied the patterndb matching, such as the output file template or rewrite rules, etc.
You could also do this with PCRE capturing, but it would be less efficient. How many messages/sec. are you seeing and planning to support?
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Thanks Martin for this example - very much appreciated. -Christopher On Thu, 2011-01-06 at 20:44 -0600, Martin Holste wrote:
Give this a shot for a patterndb.xml:
<patterndb version="3"> <ruleset name="testapp"> <pattern>testapp</pattern> <rules> <rule id="x" class="x"> <patterns> <pattern>@ESTRING:field_name::@@ESTRING:field_name_2::@@ESTRING:level: @:@ESTRING:file_name: @</pattern> </patterns> <examples> <example> <test_message program="testapp">00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: PartitionInfo [0x275f720]</test_message> <test_values> <test_value name="field_name">00000000000000a7</test_value> <test_value name="field_name_2">00007f62d170a910</test_value> <test_value name="level">DEBUG</test_value> <test_value name="file_name">part.cpp</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb>
On Thu, Jan 6, 2011 at 6:52 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
On Thu, Jan 06, 2011 at 06:01:13PM -0500, Christopher Barry wrote:
Greetings,
I'm new to syslog-ng, and I'm trying to key off of a custom field in a log entry, and put the message in a particular directory named for the field.
a sample log entry would look like this: Dec 16 14:08:51 u910-05 testapp: \ 00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: \ | PartitionInfo [0x275f720]
'\' denotes line continuation.
The field I want to key off in this line is: 00000000000000a7
I created a filter to only get stuff from 'testapp', but now I want to make the destination be tied to the field. The field is positional, but can have any value, and cannot be known a priori.
Not sure how to go about it. Any links to examples or where to start would be very much appreciated.
-- Thanks, -Christopher
This is certainly doable.
What you need is the patterndb feature described here:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...
You will create a series of rules which match your messages to the level of granularity desired, then you can capture variables from the message with names you select. Then you can use the variables anywhere 'downstream' from where you applied the patterndb matching, such as the output file template or rewrite rules, etc.
You could also do this with PCRE capturing, but it would be less efficient. How many messages/sec. are you seeing and planning to support?
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2011-01-06 at 16:52 -0800, Matthew Hall wrote:
On Thu, Jan 06, 2011 at 06:01:13PM -0500, Christopher Barry wrote:
Greetings,
I'm new to syslog-ng, and I'm trying to key off of a custom field in a log entry, and put the message in a particular directory named for the field.
a sample log entry would look like this: Dec 16 14:08:51 u910-05 testapp: \ 00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: \ | PartitionInfo [0x275f720]
'\' denotes line continuation.
The field I want to key off in this line is: 00000000000000a7
I created a filter to only get stuff from 'testapp', but now I want to make the destination be tied to the field. The field is positional, but can have any value, and cannot be known a priori.
Not sure how to go about it. Any links to examples or where to start would be very much appreciated.
-- Thanks, -Christopher
This is certainly doable.
What you need is the patterndb feature described here:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...
You will create a series of rules which match your messages to the level of granularity desired, then you can capture variables from the message with names you select. Then you can use the variables anywhere 'downstream' from where you applied the patterndb matching, such as the output file template or rewrite rules, etc.
You could also do this with PCRE capturing, but it would be less efficient. How many messages/sec. are you seeing and planning to support?
Matthew.
Thanks Matthew, syslog-ng (2.0.9-4.2) is the packaged version in Ubuntu 10.04. Can I use this version to create the patterns that have been mentioned in this thread, or do I need to pull down a newer (3.x) .deb from balabit? If a newer one is appropriate or required, which version is recommended? -- Thanks a lot, -Christopher
On Fri, 2011-01-07 at 11:58 -0500, Christopher Barry wrote:
syslog-ng (2.0.9-4.2) is the packaged version in Ubuntu 10.04. Can I use this version to create the patterns that have been mentioned in this thread, or do I need to pull down a newer (3.x) .deb from balabit? If a newer one is appropriate or required, which version is recommended?
I would strongly recommend 3.x. Debian has 3.1.3, and that version will probably work just fine on Ubuntu 10.04 (haven't tried, you might need to recompile in the worst case). -- |8]
On Fri, Jan 07, 2011 at 08:18:00PM +0100, Gergely Nagy wrote:
On Fri, 2011-01-07 at 11:58 -0500, Christopher Barry wrote:
syslog-ng (2.0.9-4.2) is the packaged version in Ubuntu 10.04. Can I use this version to create the patterns that have been mentioned in this thread, or do I need to pull down a newer (3.x) .deb from balabit? If a newer one is appropriate or required, which version is recommended?
I would strongly recommend 3.x. Debian has 3.1.3, and that version will probably work just fine on Ubuntu 10.04 (haven't tried, you might need to recompile in the worst case).
I'm with him on this, but I'm stricter. I recommend using 3.2 because it's got the most reliable bug fixes and features and syntax, such as correlation between events. Matthew.
On Fri, Jan 07, 2011 at 11:37:05AM -0800, Matthew Hall wrote:
On Fri, Jan 07, 2011 at 08:18:00PM +0100, Gergely Nagy wrote:
On Fri, 2011-01-07 at 11:58 -0500, Christopher Barry wrote:
syslog-ng (2.0.9-4.2) is the packaged version in Ubuntu 10.04. Can I use this version to create the patterns that have been mentioned in this thread, or do I need to pull down a newer (3.x) .deb from balabit? If a newer one is appropriate or required, which version is recommended?
I would strongly recommend 3.x. Debian has 3.1.3, and that version will probably work just fine on Ubuntu 10.04 (haven't tried, you might need to recompile in the worst case).
I'm with him on this, but I'm stricter. I recommend using 3.2 because it's got the most reliable bug fixes and features and syntax, such as correlation between events.
Matthew.
Forgot to mention: There is a debian/ directory in the sources which should make it easy to create a deb. Simply install all the packages in debian/control Build-Depends and Depends, along with build-essential. Then run debian/rules binary. You should get a working package pretty easily after that. Matthew.
On Thu, 2011-01-06 at 16:52 -0800, Matthew Hall wrote:
On Thu, Jan 06, 2011 at 06:01:13PM -0500, Christopher Barry wrote:
Greetings,
I'm new to syslog-ng, and I'm trying to key off of a custom field in a log entry, and put the message in a particular directory named for the field.
a sample log entry would look like this: Dec 16 14:08:51 u910-05 testapp: \ 00000000000000a7:00007f62d170a910:DEBUG :part.cpp : 183: \ | PartitionInfo [0x275f720]
'\' denotes line continuation.
The field I want to key off in this line is: 00000000000000a7
I created a filter to only get stuff from 'testapp', but now I want to make the destination be tied to the field. The field is positional, but can have any value, and cannot be known a priori.
Not sure how to go about it. Any links to examples or where to start would be very much appreciated.
-- Thanks, -Christopher
This is certainly doable.
What you need is the patterndb feature described here:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...
You will create a series of rules which match your messages to the level of granularity desired, then you can capture variables from the message with names you select. Then you can use the variables anywhere 'downstream' from where you applied the patterndb matching, such as the output file template or rewrite rules, etc.
You could also do this with PCRE capturing, but it would be less efficient. How many messages/sec. are you seeing and planning to support?
If this stream can easily separated from the rest of your logs, then csv-parser with a custom separator might be somewhat easier to use, and still you can apply db-parser() filtering after the csv-parser has exploded the fields into name-value pairs. -- Bazsi
participants (5)
-
Balazs Scheidler
-
Christopher Barry
-
Gergely Nagy
-
Martin Holste
-
Matthew Hall