Feature Request - patterndb match set
I have come across some odd lines that really can't be matched/parsed by the patterndb 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number. I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair. Failing that I would propose that a @SET@ parser. @SET:name:character set@ This will match a sequence of characters that contain any of, and only those characters listed by "character set" This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled. Comments? Evan
I agree. Since ANYSTRING does not work in the middle of a pattern, authors are left without an option for variable-length matches when you can't use (E|Q)STRING, such as an unknown number of repeating spaces. I think SET would be fairly efficient since it would behave a lot like a slightly modified version of ESTRING. On Sat, Nov 26, 2011 at 11:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
I have come across some odd lines that really can't be matched/parsed by the patterndb
2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf
I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number.
I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair.
Failing that I would propose that a @SET@ parser.
@SET:name:character set@
This will match a sequence of characters that contain any of, and only those characters listed by "character set"
This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled.
Comments?
Evan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Evan, On 11/27/2011 06:10 AM, Evan Rempel wrote:
I have come across some odd lines that really can't be matched/parsed by the patterndb
2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf
I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number.
I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair.
Failing that I would propose that a @SET@ parser.
@SET:name:character set@
This will match a sequence of characters that contain any of, and only those characters listed by "character set"
This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled.
Comments?
Evan This is something I would have needed recently as well, I ran across the same problem with squid logs and padded usernames. STRING is not okay, since you can only extend the set of matched chars, not specify them and it will match the following tokens as well. I never tried to do a parser before, but it seemed quite easy, so I'm sending a patch in a separate thread that implements your idea and let's see what Bazsi thinks about it.
Balint
Thanks Balint The patch was not quite complete (don't you hate copy and paste!) as it did not reference your new parser. A small fix, and it worked like a charm. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balint Kovacs [balint.kovacs@balabit.com] Sent: Sunday, November 27, 2011 9:47 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] Feature Request - patterndb match set Hi Evan, On 11/27/2011 06:10 AM, Evan Rempel wrote:
I have come across some odd lines that really can't be matched/parsed by the patterndb
2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf
I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number.
I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair.
Failing that I would propose that a @SET@ parser.
@SET:name:character set@
This will match a sequence of characters that contain any of, and only those characters listed by "character set"
This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled.
Comments?
Evan This is something I would have needed recently as well, I ran across the same problem with squid logs and padded usernames. STRING is not okay, since you can only extend the set of matched chars, not specify them and it will match the following tokens as well. I never tried to do a parser before, but it seemed quite easy, so I'm sending a patch in a separate thread that implements your idea and let's see what Bazsi thinks about it.
Balint ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Evan, Ah, I really shouldn't code half asleep ;) Fixed it and also done the unit tests. Thanks for the feedback! Balint On 11/28/2011 01:25 AM, Evan Rempel wrote:
Thanks Balint
The patch was not quite complete (don't you hate copy and paste!) as it did not reference your new parser. A small fix, and it worked like a charm.
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balint Kovacs [balint.kovacs@balabit.com] Sent: Sunday, November 27, 2011 9:47 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] Feature Request - patterndb match set
Hi Evan,
On 11/27/2011 06:10 AM, Evan Rempel wrote:
I have come across some odd lines that really can't be matched/parsed by the patterndb
2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf
I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number.
I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair.
Failing that I would propose that a @SET@ parser.
@SET:name:character set@
This will match a sequence of characters that contain any of, and only those characters listed by "character set"
This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled.
Comments?
Evan This is something I would have needed recently as well, I ran across the same problem with squid logs and padded usernames. STRING is not okay, since you can only extend the set of matched chars, not specify them and it will match the following tokens as well. I never tried to do a parser before, but it seemed quite easy, so I'm sending a patch in a separate thread that implements your idea and let's see what Bazsi thinks about it.
Balint
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
DDD ----- Original Message ----- From: Evan Rempel [mailto:erempel@uvic.ca] Sent: Sunday, November 27, 2011 07:25 PM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Feature Request - patterndb match set Thanks Balint The patch was not quite complete (don't you hate copy and paste!) as it did not reference your new parser. A small fix, and it worked like a charm. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balint Kovacs [balint.kovacs@balabit.com] Sent: Sunday, November 27, 2011 9:47 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] Feature Request - patterndb match set Hi Evan, On 11/27/2011 06:10 AM, Evan Rempel wrote:
I have come across some odd lines that really can't be matched/parsed by the patterndb
2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf
I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number.
I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair.
Failing that I would propose that a @SET@ parser.
@SET:name:character set@
This will match a sequence of characters that contain any of, and only those characters listed by "character set"
This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled.
Comments?
Evan This is something I would have needed recently as well, I ran across the same problem with squid logs and padded usernames. STRING is not okay, since you can only extend the set of matched chars, not specify them and it will match the following tokens as well. I never tried to do a parser before, but it seemed quite easy, so I'm sending a patch in a separate thread that implements your idea and let's see what Bazsi thinks about it.
Balint ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Sun, 2011-11-27 at 18:47 +0100, Balint Kovacs wrote:
Hi Evan,
On 11/27/2011 06:10 AM, Evan Rempel wrote:
I have come across some odd lines that really can't be matched/parsed by the patterndb
2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: Module Size Used by 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfs26 1945576 0 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: mmfslinux 326280 1 mmfs26 2011-11-25T10:49:21-08:00 mmfs@hermes0022.westgrid.uvic.ca/localhost/hermes0022/xcat2.westgrid.uvic.ca local2.info mmfs: tracedev 67148 2 mmf
I would like to match these and parse out the number. The catch is that the number is right justified which means that there is a variable number of spaces before the number.
I am open to suggestions about how to make a paterndb pattern to match this and parse the number into a tag/value pair.
Failing that I would propose that a @SET@ parser.
@SET:name:character set@
This will match a sequence of characters that contain any of, and only those characters listed by "character set"
This will allow matches of arbitrary length separators such as spaces or hyphens or other cases that can not yet be handled.
Comments?
Evan This is something I would have needed recently as well, I ran across the same problem with squid logs and padded usernames. STRING is not okay, since you can only extend the set of matched chars, not specify them and it will match the following tokens as well. I never tried to do a parser before, but it seemed quite easy, so I'm sending a patch in a separate thread that implements your idea and let's see what Bazsi thinks about it.
I've applied the @SET@ parser to 3.4 and also wrote a $(strip) template function to possibly take care of extra whitespace. That can be used in <value/> tags in the rule this way: <value name="usracct.name">$(strip ${value.parsed.by.the.pattern})</value> That's also pushed to 3.4. -- Bazsi
participants (5)
-
Balazs Scheidler
-
Balint Kovacs
-
Evan Rempel
-
Girish-Agarwal
-
Martin Holste