Hi, I've figured it is not nice from me that I keep pushing stuff to "master" without getting proper review from the list. So here's a attempt to change that. I've just pushed an experimental patch to the "syslog-parser" branch on github that allows the syslog parsing functionality to be used as a syslog-ng parser: https://github.com/balabit/syslog-ng-3.4/tree/syslog-parser Here's a short description on why this is useful (quoting the commit message): This patch creates a new parser, to explicitly parse messages as syslog messages. This can be used to selectively parse these messages based on some filters. Imagine this use-case: log { source { udp(port(2000) flags(no-parse)); }; parser { log { junction { log { filter { netmask("127.0.0.1/32"); }; parser { syslog-parser(); }; flags(final); }; log { filter { netmask("127.0.0.2/32"); }; parser { csv-parser(columns("C1", "C2", "C3")); }; flags(final); }; }; }; }; destination { file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n")); }; }; Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2 as a csv-parser() style messages. I'd welcome any kind of feedback, code or functionality wise. I still have some stuff to fix in this, but after initial feedback I'm going to merge it to master. Thanks in advance. -- Bazsi
Hi, I've finished up the final touches and merged this to master. On Wed, 2012-10-24 at 21:05 +0200, Balazs Scheidler wrote:
Hi,
I've figured it is not nice from me that I keep pushing stuff to "master" without getting proper review from the list. So here's a attempt to change that.
I've just pushed an experimental patch to the "syslog-parser" branch on github that allows the syslog parsing functionality to be used as a syslog-ng parser:
https://github.com/balabit/syslog-ng-3.4/tree/syslog-parser
Here's a short description on why this is useful (quoting the commit message):
This patch creates a new parser, to explicitly parse messages as syslog messages. This can be used to selectively parse these messages based on some filters.
Imagine this use-case:
log { source { udp(port(2000) flags(no-parse)); }; parser { log { junction { log { filter { netmask("127.0.0.1/32"); }; parser { syslog-parser(); }; flags(final); }; log { filter { netmask("127.0.0.2/32"); }; parser { csv-parser(columns("C1", "C2", "C3")); }; flags(final); }; }; };
}; destination { file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n")); }; };
Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2 as a csv-parser() style messages.
I'd welcome any kind of feedback, code or functionality wise. I still have some stuff to fix in this, but after initial feedback I'm going to merge it to master.
Thanks in advance.
-- Bazsi
I am having a problem with the dbparser when messages have identical leading portions. In my case I have two messages xlog: backup pg_xlog/000000010000014700000076 xlog: backup pg_xlog/000000010000014700000076 failed The first of these two messages is a success, and should be safely ignored. The second of these two messages is a failure, and I want to tag it and route it to a ticket creation program so that our Operations Center can investigate. Patterns are; xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed If these patterns are placed into the pattern database in this order, then the first pattern will match the failed log message, which is completely wrong. I would ignore the message rather than route it to my ticketing system. If I change the order of them, then all works correctly. I don't think that "hand tuning" the xml file is the correct approach, and in my case can not actually be done. My patterndb.xml file is created by a program that reads all of the patterns and tests messages from an external database. I've worked around this by sorting the patterns descending by the length of the patterns, but if I had the two patterns; xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@ESTRING:: @failed then that "trick" does not work either. As I see it, the dbparser should match complete messages only. That would make the order completely irrelevant. Incidentally, I have the same problem with the "program" matching that the dbparser users. Take the mimedefang program as an example. There are multiple components with program names of; mimedefang mimedefang-multiplexor mimedefang.pl but the program mimedefang will match them all. This means that the pattern database file must either; 1. Have all of the message patterns under mimedefang 2. have the program sections sorted in reverse order so that the shortest program matches last. Again, I think that the pattern database should match the complete program name. Using multiple patterns for the program name is technically sufficient, but it would be nice to use a regular expression so that I can match imapd imapds with the program impads? As usual, all comments welcome, especially those that explain the history of why it was done this way and how the issue can be addressed. Thanks to everyone for such an active community. Evan.
I am confused by the silence on this issue. Should I open this as a bug, or is this the expected behaviour? If this is the expected behaviour, how do other work around this issue? Thanks, On 10/31/2012 09:01 AM, Evan Rempel wrote:
I am having a problem with the dbparser when messages have identical leading portions.
In my case I have two messages
xlog: backup pg_xlog/000000010000014700000076 xlog: backup pg_xlog/000000010000014700000076 failed
The first of these two messages is a success, and should be safely ignored. The second of these two messages is a failure, and I want to tag it and route it to a ticket creation program so that our Operations Center can investigate.
Patterns are;
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed
If these patterns are placed into the pattern database in this order, then the first pattern will match the failed log message, which is completely wrong. I would ignore the message rather than route it to my ticketing system.
If I change the order of them, then all works correctly.
I don't think that "hand tuning" the xml file is the correct approach, and in my case can not actually be done. My patterndb.xml file is created by a program that reads all of the patterns and tests messages from an external database.
I've worked around this by sorting the patterns descending by the length of the patterns, but if I had the two patterns;
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@ESTRING:: @failed
then that "trick" does not work either.
As I see it, the dbparser should match complete messages only. That would make the order completely irrelevant.
Incidentally, I have the same problem with the "program" matching that the dbparser users.
Take the mimedefang program as an example. There are multiple components with program names of;
mimedefang mimedefang-multiplexor mimedefang.pl
but the program mimedefang will match them all. This means that the pattern database file must either;
1. Have all of the message patterns under mimedefang 2. have the program sections sorted in reverse order so that the shortest program matches last.
Again, I think that the pattern database should match the complete program name. Using multiple patterns for the program name is technically sufficient, but it would be nice to use a regular expression so that I can match
imapd imapds
with the program impads?
As usual, all comments welcome, especially those that explain the history of why it was done this way and how the issue can be addressed.
Thanks to everyone for such an active community.
Evan.
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Unix Services, University Systems, University of Victoria
On Thu, Nov 1, 2012 at 8:55 PM, Evan Rempel <erempel@uvic.ca> wrote:
I am confused by the silence on this issue.
We have a long weekend in hungary, that may be the reason for the delay in answering.
Should I open this as a bug, or is this the expected behaviour?
Personally, I'd say this is not the desired behaviour. Perhaps it was done this way initially for one reason or the other, but that doesn't mean it should stay that way. I'd say, open a bug. I'll be poking around in dbparser next week, so I'll check if I can do something about it. -- |8]
Hi Evan, I would say, this is partially intended. DBParser will alway do prefix matching and this is a good thing, as you do not need to describe the whole message. This way it is less prone to message format changes, if anything is appended to the end of the message, the pattern will still match and many times only the leading portion of the message needs to be parsed. I'm actively using this assumption in my patterns and others are probably doing the same, so changing this behaviour would result in many patterns not matching anymore. Probably a parser marking the end of message would be a good solution. That's a bit out of the logic of patterndb but if memory serves me right it won't be too hard to do. A possible (completely untested) workaround would be to use the PCRE parser in 3.4 HEAD and use $ to match the end of string. I'm not sure why the order matters, others knowing the dbparser codebase better probably have an answer off the top of their heads. I would assume that dbparser should always do a best match and not linear evaluation, so I'd say this is not intended and is worth a bugreport. The above is also valid for the program pattern, however, you can use parsers in that pattern as well, and the SET parser should be more performant than a regex. Balint On 10/31/2012 05:01 PM, Evan Rempel wrote:
I am having a problem with the dbparser when messages have identical leading portions.
In my case I have two messages
xlog: backup pg_xlog/000000010000014700000076 xlog: backup pg_xlog/000000010000014700000076 failed
The first of these two messages is a success, and should be safely ignored. The second of these two messages is a failure, and I want to tag it and route it to a ticket creation program so that our Operations Center can investigate.
Patterns are;
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed
If these patterns are placed into the pattern database in this order, then the first pattern will match the failed log message, which is completely wrong. I would ignore the message rather than route it to my ticketing system.
If I change the order of them, then all works correctly.
I don't think that "hand tuning" the xml file is the correct approach, and in my case can not actually be done. My patterndb.xml file is created by a program that reads all of the patterns and tests messages from an external database.
I've worked around this by sorting the patterns descending by the length of the patterns, but if I had the two patterns;
xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@ESTRING:: @failed
then that "trick" does not work either.
As I see it, the dbparser should match complete messages only. That would make the order completely irrelevant.
Incidentally, I have the same problem with the "program" matching that the dbparser users.
Take the mimedefang program as an example. There are multiple components with program names of;
mimedefang mimedefang-multiplexor mimedefang.pl
but the program mimedefang will match them all. This means that the pattern database file must either;
1. Have all of the message patterns under mimedefang 2. have the program sections sorted in reverse order so that the shortest program matches last.
Again, I think that the pattern database should match the complete program name. Using multiple patterns for the program name is technically sufficient, but it would be nice to use a regular expression so that I can match
imapd imapds
with the program impads?
As usual, all comments welcome, especially those that explain the history of why it was done this way and how the issue can be addressed.
Thanks to everyone for such an active community.
Evan. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I would like to write json object to syslog-ng. These are json objects that would match what syslog-ng would produce using its json template function. Is there any way to get syslog-ng to parse these as input lines and populate all of the internal tags, value pairs etc? Thanks, Evan.
Hi, I think (this might not actually work) the using the no-parse flag and the json-parser of 3.4 might do this: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guide... Hopefully others will correct me if I'm wrong. HTH Robert On 11/22/2012 12:43 AM, Evan Rempel wrote:
I would like to write json object to syslog-ng. These are json objects that would match what syslog-ng would produce using its json template function.
Is there any way to get syslog-ng to parse these as input lines and populate all of the internal tags, value pairs etc?
Thanks, Evan. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Fekete Robert <frobert@balabit.hu> writes:
On 11/22/2012 12:43 AM, Evan Rempel wrote:
I would like to write json object to syslog-ng. These are json objects that would match what syslog-ng would produce using its json template function.
Is there any way to get syslog-ng to parse these as input lines and populate all of the internal tags, value pairs etc?
I think (this might not actually work) the using the no-parse flag and the json-parser of 3.4 might do this: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guide...
Hopefully others will correct me if I'm wrong.
You are correct, the json-parser in 3.4 is the right tool for this job. Something along these lines should work: source s_json { tcp(flags(no-parse)); }; parser p_json { json-parser(); }; log { source(s_json); parser(p_json); [...] }; I'm not 100% sure it will handle tags properly, as I never tried that (everything else should just work, though). But if it does not, we can probably find some way to fix that. -- |8]
participants (6)
-
Balazs Scheidler
-
Balint Kovacs
-
Evan Rempel
-
Fekete Robert
-
Gergely Nagy
-
Gergely Nagy