Rewrite regex not working entirely
Hi List, I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog). Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029] I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+). Here is my statement for the rewrite rule. rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); }; The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5?? Regards, Max
You have defined your regular expresion as "posix" which does not have the \d \s etc. If you change the type to "pcre" then it should work for you. On 1/15/19 2:01 PM, N. Max Pierson wrote:
Hi List,
I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog).
Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029]
I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+).
Here is my statement for the rewrite rule.
rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); };
The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5??
Regards, Max
Hi Evan, I have tried both pcre and posix and neither seem to work. On Tue, Jan 15, 2019 at 4:08 PM Evan Rempel <erempel@uvic.ca> wrote:
You have defined your regular expresion as "posix" which does not have the \d \s etc. If you change the type to "pcre" then it should work for you.
On 1/15/19 2:01 PM, N. Max Pierson wrote:
Hi List,
I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog).
Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029]
I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+).
Here is my statement for the rewrite rule.
rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); };
The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5??
Regards, Max
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, try experimenting with a combination of single quotes and escaping \\
Hi Max, I tried "\w\d" , etc... in "pcre" type subst rewrite rule on 3.19. My config: @version: 3.19 @include "scl.conf" source s_udp5001 { udp( port(5001) keep-hostname(yes) flags(no-parse) ); }; destination d_test { file( "/tmp/test.log" ); }; rewrite r_chars { subst( "^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("$MESSAGE"), type("pcre"), flags("ignore-case") ); }; rewrite r_pcre { subst( '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s', "", value("$MESSAGE"), type("pcre"), flags("ignore-case") ); }; log { source(s_udp5001); #rewrite(r_chars); rewrite(r_pcre); destination(d_test); }; I think you need to make sure, that the regular expression is set between single quotes (e.g.: '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s') Best regards, Attila On Tue, Jan 15, 2019 at 11:12 PM N. Max Pierson <nmaxpierson@gmail.com> wrote:
Hi Evan,
I have tried both pcre and posix and neither seem to work.
On Tue, Jan 15, 2019 at 4:08 PM Evan Rempel <erempel@uvic.ca> wrote:
You have defined your regular expresion as "posix" which does not have the \d \s etc. If you change the type to "pcre" then it should work for you.
On 1/15/19 2:01 PM, N. Max Pierson wrote:
Hi List,
I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog).
Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029]
I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+).
Here is my statement for the rewrite rule.
rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); };
The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5??
Regards, Max
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Atilla, I switched the double quotes to single quotes and that fixed the issue. I do not believe the docs stated to use single quotes for full regular expressions to work which is why I used double quotes but in either case this resolved the issue. Thanks for the feedback! Regards, Max On Wed, Jan 16, 2019 at 5:41 AM Szakacs, Attila <attila.szakacs@balabit.com> wrote:
Hi Max,
I tried "\w\d" , etc... in "pcre" type subst rewrite rule on 3.19. My config:
@version: 3.19 @include "scl.conf"
source s_udp5001 { udp( port(5001) keep-hostname(yes) flags(no-parse) ); };
destination d_test { file( "/tmp/test.log" ); };
rewrite r_chars { subst( "^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("$MESSAGE"), type("pcre"), flags("ignore-case") ); };
rewrite r_pcre { subst( '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s', "", value("$MESSAGE"), type("pcre"), flags("ignore-case") ); };
log { source(s_udp5001); #rewrite(r_chars); rewrite(r_pcre); destination(d_test); };
I think you need to make sure, that the regular expression is set between single quotes (e.g.: '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s')
Best regards, Attila
On Tue, Jan 15, 2019 at 11:12 PM N. Max Pierson <nmaxpierson@gmail.com> wrote:
Hi Evan,
I have tried both pcre and posix and neither seem to work.
On Tue, Jan 15, 2019 at 4:08 PM Evan Rempel <erempel@uvic.ca> wrote:
You have defined your regular expresion as "posix" which does not have the \d \s etc. If you change the type to "pcre" then it should work for you.
On 1/15/19 2:01 PM, N. Max Pierson wrote:
Hi List,
I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog).
Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029]
I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+).
Here is my statement for the rewrite rule.
rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); };
The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5??
Regards, Max
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Max, We have some guidelines for regexes, how to optimize them, syntax and others: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... Regards, Gabor On Wed, Jan 16, 2019 at 5:33 PM N. Max Pierson <nmaxpierson@gmail.com> wrote:
Hi Atilla,
I switched the double quotes to single quotes and that fixed the issue. I do not believe the docs stated to use single quotes for full regular expressions to work which is why I used double quotes but in either case this resolved the issue.
Thanks for the feedback!
Regards, Max
On Wed, Jan 16, 2019 at 5:41 AM Szakacs, Attila < attila.szakacs@balabit.com> wrote:
Hi Max,
I tried "\w\d" , etc... in "pcre" type subst rewrite rule on 3.19. My config:
@version: 3.19 @include "scl.conf"
source s_udp5001 { udp( port(5001) keep-hostname(yes) flags(no-parse) ); };
destination d_test { file( "/tmp/test.log" ); };
rewrite r_chars { subst( "^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("$MESSAGE"), type("pcre"), flags("ignore-case") ); };
rewrite r_pcre { subst( '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s', "", value("$MESSAGE"), type("pcre"), flags("ignore-case") ); };
log { source(s_udp5001); #rewrite(r_chars); rewrite(r_pcre); destination(d_test); };
I think you need to make sure, that the regular expression is set between single quotes (e.g.: '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s')
Best regards, Attila
On Tue, Jan 15, 2019 at 11:12 PM N. Max Pierson <nmaxpierson@gmail.com> wrote:
Hi Evan,
I have tried both pcre and posix and neither seem to work.
On Tue, Jan 15, 2019 at 4:08 PM Evan Rempel <erempel@uvic.ca> wrote:
You have defined your regular expresion as "posix" which does not have the \d \s etc. If you change the type to "pcre" then it should work for you.
On 1/15/19 2:01 PM, N. Max Pierson wrote:
Hi List,
I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog).
Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029]
I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+).
Here is my statement for the rewrite rule.
rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); };
The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5??
Regards, Max
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Evan Rempel
-
Fabien Wernli
-
N. Max Pierson
-
Nagy, Gábor
-
Szakacs, Attila