Re: [syslog-ng] weird filter problem
Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x My advice my be specific to version 3.4 :-( Evan Rempel 250.271.7691 University Systems, University of Victoria Evan Rempel <erempel@uvic.ca> wrote: This definitely works. I'm using it right now. If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton <r.fulton@auckland.ac.nz> wrote: On 11/05/2013, at 2:26 PM, Evan Rempel <erempel@uvic.ca> wrote:
Try this filter
filter f_unknown { tags(".classifier.unknown"); };
This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start: Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory So far as I can tell all the lib files are present and correct and in the same place as the previous version? I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use. Russell ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The issue is probably where the filter resides. I use that filter (in fact, it's in an optional ELSA config right now) and it works, but you have to remember that ${.classifier.class} isn't set until after the patterndb parser is run, so the filter() statement has to be after parser(p_db); On Fri, May 10, 2013 at 11:51 PM, Evan Rempel <erempel@uvic.ca> wrote:
Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x
My advice my be specific to version 3.4 :-(
Evan Rempel 250.271.7691 University Systems, University of Victoria
Evan Rempel <erempel@uvic.ca> wrote:
This definitely works. I'm using it right now.
If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database.
Evan Rempel 250.271.7691 University Systems, University of Victoria
Russell Fulton <r.fulton@auckland.ac.nz> wrote:
On 11/05/2013, at 2:26 PM, Evan Rempel <erempel@uvic.ca> wrote:
Try this filter
filter f_unknown { tags(".classifier.unknown"); };
This always appears to return true. I.e. this filter includes everything. Negating it includes nothing.
I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start:
Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory
So far as I can tell all the lib files are present and correct and in the same place as the previous version?
I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use.
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 13/05/2013, at 3:58 PM, Martin Holste <mcholste@gmail.com> wrote:
The issue is probably where the filter resides. I use that filter (in fact, it's in an optional ELSA config right now) and it works, but you have to remember that ${.classifier.class} isn't set until after the patterndb parser is run, so the filter() statement has to be after parser(p_db);
I finally figured out what the issue was here. It had to be something totally, idiotically simple and it was. Martin was on the right track with the order of filters relative to parser(p_db); What had happened was that I had originally the filter in a second log {} clause after one that contained the parser() entry so everything worked. Martin introduce the elsa_syslog.conf include and I moved all my local mods into there so now the filter was in a log{} clause that did not have a parser() entry and was now before the one that had it. I won't tell how many hours careful elimination it took to track this down. For elsa users if you put new log{} clauses in the include file you must have a parse() entry in them if you want to do anything with the classifier results. Question: Will having two parser() entries result in the log message being parsed twice? My guess is that it will. R
On Fri, May 10, 2013 at 11:51 PM, Evan Rempel <erempel@uvic.ca> wrote: Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x
My advice my be specific to version 3.4 :-(
Evan Rempel 250.271.7691 University Systems, University of Victoria
Evan Rempel <erempel@uvic.ca> wrote:
This definitely works. I'm using it right now.
If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database.
Evan Rempel 250.271.7691 University Systems, University of Victoria
Russell Fulton <r.fulton@auckland.ac.nz> wrote:
On 11/05/2013, at 2:26 PM, Evan Rempel <erempel@uvic.ca> wrote:
Try this filter
filter f_unknown { tags(".classifier.unknown"); };
This always appears to return true. I.e. this filter includes everything. Negating it includes nothing.
I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start:
Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory
So far as I can tell all the lib files are present and correct and in the same place as the previous version?
I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use.
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
That's a great question--I have no idea if two parser entries mandate double parsing. If you want to make sure that only your custom log {} statement will be used, you can use flags(final) in your log {} stanza to ensure that no messages will continue on to the other log statements. On Tue, May 14, 2013 at 11:22 PM, Russell Fulton <r.fulton@auckland.ac.nz>wrote:
On 13/05/2013, at 3:58 PM, Martin Holste <mcholste@gmail.com> wrote:
The issue is probably where the filter resides. I use that filter (in fact, it's in an optional ELSA config right now) and it works, but you have to remember that ${.classifier.class} isn't set until after the patterndb parser is run, so the filter() statement has to be after parser(p_db);
I finally figured out what the issue was here. It had to be something totally, idiotically simple and it was.
Martin was on the right track with the order of filters relative to parser(p_db);
What had happened was that I had originally the filter in a second log {} clause after one that contained the parser() entry so everything worked. Martin introduce the elsa_syslog.conf include and I moved all my local mods into there so now the filter was in a log{} clause that did not have a parser() entry and was now before the one that had it.
I won't tell how many hours careful elimination it took to track this down.
For elsa users if you put new log{} clauses in the include file you must have a parse() entry in them if you want to do anything with the classifier results.
Question: Will having two parser() entries result in the log message being parsed twice? My guess is that it will.
R
On Fri, May 10, 2013 at 11:51 PM, Evan Rempel <erempel@uvic.ca> wrote: Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x
My advice my be specific to version 3.4 :-(
Evan Rempel 250.271.7691 University Systems, University of Victoria
Evan Rempel <erempel@uvic.ca> wrote:
This definitely works. I'm using it right now.
If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database.
Evan Rempel 250.271.7691 University Systems, University of Victoria
Russell Fulton <r.fulton@auckland.ac.nz> wrote:
On 11/05/2013, at 2:26 PM, Evan Rempel <erempel@uvic.ca> wrote:
Try this filter
filter f_unknown { tags(".classifier.unknown"); };
This always appears to return true. I.e. this filter includes everything. Negating it includes nothing.
I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start:
Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory
So far as I can tell all the lib files are present and correct and in the same place as the previous version?
I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use.
Russell
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- You received this message because you are subscribed to the Google Groups "enterprise-log-search-and-archive" group. To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-log-search-and-archive+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Yeah, two parser references cause it to be parsed twice. On May 15, 2013 5:08 PM, "Martin Holste" <mcholste@gmail.com> wrote:
That's a great question--I have no idea if two parser entries mandate double parsing. If you want to make sure that only your custom log {} statement will be used, you can use flags(final) in your log {} stanza to ensure that no messages will continue on to the other log statements.
On Tue, May 14, 2013 at 11:22 PM, Russell Fulton <r.fulton@auckland.ac.nz>wrote:
On 13/05/2013, at 3:58 PM, Martin Holste <mcholste@gmail.com> wrote:
The issue is probably where the filter resides. I use that filter (in fact, it's in an optional ELSA config right now) and it works, but you have to remember that ${.classifier.class} isn't set until after the patterndb parser is run, so the filter() statement has to be after parser(p_db);
I finally figured out what the issue was here. It had to be something totally, idiotically simple and it was.
Martin was on the right track with the order of filters relative to parser(p_db);
What had happened was that I had originally the filter in a second log {} clause after one that contained the parser() entry so everything worked. Martin introduce the elsa_syslog.conf include and I moved all my local mods into there so now the filter was in a log{} clause that did not have a parser() entry and was now before the one that had it.
I won't tell how many hours careful elimination it took to track this down.
For elsa users if you put new log{} clauses in the include file you must have a parse() entry in them if you want to do anything with the classifier results.
Question: Will having two parser() entries result in the log message being parsed twice? My guess is that it will.
R
On Fri, May 10, 2013 at 11:51 PM, Evan Rempel <erempel@uvic.ca> wrote: Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x
My advice my be specific to version 3.4 :-(
Evan Rempel 250.271.7691 University Systems, University of Victoria
Evan Rempel <erempel@uvic.ca> wrote:
This definitely works. I'm using it right now.
If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database.
Evan Rempel 250.271.7691 University Systems, University of Victoria
Russell Fulton <r.fulton@auckland.ac.nz> wrote:
On 11/05/2013, at 2:26 PM, Evan Rempel <erempel@uvic.ca> wrote:
Try this filter
filter f_unknown { tags(".classifier.unknown"); };
This always appears to return true. I.e. this filter includes everything. Negating it includes nothing.
I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start:
Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory
So far as I can tell all the lib files are present and correct and in the same place as the previous version?
I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use.
Russell
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- You received this message because you are subscribed to the Google Groups "enterprise-log-search-and-archive" group. To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-log-search-and-archive+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 13/05/2013, at 3:58 PM, Martin Holste <mcholste@gmail.com> wrote:
The issue is probably where the filter resides. I use that filter (in fact, it's in an optional ELSA config right now) and it works, but you have to remember that ${.classifier.class} isn't set until after the patterndb parser is run, so the filter() statement has to be after parser(p_db);
yes. that was the issue. I had multiple log{} statements and I reordered them not realising that one was missing parser(p_db) Now the question is: If I have two log{} clauses both with parser(p_db) does the parsing get done twice? Russell
On Fri, May 10, 2013 at 11:51 PM, Evan Rempel <erempel@uvic.ca> wrote: Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x
My advice my be specific to version 3.4 :-(
Evan Rempel 250.271.7691 University Systems, University of Victoria
Evan Rempel <erempel@uvic.ca> wrote:
This definitely works. I'm using it right now.
If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database.
Evan Rempel 250.271.7691 University Systems, University of Victoria
Russell Fulton <r.fulton@auckland.ac.nz> wrote:
On 11/05/2013, at 2:26 PM, Evan Rempel <erempel@uvic.ca> wrote:
Try this filter
filter f_unknown { tags(".classifier.unknown"); };
This always appears to return true. I.e. this filter includes everything. Negating it includes nothing.
I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start:
Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory
So far as I can tell all the lib files are present and correct and in the same place as the previous version?
I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use.
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Martin Holste
-
Russell Fulton