I am trying to filter messages matching text stored in a txt file (plain txt , exact match , one word each line). but its not working filter f_userlist { in-list("/etc/syslog-ng/userlist.list", value("MESSAGE")); }; ---> NOT WORKING however it works with value(“PROGRAM”) filter f_whitelist { in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); }; --->WORKING List ( userlist.list ) is not long and has less than 10 words to match. anything missing ? or in-list filter doenot work with message contents . any troubleshooting tips will e helpful.
Hello, The *in-list* should work the same way for both *program* and *message*. It is a little hard to help without the rest of the *relevant* configuration. Therefore I created my dummy config and tested with it. $ cat /tmp/message-filter.txt :) $ cat /tmp/in-list.conf filter f_smile { in-list("/tmp/message-filter.txt", value("MESSAGE")); }; source s_stdin { file("/dev/stdin" flags(no-parse)); }; destination d_stdout { file("/dev/stdout"); }; log { source(s_stdin); filter(f_smile); destination(d_stdout); }; $ syslog-ng -f /tmp/in-list.conf syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' [2017-10-04T07:41:31.341435] WARNING: Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file to indicate this explicitly; ... :( :) Oct 4 07:41:50 peterkokai-work/peterkokai-work :) doomed to fail :) [EOF] This must be an exact match, which is why it seems a little fishy that you want to match *MESSAGE* macro :) -- Kokan On Tue, Oct 3, 2017 at 10:10 PM Gopi Joshi <gkjoshi@gmail.com> wrote:
I am trying to filter messages matching text stored in a txt file (plain txt , exact match , one word each line). but its not working
filter f_userlist { in-list("/etc/syslog-ng/userlist.list", value("MESSAGE")); }; ---> NOT WORKING
however it works with value(“PROGRAM”)
filter f_whitelist { in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); }; --->WORKING
List ( userlist.list ) is not long and has less than 10 words to match. anything missing ? or in-list filter doenot work with message contents . any troubleshooting tips will e helpful.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Tue, Oct 03, 2017 at 04:10:39PM -0400, Gopi Joshi wrote:
I am trying to filter messages matching text stored in a txt file (plain txt , exact match , one word each line). but its not working
As Péter suggested, you should first extract the username from the MESSAGE, for instance using one of the parsers (kv, csv, db, …), and then apply the in-list filter to that extracted macro Cheers
Here is an example for using the inlist filter together with the patterndb parser: https://czanik.blogs.balabit.com/2013/09/black-cat-white-cat/ In this case patterndb extract IP addresses from log messages which are checked using the inlist filter. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Wed, Oct 4, 2017 at 8:14 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Tue, Oct 03, 2017 at 04:10:39PM -0400, Gopi Joshi wrote:
I am trying to filter messages matching text stored in a txt file (plain txt , exact match , one word each line). but its not working
As Péter suggested, you should first extract the username from the MESSAGE, for instance using one of the parsers (kv, csv, db, …), and then apply the in-list filter to that extracted macro
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks Peter and Wernli. I am trying to use kv parser to extract username from MESSAGE template so that i can use it to compare further . from below VPN logs , i am trying to extract value LOCAL\user1 and user1 (value-separator("(") pair-separator(")") Teardown TCP connection 51507415 for outside:2.2.2.2/60709*(LOCAL\user1l)* to inside:5,5,5,5/443 duration 0:00:06 bytes 0 Failover primary closed *(user1)* i donot see it maching and any value being saved to $KEY1 to refer the match , Below is the conf file . @version: 3.12 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(port(514) flags(no-parse)); }; #### PARSER ### parser p_kv { kv-parser(value-separator("(") pair-separator(")") prefix("kv.")); }; ##### VPN LOG #### #filter f_users { in-list("/var/syslog-ng/user.list", value("${KEY1}")); }; --- Not in Use at present filter f-syslog { host("1.1.1.1"); }; destination log-vpntest { file("/var/log/vpn/vpntest.log" perm(0644) template("$ISODATE $HOST ${kv.KEY1} $KEY1\n") ); }; log { source(s_network);parser(p_kv);filter(f-syslog);destination(log-vpntest); }; #### VPN LOG ENDS ### Regards Gopi Joshi On Wed, Oct 4, 2017 at 2:54 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Here is an example for using the inlist filter together with the patterndb parser: https://czanik.blogs.balabit.com/2013/09/black-cat-white-cat/ In this case patterndb extract IP addresses from log messages which are checked using the inlist filter.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Wed, Oct 4, 2017 at 8:14 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Tue, Oct 03, 2017 at 04:10:39PM -0400, Gopi Joshi wrote:
I am trying to filter messages matching text stored in a txt file (plain txt , exact match , one word each line). but its not working
As Péter suggested, you should first extract the username from the MESSAGE, for instance using one of the parsers (kv, csv, db, …), and then apply the in-list filter to that extracted macro
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Czanik, Péter
-
Fabien Wernli
-
Gopi Joshi
-
Kókai Péter