RE: [syslog-ng] Ignoring previously handled/filtered messages
IMHO, I WANT the messages to go to several logfiles. For instance, I want a message mail.fatal to go to the fatal logfile as well as the mail logfile. This way I can trigger on messages in the fatal logfile, look in the mail logfile and see if there were any messages in from mail that lead up to the fatal situation. However, when you do the same thing loggin to a remote host, this procedure results in four (!) messages being generated. A better way to handle 'double' messages would be very very nice. : -----Original Message----- : From: John Goggan [mailto:jgoggan@dcg.com] : Sent: Wednesday, May 24, 2000 4:46 PM : To: syslog-ng@venus.terrasoft.hu : Subject: Re: [syslog-ng] Ignoring previously handled/filtered messages : : : : ger wrote: : > try : > filter f_daemon { facility(daemon) and not program(name);}; : > : > where "name" is whatever program you're trying to filter out. : : Thanks to Ger and Ilya for their responses. While this does : work, it really : seems like a poor way to do it to me -- and, I believe, is : also quite a bit : different from the way standard syslog handles things. The : problem is that it : should just be easy to make it so that messages are handled : only ONCE and then : not sent to additional logs unless specifically desires. How : many people : really want the same syslog message sent to multiple logs : most of the time? : It just seems that the default (which does this) is the : opposite of what most : would want. : : Plus, what needs to be done to get around this just doesn't : make any sense. : This means that if I want separate logs for imapd, named, : telnetd, and ftpd : (to name just a few common daemons) -- and I want any other : daemons that pop : up to still go to a daemon.log, I have to do this in my : filter section: : : filter f_imapd { program("imapd"); }; : filter f_named { program("named"); }; : filter f_telnetd { program("telnetd"); }; : filter f_ftpd { program("ftpd"); }; : filter f_auth { facility(auth) : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : filter f_daemon { facility(daemon) : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : filter f_messages { level(info..warn) : and not facility(auth, authpriv, mail, news); : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : : That just seems like a really bad design! I have to make : sure that I keep all : of the names updated in several different locations. If I : want to add a log : for just one more daemon, I have to add it to at least 3 : other filters besides : its own filter! : : Aren't other people experiencing such problems? Or is no one : out there using : syslog-ng to filter out specific daemons while still catching : the unlisted : daemons in a daemon.log file? : : There must be a better way... : : - John... : : _______________________________________________ : syslog-ng maillist - syslog-ng@lists.balabit.hu : http://lists.balabit.hu/mailman/listinfo/syslog-ng :
Agreed. What I'd like is a clause that means "log this only if not yet handled". That way, we get it both ways -- if you don't specify anything, you get the current behavior, in which it would go to both logs. Plus, with the "not-yet-logged" clause, I could catch everything where I want it, and then do the "not-yet-logged" clause just for the remaining facility(daemon), for example. - John... "Ohrberg, Mick" wrote:
IMHO, I WANT the messages to go to several logfiles. For instance, I want a message mail.fatal to go to the fatal logfile as well as the mail logfile. This way I can trigger on messages in the fatal logfile, look in the mail logfile and see if there were any messages in from mail that lead up to the fatal situation.
However, when you do the same thing loggin to a remote host, this procedure results in four (!) messages being generated. A better way to handle 'double' messages would be very very nice.
: -----Original Message----- : From: John Goggan [mailto:jgoggan@dcg.com] : Sent: Wednesday, May 24, 2000 4:46 PM : To: syslog-ng@venus.terrasoft.hu : Subject: Re: [syslog-ng] Ignoring previously handled/filtered messages : : : : ger wrote: : > try : > filter f_daemon { facility(daemon) and not program(name);}; : > : > where "name" is whatever program you're trying to filter out. : : Thanks to Ger and Ilya for their responses. While this does : work, it really : seems like a poor way to do it to me -- and, I believe, is : also quite a bit : different from the way standard syslog handles things. The : problem is that it : should just be easy to make it so that messages are handled : only ONCE and then : not sent to additional logs unless specifically desires. How : many people : really want the same syslog message sent to multiple logs : most of the time? : It just seems that the default (which does this) is the : opposite of what most : would want. : : Plus, what needs to be done to get around this just doesn't : make any sense. : This means that if I want separate logs for imapd, named, : telnetd, and ftpd : (to name just a few common daemons) -- and I want any other : daemons that pop : up to still go to a daemon.log, I have to do this in my : filter section: : : filter f_imapd { program("imapd"); }; : filter f_named { program("named"); }; : filter f_telnetd { program("telnetd"); }; : filter f_ftpd { program("ftpd"); }; : filter f_auth { facility(auth) : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : filter f_daemon { facility(daemon) : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : filter f_messages { level(info..warn) : and not facility(auth, authpriv, mail, news); : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : : That just seems like a really bad design! I have to make : sure that I keep all : of the names updated in several different locations. If I : want to add a log : for just one more daemon, I have to add it to at least 3 : other filters besides : its own filter! : : Aren't other people experiencing such problems? Or is no one : out there using : syslog-ng to filter out specific daemons while still catching : the unlisted : daemons in a daemon.log file? : : There must be a better way... : : - John... : : _______________________________________________ : syslog-ng maillist - syslog-ng@lists.balabit.hu : http://lists.balabit.hu/mailman/listinfo/syslog-ng :
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu http://lists.balabit.hu/mailman/listinfo/syslog-ng
Agreed. What I'd like is a clause that means "log this only if not yet handled". That way, we get it both ways -- if you don't specify anything, you get the current behavior, in which it would go to both logs. Plus, with the "not-yet-logged" clause, I could catch everything where I want it, and then do the "not-yet-logged" clause just for the remaining facility(daemon), for example.
- John...
The DEFAULT filter exists for exactly this purpose. It may be buggy since I haven't touched that feature since 1.3.xx (when I implemented it), but the syntax looks like this: log { source(src); filter(DEFAULT); destination(dst); }; DEFAULT filters all not-yet-handled messages. I'll check out the other not expression bugs as well. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
Agreed. What I'd like is a clause that means "log this only if not yet handled". That way, we get it both ways -- if you don't specify anything, you get the current behavior, in which it would go to both logs. Plus, with the "not-yet-logged" clause, I could catch everything where I want it, and then do the "not-yet-logged" clause just for the remaining facility(daemon), for example.
The DEFAULT filter exists for exactly this purpose. It may be buggy since I haven't touched that feature since 1.3.xx (when I implemented it), but the syntax looks like this:
log { source(src); filter(DEFAULT); destination(dst); };
DEFAULT filters all not-yet-handled messages. I'll check out the other not expression bugs as well.
I tested this, and it seems to work. The configuration I tried: options { keep_hostname(yes); }; source src { unix-stream("proba2"); internal(); }; destination ftpd { file("ftplog"); }; destination named { file("namedlog"); }; destination daemon { file("daemonlog"); }; filter f_ftpd { match("ftp"); }; filter f_named { match("named"); }; filter f_daemon { facility(daemon); }; log { source(src); filter(f_ftpd); destination(ftpd); }; log { source(src); filter(f_named); destination(named); }; log { source(src); filter(f_daemon); filter(DEFAULT); destination(daemon); }; The lines I logged: balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "ftp" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "named" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "qqq" All of them went to the expected location. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
Balazs Scheidler wrote:
I tested this, and it seems to work. The configuration I tried:
options { keep_hostname(yes); };
source src { unix-stream("proba2"); internal(); };
destination ftpd { file("ftplog"); }; destination named { file("namedlog"); }; destination daemon { file("daemonlog"); };
filter f_ftpd { match("ftp"); }; filter f_named { match("named"); }; filter f_daemon { facility(daemon); };
log { source(src); filter(f_ftpd); destination(ftpd); }; log { source(src); filter(f_named); destination(named); }; log { source(src); filter(f_daemon); filter(DEFAULT); destination(daemon); };
I'm confused here... Does that last line above cause JUST the unlogged filter(f_daemon) messages to go to the daemon destination? Or would it also catch any other unfiltered (non-daemon-facility) messages also? Do multiple filters on a log line treated as "OR" or as "AND" filters? Meaning does it only log where the messages passes both filters? Or where it passes either filter listed? If it does an "AND" and it must pass both filters, then you are correct and this will do exactly what I desire. Thanks!
The lines I logged:
balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "ftp" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "named" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "qqq"
All of them went to the expected location.
So, the "ftp" one went only to the ftplog, the "named" when only to the namedlog, and the "qqq" went only to the daemonlog, correct? Thanks for your time and explaining this... - John...
I tested this, and it seems to work. The configuration I tried:
options { keep_hostname(yes); };
source src { unix-stream("proba2"); internal(); };
destination ftpd { file("ftplog"); }; destination named { file("namedlog"); }; destination daemon { file("daemonlog"); };
filter f_ftpd { match("ftp"); }; filter f_named { match("named"); }; filter f_daemon { facility(daemon); };
log { source(src); filter(f_ftpd); destination(ftpd); }; log { source(src); filter(f_named); destination(named); }; log { source(src); filter(f_daemon); filter(DEFAULT); destination(daemon); };
I'm confused here... Does that last line above cause JUST the unlogged filter(f_daemon) messages to go to the daemon destination? Or would it also catch any other unfiltered (non-daemon-facility) messages also?
Do multiple filters on a log line treated as "OR" or as "AND" filters? Meaning does it only log where the messages passes both filters? Or where it passes either filter listed? If it does an "AND" and it must pass both filters, then you are correct and this will do exactly what I desire. Thanks!
They are ANDed. I've written a paragraph about log path evaluation in the documentation.
The lines I logged:
balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "ftp" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "named" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "qqq"
All of them went to the expected location.
So, the "ftp" one went only to the ftplog, the "named" when only to the namedlog, and the "qqq" went only to the daemonlog, correct?
yes. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt
On Sat, 27 May 2000, Balazs Scheidler wrote:
log { source(src); filter(f_daemon); filter(DEFAULT); destination(daemon); };
Do multiple filters on a log line treated as "OR" or as "AND" filters? Meaning does it only log where the messages passes both filters? Or where it passes either filter listed? If it does an "AND" and it must pass both filters, then you are correct and this will do exactly what I desire. Thanks!
They are ANDed. I've written a paragraph about log path evaluation in the documentation.
This doesn't seem to be true of the DEFAULT filter. I just tried the following (this is my first log directive): destination named { file("/var/log/named.log"); }; filter f_named { program("named"); }; log { source("src"); filter("f_named"); filter(DEFAULT); destination("named"); }; and found the following in named.log May 31 16:07:29 src@morpheus syslog-ng[332]: syslog-ng version 1.4.4 starting May 31 16:07:29 src@morpheus syslog: syslogd startup succeeded I'd really like the DEFAULT rule to work this way, it would be perfect for limiting some log files to unique messages and others to containing everything. -- Dan Rich <drich@employees.org> | http://www.employees.org/~drich/ | "Danger, you haven't seen the last of me!" | "No, but the first of you turns my stomach!" | -- The Firesign Theatre's Nick Danger
Balazs Scheidler wrote:
Agreed. What I'd like is a clause that means "log this only if not yet handled". That way, we get it both ways -- if you don't specify anything, you get the current behavior, in which it would go to both logs. Plus, with the "not-yet-logged" clause, I could catch everything where I want it, and then do the "not-yet-logged" clause just for the remaining facility(daemon), for example.
The DEFAULT filter exists for exactly this purpose. It may be buggy since I haven't touched that feature since 1.3.xx (when I implemented it), but the syntax looks like this:
log { source(src); filter(DEFAULT); destination(dst); };
DEFAULT filters all not-yet-handled messages. I'll check out the other not expression bugs as well.
I tested this, and it seems to work. The configuration I tried:
options { keep_hostname(yes); };
source src { unix-stream("proba2"); internal(); };
destination ftpd { file("ftplog"); }; destination named { file("namedlog"); }; destination daemon { file("daemonlog"); };
filter f_ftpd { match("ftp"); }; filter f_named { match("named"); }; filter f_daemon { facility(daemon); };
log { source(src); filter(f_ftpd); destination(ftpd); }; log { source(src); filter(f_named); destination(named); }; log { source(src); filter(f_daemon); filter(DEFAULT); destination(daemon); };
The lines I logged:
balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "ftp" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "named" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "qqq"
All of them went to the expected location.
Well, not for me :-) ( V 1.4.3) Since I can't make it work, I made a very simple configuration and found strange things: log { source( local); source( network); filter( f_bash); destination( d_bash); }; log { source( local); source( network); filter( f_auth); # filter( DEFAULT); destination( d_auth); }; bash-log line appers in both file as expected. If I remove the comment, when, bash-log appers only in bash file, but some strange new things appers in the deamon file: some old kernel one ( few days old) and inedt one "Jun 2 12:36:18 ts inetd[188]: /usr/openwin/bin/Xaserver: Hangup" Since filter directive are supposed to be ANDed, I am a bit confused. I am still debugging... Un P.S: Thanks to John Goggan, explaning my problem better than me :-)
Un L'Unique wrote:
Balazs Scheidler wrote:
Agreed. What I'd like is a clause that means "log this only if not yet handled". That way, we get it both ways -- if you don't specify anything, you get the current behavior, in which it would go to both logs. Plus, with the "not-yet-logged" clause, I could catch everything where I want it, and then do the "not-yet-logged" clause just for the remaining facility(daemon), for example.
The DEFAULT filter exists for exactly this purpose. It may be buggy since I haven't touched that feature since 1.3.xx (when I implemented it), but the syntax looks like this:
log { source(src); filter(DEFAULT); destination(dst); };
DEFAULT filters all not-yet-handled messages. I'll check out the other not expression bugs as well.
I tested this, and it seems to work. The configuration I tried:
options { keep_hostname(yes); };
source src { unix-stream("proba2"); internal(); };
destination ftpd { file("ftplog"); }; destination named { file("namedlog"); }; destination daemon { file("daemonlog"); };
filter f_ftpd { match("ftp"); }; filter f_named { match("named"); }; filter f_daemon { facility(daemon); };
log { source(src); filter(f_ftpd); destination(ftpd); }; log { source(src); filter(f_named); destination(named); }; log { source(src); filter(f_daemon); filter(DEFAULT); destination(daemon); };
The lines I logged:
balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "ftp" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "named" balabit:~/src/syslog-ng-1.4/src$ logger -u proba2 -p daemon.info "qqq"
All of them went to the expected location.
Well, not for me :-) ( V 1.4.3)
Since I can't make it work, I made a very simple configuration and found strange things: log { source( local); source( network);
filter( f_bash); destination( d_bash); };
log { source( local); source( network);
filter( f_auth); # filter( DEFAULT); destination( d_auth); };
bash-log line appers in both file as expected. If I remove the comment, when, bash-log appers only in bash file, but some strange new things appers in the deamon file: some old kernel one ( few days old) and inedt one "Jun 2 12:36:18 ts inetd[188]: /usr/openwin/bin/Xaserver: Hangup"
Since filter directive are supposed to be ANDed, I am a bit confused.
I am still debugging...
So I thing have found what's wrong: there is: log { source( local); source( network); destination( all); }; at the begging of my log section, which get all message leaving nothing for the faclity( DEFAULT); statment. So I remove it and put the DEFAULT in all "general" log {} but find a new strange beahvior: auth facility goes to user log !. Finaly I found this mistake too: the log {} user was the first one containing filter( DEFAULT); . So my conclusion: - filter( DEFAULT); don't allow to "duplicate those but don't duplicate thease" - filter( DEFAULT); make forward every log to another host difficult to get - filter( DEFAULT); have bug. - filter( DEFAULT); is too hard to understand for me :-) Could we have another way to "duplicate those but don't duplicate thease" quickly ? I hope there is not more important stuff in the corner... Un
Balazs Scheidler wrote:
The DEFAULT filter exists for exactly this purpose. It may be buggy since I haven't touched that feature since 1.3.xx (when I implemented it), but the syntax looks like this:
log { source(src); filter(DEFAULT); destination(dst); };
DEFAULT filters all not-yet-handled messages. I'll check out the other not expression bugs as well.
As I tried to explain before, I have looked into filter(DEFAULT), but it does NOT do what I want. The problem is that it only catches EVERYTHING that has not yet been logged. I want to do it for specific things (such as a specific facility). I think the idea of the DEFAULT filter is what I'm looking for -- but not on such a generic level. Again, my original desire was as follows: -log specific daemons (ftpd, telnetd, imapd, sendmail) to their own log files. -log all other non-specified facility(daemon) messages to a daemon log file and no where else. Currently, I can do this if I specify the specific daemons as "not program(whatever)" in the facility(daemon) filter. The problem is that each time I add a new daemon to have its own log, I have to remember to add it to at least 3 different NOT filters if I don't want it to show up in the daemon or other various logs! This is bad. The filter(DEFAULT) will only catch EVERYTHING that hasn't been logged. This isn't good since it will get much more than the leftover facility(daemon) messages. If filter(DEFAULT) could be made more specific, then I think it would be what some of us are asking for. For example, if I could do "filter(daemon,DEFAULT)" to get all of the facility(daemon) messages that hadn't been logged yet, that would make it much more useful. Of course, I can see people wanting to do "filter(debug,DEFAULT)" instead to try to catch all of the unfiltered level debug messages, so maybe some other syntax to be able to specify a facility or level or whatever would be more appropriate... Thanks... - John...
Agreed. What I'd like is a clause that means "log this only if not yet handled". That way, we get it both ways -- if you don't specify anything, you get the current behavior, in which it would go to both logs. Plus, with the "not-yet-logged" clause, I could catch everything where I want it, and then do the "not-yet-logged" clause just for the remaining facility(daemon), for example. - John... "Ohrberg, Mick" wrote:
IMHO, I WANT the messages to go to several logfiles. For instance, I want a message mail.fatal to go to the fatal logfile as well as the mail logfile. This way I can trigger on messages in the fatal logfile, look in the mail logfile and see if there were any messages in from mail that lead up to the fatal situation.
However, when you do the same thing loggin to a remote host, this procedure results in four (!) messages being generated. A better way to handle 'double' messages would be very very nice.
: -----Original Message----- : From: John Goggan [mailto:jgoggan@dcg.com] : Sent: Wednesday, May 24, 2000 4:46 PM : To: syslog-ng@venus.terrasoft.hu : Subject: Re: [syslog-ng] Ignoring previously handled/filtered messages : : : : ger wrote: : > try : > filter f_daemon { facility(daemon) and not program(name);}; : > : > where "name" is whatever program you're trying to filter out. : : Thanks to Ger and Ilya for their responses. While this does : work, it really : seems like a poor way to do it to me -- and, I believe, is : also quite a bit : different from the way standard syslog handles things. The : problem is that it : should just be easy to make it so that messages are handled : only ONCE and then : not sent to additional logs unless specifically desires. How : many people : really want the same syslog message sent to multiple logs : most of the time? : It just seems that the default (which does this) is the : opposite of what most : would want. : : Plus, what needs to be done to get around this just doesn't : make any sense. : This means that if I want separate logs for imapd, named, : telnetd, and ftpd : (to name just a few common daemons) -- and I want any other : daemons that pop : up to still go to a daemon.log, I have to do this in my : filter section: : : filter f_imapd { program("imapd"); }; : filter f_named { program("named"); }; : filter f_telnetd { program("telnetd"); }; : filter f_ftpd { program("ftpd"); }; : filter f_auth { facility(auth) : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : filter f_daemon { facility(daemon) : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : filter f_messages { level(info..warn) : and not facility(auth, authpriv, mail, news); : and not program("imapd") : and not program("named") : and not program("telnetd") : and not program("ftpd"); }; : : That just seems like a really bad design! I have to make : sure that I keep all : of the names updated in several different locations. If I : want to add a log : for just one more daemon, I have to add it to at least 3 : other filters besides : its own filter! : : Aren't other people experiencing such problems? Or is no one : out there using : syslog-ng to filter out specific daemons while still catching : the unlisted : daemons in a daemon.log file? : : There must be a better way... : : - John... : : _______________________________________________ : syslog-ng maillist - syslog-ng@lists.balabit.hu : http://lists.balabit.hu/mailman/listinfo/syslog-ng :
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu http://lists.balabit.hu/mailman/listinfo/syslog-ng
participants (5)
-
Balazs Scheidler
-
Dan Rich
-
John Goggan
-
Ohrberg, Mick
-
Un L'Unique