Hello, I ran into an interesting situation. Yesterday I created patterns for uw-imapd. Today I looked at some log files from courier imapd and found, that this imap implementation also uses "imapd" for $PROGRAM (and "imapd-ssl" for port 993 connections). "imapd" and "imapd-ssl" messages looked the same. So, here is a list of questions: - how to handle at file/ruleset/etc level when two applications have the same $PROGRAM - how to handle, when the same application uses different $PROGRAM in different situations Also, I took a look at postfix logs, and they look rather ugly: Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: connect from czp.localnet[192.168.2.179] Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: 8434B41C30: client=czp.localnet[192.168.2.179], sasl_method=PLAIN, sasl_username=czanik Oct 13 21:35:29 ubuntu postfix/cleanup[3946]: 8434B41C30: message-id=<4CB609F2.8@blabla.com> Oct 13 21:35:29 ubuntu postfix/qmgr[3570]: 8434B41C30: from=<czanik@blabla.com>, size=619, nrcpt=1 (queue active) Oct 13 21:35:29 ubuntu postfix/smtpd[3942]: disconnect from czp.localnet[192.168.2.179] Oct 13 21:35:50 ubuntu postfix/smtp[3947]: connect to targetmachine[1.2.3.4]:25: Connection timed out Oct 13 21:35:50 ubuntu postfix/smtp[3947]: 8434B41C30: to=<czanik@targetmachine>, relay=none, delay=21, delays=0.02/0.01/21/0, dsn=4.4.1, status=deferred (connect to targetmachine[1.2.3.4]:25: Connection timed out) This is an SMTP authentication, and then the e-mail is tried to be delivered to targetmachine. There are many different names as $PROGRAM, $PID also has many different values. But "8434B41C30" could easily be used as session identifier for all of this. The question is the same: how should pattern name be handled? Bye, -- Peter Czanik (CzP) <czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
Hello CzP,
- how to handle at file/ruleset/etc level when two applications have the same $PROGRAM
- how to handle, when the same application uses different $PROGRAM in different situations
There are many different names as $PROGRAM,
The question is the same: how should pattern name be handled?
I ran into the same problem with some brain damaged programs and log messages where it was not possible to get a reliable program name match. Fortunately the problem is not too difficult to work around. I've collected some explanations from the manual to help you out. Basically, depending on the situation, you can: 1) use syslog-ng parser(s) to match the program name and then reclassify the messages using a rule based on checking the value in the program name parser(s) 2) use a substring of the program name which starts the program's name in each case. This should work for postfix since each subdaemon's name begins with 'postfix/'. 3) leave the program name blank, and then that pattern DB becomes a 'fallback' DB which is checked for any messages nothing else can classify. This is what I've chosen to do for the brain damaged programs I need to support. So far I haven't noticed a performance problem but I only loaded the system lightly. Hope this helps, Matthew Hall. Notes from manuals: [T]he rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The ... content of the $PROGRAM macro is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message. If the $PROGRAM part of a message is empty, rules with an empty Program Pattern are used to classify the message. The Program Pattern can be a string that specifies the name of the appliation or the beginning of its name (e.g., to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. pattern: The name of the application — syslog-ng matches this value to the $PROGRAM header of the syslog message to find the rulesets applicable to the syslog message. This element is also called the program pattern. If the <pattern> element of a ruleset is not specified, -ng will use this ruleset as a fallback ruleset: it will apply the ruleset to messages that have an empty PROGRAM header, or if none of the program patterns matched the PROGRAM header of the incoming message.
Hello, On 10/14/2010 08:59 AM, Matthew Hall wrote:
Basically, depending on the situation, you can: 1) use syslog-ng parser(s) to match the program name and then reclassify the messages using a rule based on checking the value in the program name parser(s)
2) use a substring of the program name which starts the program's name in each case. This should work for postfix since each subdaemon's name begins with 'postfix/'.
3) leave the program name blank, and then that pattern DB becomes a 'fallback' DB which is checked for any messages nothing else can classify. This is what I've chosen to do for the brain damaged programs I need to support. So far I haven't noticed a performance problem but I only loaded the system lightly.
Option 1) needs changes also in syslog-ng.conf, so it's not good as a general solution. Option 3) looks ugly too, and might be slower / less reliable when a large number of patterns are used. So far 2) seems to be the most useful for general use. This can handle postfix/* and imapd*. I still wonder how to handle when then same $PROGRAM is used for different applications, like "imapd" for both wu-imapd and courier-imapd. First I thought, that all should go to the same imapd.pdb, but under different rulesets. But that has a problem: courier is not just imap, but also a pop3 server. Also http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;h... says, that "Applications are grouped by their respective function and each application gets a single file that lists all the patterns of that application." So it would be more appropriate to rename imapd.pdb to wu-imapd.pdb (as it would only have patterns for Washington University IMAPd), and have a separate courier.pdb, or even courier-imapd.pdb and courier-pop3d.pdb. What do you think? Bye, -- Peter Czanik (CzP) <czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
Hi CzP, On Thu, Oct 14, 2010 at 12:12:03PM +0200, Peter Czanik wrote:
"Applications are grouped by their respective function and each application gets a single file that lists all the patterns of that application."
Seems clear enough.
So it would be more appropriate to rename imapd.pdb to wu-imapd.pdb (as it would only have patterns for Washington University IMAPd), and have a separate courier.pdb, or even courier-imapd.pdb and courier-pop3d.pdb. What do you think?
Yes, in the samples I have seen there are many Pattern DBs named after brands, companies, and/or specific software products (like WU IMAPd or Couried IMAPd, etc.) So I don't think there's anything wrong with naming your XMLs after the program they support. That makes it easy for the administrator to find and enable the right XMLs for their machines.
Bye, Peter Czanik (CzP) <czanik@balabit.hu>
Thanks for all the hard work making nice patterns. I have some I'll contribute soon once I've given them a proper production test. Regards, Matthew Hall.
On Thu, 2010-10-14 at 12:12 +0200, Peter Czanik wrote:
Hello,
On 10/14/2010 08:59 AM, Matthew Hall wrote:
Basically, depending on the situation, you can: 1) use syslog-ng parser(s) to match the program name and then reclassify the messages using a rule based on checking the value in the program name parser(s)
2) use a substring of the program name which starts the program's name in each case. This should work for postfix since each subdaemon's name begins with 'postfix/'.
3) leave the program name blank, and then that pattern DB becomes a 'fallback' DB which is checked for any messages nothing else can classify. This is what I've chosen to do for the brain damaged programs I need to support. So far I haven't noticed a performance problem but I only loaded the system lightly.
Option 1) needs changes also in syslog-ng.conf, so it's not good as a general solution. Option 3) looks ugly too, and might be slower / less reliable when a large number of patterns are used. So far 2) seems to be the most useful for general use. This can handle postfix/* and imapd*. I still wonder how to handle when then same $PROGRAM is used for different applications, like "imapd" for both wu-imapd and courier-imapd. First I thought, that all should go to the same imapd.pdb, but under different rulesets. But that has a problem: courier is not just imap, but also a pop3 server. Also http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;h... says, that "Applications are grouped by their respective function and each application gets a single file that lists all the patterns of that application." So it would be more appropriate to rename imapd.pdb to wu-imapd.pdb (as it would only have patterns for Washington University IMAPd), and have a separate courier.pdb, or even courier-imapd.pdb and courier-pop3d.pdb. What do you think?
you should use separate files based on the brand/name of the product and both should use imapd as the program name pattern. The rules will be merged internally by syslog-ng, but I guess the messages themselves are different enough to make a distinction. Or if not, hopefully the same-looking messages also serve the same purpose. Also please note that you can use multiple <pattern> tags for a given ruleset, e.g. <ruleset ...> <patterns> <pattern>imapd</pattern> <pattern>imapd-ssl</pattern> </patterns> </ruleset> Also, program names, just like messages use the longest prefix match, so as Matthew has put it, you can always use the longest unique prefix. And also, you can also use parsers within the program name pattern if that helps somehow. -- Bazsi
participants (3)
-
Balazs Scheidler
-
Matthew Hall
-
Peter Czanik