patterndb: collect login/logout samples
Hi, After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone. My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication. As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events. I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random). If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated. The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself. -- Bazsi
Hi, Not sure whether the following should be caught. This message is displayed when an unknown user attempts to log in: Jul 13 14:29:34 centos53 sshd[12779]: Failed password for invalid user xxxx from 127.0.0.1 port 40102 ssh2 When the DenyGroups and/or DenyUsers keywords for sshd are used to restrict access (for users in LDAP), the following messages are displayed for users that are not allowed to login: Jul 13 15:05:54 centos53 sshd[13031]: User siem from centos53 not allowed because listed in DenyUsers Jul 13 15:05:58 centos53 sshd[13031]: Failed password for invalid user siem from 127.0.0.1 port 53618 ssh2 and Jul 13 15:09:15 centos53 sshd[13061]: User siem from centos53 not allowed because a group is listed in DenyGroups Jul 13 15:09:22 centos53 sshd[13061]: Failed password for invalid user siem from 127.0.0.1 port 37397 ssh2 When the AllowGroups and/or AllowUsers keywords are used, the following messages are displayed: Jul 13 15:22:01 centos53 sshd[13155]: User siem from centos53 not allowed because not listed in AllowUsers Jul 13 15:22:05 centos53 sshd[13155]: Failed password for invalid user siem from 127.0.0.1 port 49085 ssh2 and Jul 13 15:23:48 centos53 sshd[13180]: User siem from centos53 not allowed because none of user's groups are listed in AllowGroups Jul 13 15:23:53 centos53 sshd[13180]: Failed password for invalid user siem from 127.0.0.1 port 33481 ssh2 regards, Siem Korteweg -----Oorspronkelijk bericht----- Van: syslog-ng-bounces@lists.balabit.hu namens Balazs Scheidler Verzonden: di 13-7-2010 13:25 Aan: syslog-ng@lists.balabit.hu Onderwerp: [syslog-ng] patterndb: collect login/logout samples Hi, After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone. My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication. As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events. I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random). If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated. The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself. -- Bazsi _____________________________________________________________________________ _ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi, Thanks, this information is very useful. I'll add them as patterndb rules into the current set. btw: it would probably also make sense to mark the status of individual rulesets, as the current version is really experimental. On Tue, 2010-07-13 at 15:29 +0200, Siem Korteweg wrote:
Hi,
Not sure whether the following should be caught.
This message is displayed when an unknown user attempts to log in:
Jul 13 14:29:34 centos53 sshd[12779]: Failed password for invalid user xxxx from 127.0.0.1 port 40102 ssh2
When the DenyGroups and/or DenyUsers keywords for sshd are used to restrict access (for users in LDAP), the following messages are displayed for users that are not allowed to login:
Jul 13 15:05:54 centos53 sshd[13031]: User siem from centos53 not allowed because listed in DenyUsers Jul 13 15:05:58 centos53 sshd[13031]: Failed password for invalid user siem from 127.0.0.1 port 53618 ssh2
and
Jul 13 15:09:15 centos53 sshd[13061]: User siem from centos53 not allowed because a group is listed in DenyGroups Jul 13 15:09:22 centos53 sshd[13061]: Failed password for invalid user siem from 127.0.0.1 port 37397 ssh2
When the AllowGroups and/or AllowUsers keywords are used, the following messages are displayed:
Jul 13 15:22:01 centos53 sshd[13155]: User siem from centos53 not allowed because not listed in AllowUsers Jul 13 15:22:05 centos53 sshd[13155]: Failed password for invalid user siem from 127.0.0.1 port 49085 ssh2
and
Jul 13 15:23:48 centos53 sshd[13180]: User siem from centos53 not allowed because none of user's groups are listed in AllowGroups Jul 13 15:23:53 centos53 sshd[13180]: Failed password for invalid user siem from 127.0.0.1 port 33481 ssh2
-- Bazsi
Here's one for an Apache basic auth failure on SLES 10 with the default Apache log format: [Mon Jul 12 08:55:22 2010] [error] [client 10.10.66.7] user xxxx: authentication failure for "/": Password Mismatch On Tue, Jul 13, 2010 at 10:01 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
Hi,
Thanks, this information is very useful. I'll add them as patterndb rules into the current set.
btw: it would probably also make sense to mark the status of individual rulesets, as the current version is really experimental.
On Tue, 2010-07-13 at 15:29 +0200, Siem Korteweg wrote:
Hi,
Not sure whether the following should be caught.
This message is displayed when an unknown user attempts to log in:
Jul 13 14:29:34 centos53 sshd[12779]: Failed password for invalid user xxxx from 127.0.0.1 port 40102 ssh2
When the DenyGroups and/or DenyUsers keywords for sshd are used to restrict access (for users in LDAP), the following messages are displayed for users that are not allowed to login:
Jul 13 15:05:54 centos53 sshd[13031]: User siem from centos53 not allowed because listed in DenyUsers Jul 13 15:05:58 centos53 sshd[13031]: Failed password for invalid user siem from 127.0.0.1 port 53618 ssh2
and
Jul 13 15:09:15 centos53 sshd[13061]: User siem from centos53 not allowed because a group is listed in DenyGroups Jul 13 15:09:22 centos53 sshd[13061]: Failed password for invalid user siem from 127.0.0.1 port 37397 ssh2
When the AllowGroups and/or AllowUsers keywords are used, the following messages are displayed:
Jul 13 15:22:01 centos53 sshd[13155]: User siem from centos53 not allowed because not listed in AllowUsers Jul 13 15:22:05 centos53 sshd[13155]: Failed password for invalid user siem from 127.0.0.1 port 49085 ssh2
and
Jul 13 15:23:48 centos53 sshd[13180]: User siem from centos53 not allowed because none of user's groups are listed in AllowGroups Jul 13 15:23:53 centos53 sshd[13180]: Failed password for invalid user siem from 127.0.0.1 port 33481 ssh2
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Tue, 2010-07-13 at 10:35 -0500, Martin Holste wrote:
Here's one for an Apache basic auth failure on SLES 10 with the default Apache log format:
[Mon Jul 12 08:55:22 2010] [error] [client 10.10.66.7] user xxxx: authentication failure for "/": Password Mismatch
I've created an apache2.pdb file under 'file-service' directory (but I'm open for suggestion regarding the directory name) and added your pattern as: + <rule provider="patterndb" id="5402ccee-d854-4f1e-877c-3c9332b6cc0e" class="system"> + <patterns> + <pattern>[error] [client @ESTRING:usracct.device:]@ user @ESTRING:usracct.username::@ authentication failure for @QSTRING:usracct.object:"@: @ANYSTRING:details@</pattern> + </patterns> + <examples> + <example> + <test_message program="sshd">[error] [client 10.10.66.7] user xxxx: authentication failure for "/": Password Mismatch</test_message> + <test_values> + <test_value name="usracct.username">xxxx</test_value> + <test_value name="usracct.device">10.10.66.7</test_value> + <test_value name="usracct.service">http</test_value> + <test_value name="usracct.object">/</test_value> + <test_value name="details">Password Mismatch</test_value> + </test_values> + </example> + </examples> + <values> + <value name="usracct.type">login</value> + <value name="usracct.application">$PROGRAM</value> + <value name="secevt.verdict">REJECT</value> + </values> + <tags> + <tag>usracct</tag> + <tag>secevt</tag> + </tags> + </rule> This sample was very good, because: 1) I recognized that an "object" might be needed in the usracct schema to describe the object being accessed 2) I've found a bug in "pdbtool match --debug-pattern", fixed in OSE 3.2 tree 3) I noted that once I start adding an apache2 configuration snippet to SCL we need to make sure that the timestamp is not included in the message. My solution not to include the timestamp was that apache2 doesn't include that when it directly uses syslog(). However when reading the apache2 log files directly, it is there. I'm not sure how to handle this properly from within SCL right now, but I'll find a way to do that. -- Bazsi
Hi, On Tue, 2010-07-13 at 15:29 +0200, Siem Korteweg wrote:
Hi,
Not sure whether the following should be caught.
This message is displayed when an unknown user attempts to log in:
Jul 13 14:29:34 centos53 sshd[12779]: Failed password for invalid user xxxx from 127.0.0.1 port 40102 ssh2
When the DenyGroups and/or DenyUsers keywords for sshd are used to restrict access (for users in LDAP), the following messages are displayed for users that are not allowed to login:
Jul 13 15:05:54 centos53 sshd[13031]: User siem from centos53 not allowed because listed in DenyUsers Jul 13 15:05:58 centos53 sshd[13031]: Failed password for invalid user siem from 127.0.0.1 port 53618 ssh2
and
Jul 13 15:09:15 centos53 sshd[13061]: User siem from centos53 not allowed because a group is listed in DenyGroups Jul 13 15:09:22 centos53 sshd[13061]: Failed password for invalid user siem from 127.0.0.1 port 37397 ssh2
Are both of these logged when such an event occurs? Because if it does, then a single pattern (the 2nd line) covers both, right?
When the AllowGroups and/or AllowUsers keywords are used, the following messages are displayed:
Jul 13 15:22:01 centos53 sshd[13155]: User siem from centos53 not allowed because not listed in AllowUsers Jul 13 15:22:05 centos53 sshd[13155]: Failed password for invalid user siem from 127.0.0.1 port 49085 ssh2
and
Jul 13 15:23:48 centos53 sshd[13180]: User siem from centos53 not allowed because none of user's groups are listed in AllowGroups Jul 13 15:23:53 centos53 sshd[13180]: Failed password for invalid user siem from 127.0.0.1 port 33481 ssh2
Again, from the login/logout/failure point of view, the "invalid user" log message grasps the event of an login failure. The other two messages contain additional details about the upcoming message though but in order to connect the two an additional correllation step would need to be performed, which is not in scope right now. Here's the pattern I've added based on your sample: + <rule provider="patterndb" id="1a8891ff-6b86-4da5-b937-b789c76ef353" class="system"> + <patterns> + <pattern>Failed @ESTRING:usracct.authmethod: @for invalid user @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern> + </patterns> + <examples> + <example> + <test_message program="sshd">Failed password for invalid user siem from 127.0.0.1 port 37397 ssh2</test_message> + <test_values> + <test_value name="usracct.username">siem</test_value> + <test_value name="usracct.authmethod">password</test_value> + <test_value name="usracct.device">127.0.0.1</test_value> + <test_value name="usracct.service">ssh2</test_value> + </test_values> + </example> + </examples> + <values> + <value name="usracct.type">login</value> + <value name="usracct.sessionid">$PID</value> + <value name="usracct.application">$PROGRAM</value> + <value name="secevt.verdict">REJECT</value> + </values> + <tags> + <tag>usracct</tag> + <tag>secevt</tag> + </tags> + </rule> -- Bazsi
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
OK, so here are some: OS Linux SSH bad pwd Apr 22 16:56:39 support sshd[11354]: Failed password for root from ::ffff:10.10.10.4 port 4027 ssh2 bad user Apr 22 13:41:22 support sshd[11320]: Failed password for illegal user admin from ::ffff:10.10.10.135 port 45629 ssh2 FTP bad pwd Apr 23 14:07:49 support sshd[15069]: Failed password for ftp from ::ffff:10.10.10.171 port 35621 ssh2 OS HP-UX bad pwd Mar 12 08:24:51 server6 sshd[24742]: Failed password for john from 10.10.333.444 port 1420 ssh2 Web Apache 401 10.10.10.100 - - [23/Apr/2007:12:29:55 -0500] "GET /olu/adm/reg.html HTTP/1.1" 401 485 Is login success next, hopefully? -- Dr. Anton Chuvakin Site: http://www.chuvakin.org Blog: http://www.securitywarrior.org LinkedIn: http://www.linkedin.com/in/chuvakin Consulting: http://www.securitywarriorconsulting.com Twitter: @anton_chuvakin Google Voice: +1-510-771-7106
On Tue, 2010-07-13 at 12:37 -0700, Anton Chuvakin wrote:
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
OK, so here are some:
OS Linux SSH bad pwd Apr 22 16:56:39 support sshd[11354]: Failed password for root from ::ffff:10.10.10.4 port 4027 ssh2 bad user Apr 22 13:41:22 support sshd[11320]: Failed password for illegal user admin from ::ffff:10.10.10.135 port 45629 ssh2 FTP bad pwd Apr 23 14:07:49 support sshd[15069]: Failed password for ftp from ::ffff:10.10.10.171 port 35621 ssh2
OS HP-UX bad pwd Mar 12 08:24:51 server6 sshd[24742]: Failed password for john from 10.10.333.444 port 1420 ssh2
Web Apache 401 10.10.10.100 - - [23/Apr/2007:12:29:55 -0500] "GET /olu/adm/reg.html HTTP/1.1" 401 485
Is login success next, hopefully?
Ahh, I might have put the wording wrong. I've meant login AND logout and login failure. So let those coming as well. Great to receive these patterns. I really appreciate them. I hope to get your submissions into shape hopefully today, but worst case this week. -- Bazsi
Here's SSH with a successful public key login and subsequent logout: Jul 4 12:28:27 webserver0163 sshd[22134]: Accepted publickey for johnny from 10.10.85.208 port 50674 ssh2 Jul 4 12:28:28 webserver0163 sshd[22136]: Received disconnect from 10.10.85.208: 11: disconnected by user On Wed, Jul 14, 2010 at 2:43 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2010-07-13 at 12:37 -0700, Anton Chuvakin wrote:
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
OK, so here are some:
OS Linux SSH bad pwd Apr 22 16:56:39 support sshd[11354]: Failed password for root from ::ffff:10.10.10.4 port 4027 ssh2 bad user Apr 22 13:41:22 support sshd[11320]: Failed password for illegal user admin from ::ffff:10.10.10.135 port 45629 ssh2 FTP bad pwd Apr 23 14:07:49 support sshd[15069]: Failed password for ftp from ::ffff:10.10.10.171 port 35621 ssh2
OS HP-UX bad pwd Mar 12 08:24:51 server6 sshd[24742]: Failed password for john from 10.10.333.444 port 1420 ssh2
Web Apache 401 10.10.10.100 - - [23/Apr/2007:12:29:55 -0500] "GET /olu/adm/reg.html HTTP/1.1" 401 485
Is login success next, hopefully?
Ahh, I might have put the wording wrong. I've meant login AND logout and login failure.
So let those coming as well.
Great to receive these patterns. I really appreciate them. I hope to get your submissions into shape hopefully today, but worst case this week.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2010-07-14 at 08:25 -0500, Martin Holste wrote:
Here's SSH with a successful public key login and subsequent logout:
Jul 4 12:28:27 webserver0163 sshd[22134]: Accepted publickey for johnny from 10.10.85.208 port 50674 ssh2
This is covered already.
Jul 4 12:28:28 webserver0163 sshd[22136]: Received disconnect from 10.10.85.208: 11: disconnected by user
I was using the pam_unix event for this, because that contained more information, and I didn't want to have multiple logout events for the same session. pam_unix(sshd:session): session closed for user bazsi Hmm.. although the more I think of it, this would possibly mean that on non-PAM platforms this message is logged and the other isn't. Hmm, hmm. Also, this one comes from the "parent" sshd with a different pid than the login message, so no way to pair it with the login information. I'd leave it as is, but let me know if you have a better idea. -- Bazsi
On Tue, 2010-07-13 at 12:37 -0700, Anton Chuvakin wrote:
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
OK, so here are some:
OS Linux SSH bad pwd Apr 22 16:56:39 support sshd[11354]: Failed password for root from ::ffff:10.10.10.4 port 4027 ssh2
this was covered by the already existing patterns, though it was nice to see that they indeed worked with IPv6. I've added this as an example to test the pattern with.
bad user Apr 22 13:41:22 support sshd[11320]: Failed password for illegal user admin from ::ffff:10.10.10.135 port 45629 ssh2
this was a different incarnation of "invalid user" of the previous poster, probably this message was changed within sshd. Added as a separate rule.
FTP bad pwd Apr 23 14:07:49 support sshd[15069]: Failed password for ftp from ::ffff:10.10.10.171 port 35621 ssh2
already covered.
OS HP-UX bad pwd Mar 12 08:24:51 server6 sshd[24742]: Failed password for john from 10.10.333.444 port 1420 ssh2
also covered.
Web Apache 401 10.10.10.100 - - [23/Apr/2007:12:29:55 -0500] "GET /olu/adm/reg.html HTTP/1.1" 401 485
I've a trouble with this one, login failures in Apache are logged to the error log, it seems to be a better source than scanning the access.log for the 401 status, especially as it is normal part of the protocol and not necessarily an immediate login failure. I don't see that access.log should be going through patterndb as it is already structured. Using a csv-parser to read that would probably be easier, probably good candidate for creating an SCL block to do just that. -- Bazsi
Hey Guys, I'm way behind on adoption of the whole patterndb thing, but if you're looking for log samples, here's a good resource: http://www.ossec.net/wiki/Log_Samples ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Thu, Jul 15, 2010 at 3:26 PM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2010-07-13 at 12:37 -0700, Anton Chuvakin wrote:
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
OK, so here are some:
OS Linux SSH bad pwd Apr 22 16:56:39 support sshd[11354]: Failed password for root from ::ffff:10.10.10.4 port 4027 ssh2
this was covered by the already existing patterns, though it was nice to see that they indeed worked with IPv6. I've added this as an example to test the pattern with.
bad user Apr 22 13:41:22 support sshd[11320]: Failed password for illegal user admin from ::ffff:10.10.10.135 port 45629 ssh2
this was a different incarnation of "invalid user" of the previous poster, probably this message was changed within sshd. Added as a separate rule.
FTP bad pwd Apr 23 14:07:49 support sshd[15069]: Failed password for ftp from ::ffff:10.10.10.171 port 35621 ssh2
already covered.
OS HP-UX bad pwd Mar 12 08:24:51 server6 sshd[24742]: Failed password for john from 10.10.333.444 port 1420 ssh2
also covered.
Web Apache 401 10.10.10.100 - - [23/Apr/2007:12:29:55 -0500] "GET /olu/adm/reg.html HTTP/1.1" 401 485
I've a trouble with this one, login failures in Apache are logged to the error log, it seems to be a better source than scanning the access.log for the 401 status, especially as it is normal part of the protocol and not necessarily an immediate login failure.
I don't see that access.log should be going through patterndb as it is already structured. Using a csv-parser to read that would probably be easier, probably good candidate for creating an SCL block to do just that.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2010-07-15 at 15:54 -0400, Clayton Dukes wrote:
Hey Guys, I'm way behind on adoption of the whole patterndb thing, but if you're looking for log samples, here's a good resource: http://www.ossec.net/wiki/Log_Samples
Thanks. Great pointer. -- Bazsi
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
Some logouts + session ended's too: Jul 11 08:09:01 anton-linux CRON[24475]: pam_unix(cron:session): session closed for user root Apr 28 03:34:36 esx1 sshd(pam_unix)[9032]: session closed for user anton Just for fun: VMWare ESX login success Apr 27 01:01:12 esx1 /usr/lib/vmware/hostd/vmware-hostd[1479]: Accepted password for user root from 127.0.0.1 Will send more as I dig thru my archives... -- Dr. Anton Chuvakin Site: http://www.chuvakin.org Blog: http://www.securitywarrior.org LinkedIn: http://www.linkedin.com/in/chuvakin Consulting: http://www.securitywarriorconsulting.com Twitter: @anton_chuvakin Google Voice: +1-510-771-7106
On Tue, 2010-07-13 at 12:47 -0700, Anton Chuvakin wrote:
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
Some logouts + session ended's too:
Jul 11 08:09:01 anton-linux CRON[24475]: pam_unix(cron:session): session closed for user root
This is a cron message, not an sshd message, so not strictly a user login/logout, though it could be interpreted as such.
Apr 28 03:34:36 esx1 sshd(pam_unix)[9032]: session closed for user anton
gee, reusing the program field, just to make it more difficult. This means that we'd need several patterns for the program name field. Not difficult, just another reason to adjust the patterndb format.
Just for fun:
VMWare ESX login success
Apr 27 01:01:12 esx1 /usr/lib/vmware/hostd/vmware-hostd[1479]: Accepted password for user root from 127.0.0.1
Nice. Thanks a lot, I'll add this somewhat later. I got distracted by other things. -- Bazsi
On Thu, 2010-07-15 at 16:56 +0200, Balazs Scheidler wrote:
On Tue, 2010-07-13 at 12:47 -0700, Anton Chuvakin wrote:
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
Some logouts + session ended's too:
Jul 11 08:09:01 anton-linux CRON[24475]: pam_unix(cron:session): session closed for user root
This is a cron message, not an sshd message, so not strictly a user login/logout, though it could be interpreted as such.
Apr 28 03:34:36 esx1 sshd(pam_unix)[9032]: session closed for user anton
gee, reusing the program field, just to make it more difficult. This means that we'd need several patterns for the program name field. Not difficult, just another reason to adjust the patterndb format.
Talked to Marci about this one. patterndb seems to do a prefix match, so our 'sshd' rule will match just fine. Anyway, the ability to specify multiple patterns for the ruleset will probably be needed. Also, if the pam_unix part is not in the message, but rather in the program name field, then we need to add this as a separate rule. Here it comes: + <rule provider="patterndb" id="a2f96b71-6c5e-413e-92c2-75e9d66c0119" class="system"> + <patterns> + <pattern>session closed for user @ANYSTRING:usracct.username:@</pattern> + </patterns> + <examples> + <example> + <test_message program="sshd(pam_unix)">session closed for user bazsi</test_message> + <test_values> + <test_value name="usracct.username">bazsi</test_value> + </test_values> + </example> + </examples> + <values> + <value name="usracct.type">logout</value> + <value name="usracct.sessionid">$PID</value> + <value name="usracct.application">$PROGRAM</value> + </values> + <tags> + <tag>usracct</tag> + </tags> + </rule>
Just for fun:
VMWare ESX login success
Apr 27 01:01:12 esx1 /usr/lib/vmware/hostd/vmware-hostd[1479]: Accepted password for user root from 127.0.0.1
Nice.
Thanks a lot, I'll add this somewhat later. I got distracted by other things.
I've added this too to vm/vmware-esx.pdb Do you perhaps have the logout & login failure messages for this? Thanks. -- Bazsi
Sent: Tuesday, July 13, 2010 5:25:13 AM From: Balazs Scheidler <bazsi@balabit.hu> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] patterndb: collect login/logout samples
Hi,
After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone.
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
I took a look at that pdb format and was lost. I'll probably learn it eventually, but would just make a mess of it if I tried now. But here are a lot of examples that havent been provided yet. All messages were generated from RHEL 5 servers ssh netgroup restricted login (user is valid): Jul 13 22:58:35 slider.dev.usa.net sshd[16563]: Invalid user phemmer from 165.212.225.134 Jul 13 22:58:35 slider.dev.usa.net sshd[16563]: Failed none for invalid user phemmer from 165.212.225.134 port 49528 ssh2 ssh tcpwrapper (/etc/hosts.deny) restricted login: Jul 13 23:02:57 admin02.cms.usa.net sshd[7442]: refused connect from 165.212.15.221 (165.212.15.221) ------------------- su valid login: Jul 13 22:47:07 admin02.cms.usa.net su: pam_unix(su:session): session opened for user root by phemmer(uid=8129) su bad pass: Jul 13 22:31:07 admin02.cms.usa.net su: pam_unix(su:auth): authentication failure; logname=phemmer uid=8129 euid=0 tty=pts/13 ruser=phemmer rhost= user=root su bad user generates no message su log out: Jul 13 23:07:13 admin02.cms.usa.net su: pam_unix(su:session): session closed for user root ------------------- sudo valid login: Jul 13 22:46:46 : phemmer : HOST=admin02 : TTY=pts/13 ; PWD=/home/phemmer ; USER=root ; COMMAND=/bin/ls sudo bad pass: Jul 13 22:33:53 admin02.cms.usa.net sudo: pam_unix(sudo:auth): authentication failure; logname=phemmer uid=0 euid=0 tty=/dev/pts/13 ruser= rhost= user=phemmer Jul 13 22:34:05 admin02.cms.usa.net sudo: phemmer : 3 incorrect password attempts ; TTY=pts/13 ; PWD=/home/phemmer ; USER=root ; COMMAND=/bin/ls sudo bad user: Jul 13 22:41:13 admin02.cms.usa.net sudo: phemmer : no passwd entry for asdfh! ------------------- serial console valid login: Jul 13 22:46:02 admin02.cms.usa.net login: pam_unix(login:session): session opened for user root by LOGIN(uid=0) Jul 13 22:46:02 admin02.cms.usa.net login: DIALUP AT ttyS1 BY root Jul 13 22:46:02 admin02.cms.usa.net login: ROOT LOGIN ON ttyS1 serial console bad pass: Jul 13 22:38:34 admin02.cms.usa.net login: FAILED LOGIN 1 FROM (null) FOR root, Authentication failure serial console bad user: Jul 13 22:38:56 admin02.cms.usa.net login: pam_unix(login:auth): check pass; user unknown Jul 13 22:38:56 admin02.cms.usa.net login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=ttyS1 ruser= rhost= Jul 13 22:38:56 admin02.cms.usa.net login: pam_succeed_if(login:auth): error retrieving information about user asdfjh Jul 13 22:38:57 admin02.cms.usa.net login: FAILED LOGIN 2 FROM (null) FOR asdfjh, User not known to the underlying authentication module serial console logout: Jul 13 23:06:29 admin02.cms.usa.net login: pam_unix(login:session): session closed for user root ------------------- physical console valid login: Jul 13 22:42:54 localhost login: ROOT LOGIN ON tty1 physical console bad pass: Jul 13 22:44:30 localhost login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=root Jul 13 22:44:32 localhost login: FAILED LOGIN 1 FROM (null) FOR root, Authentication failure physical console bad user: Jul 13 22:44:57 localhost login: pam_unix(login:auth): check pass; user unknown Jul 13 22:44:57 localhost login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= Jul 13 22:44:57 localhost login: pam_succeed_if(login:auth): error retrieving information about user shdga Jul 13 22:44:59 localhost login: FAILED LOGIN 2 FROM (null) FOR shdga, User not known to the underlying authentication module physical console logout: Jul 13 23:08:28 localhost login: pam_unix(login:session): session closed for user root ------------------- VMware server messages are the exact same for both remote console application and web UI. vmware server valid login: Jul 13 22:53:49 vmware02 Hostd: Accepted password for user root from 127.0.0.1 Jul 13 22:53:49 vmware02 Hostd: [2010-07-13 22:53:49.705 'Vimsvc' 1098422592 info] [Auth]: User root Jul 13 22:53:49 vmware02 Hostd: [2010-07-13 22:53:49.706 'ha-eventmgr' 1098422592 info] Event 3 : User root@127.0.0.1 logged in Jul 13 22:53:49 vmware02 Hostd: [2010-07-13 22:53:49.706 'PropertyProvider' 1098422592 verbose] RecordOp ASSIGN: latestEvent, ha-eventmgr Jul 13 22:53:49 vmware02 Hostd: [2010-07-13 22:53:49.706 'PropertyProvider' 1098422592 verbose] RecordOp ADD: sessionList["52efdf57-6fa9-a095-a7d3-48ef63421e73"], ha-sessionmgr vmware server bad user: Jul 13 22:53:15 vmware02 Hostd: [2010-07-13 22:53:15.677 'ha-eventmgr' 47473126103232 info] Event 2 : Failed login attempt for asdf@127.0.0.1 Jul 13 22:53:15 vmware02 Hostd: [2010-07-13 22:53:15.677 'PropertyProvider' 47473126103232 verbose] RecordOp ASSIGN: latestEvent, ha-eventmgr Jul 13 22:53:15 vmware02 Hostd: Rejected password for user asdf from 127.0.0.1 Jul 13 22:53:15 vmware02 Hostd: [2010-07-13 22:53:15.677 'Vmomi' 47473126103232 info] Activation [N5Vmomi10ActivationE:0xe5eedc0] : Invoke done [login] on [vim.SessionManager:ha-sessionmgr] Jul 13 22:53:15 vmware02 Hostd: [2010-07-13 22:53:15.678 'Vmomi' 47473126103232 info] Throw vim.fault.InvalidLogin Jul 13 22:53:15 vmware02 Hostd: [2010-07-13 22:53:15.678 'Vmomi' 47473126103232 info] Result: Jul 13 22:53:15 vmware02 Hostd: (vim.fault.InvalidLogin) { dynamicType = <unset>, msg = "" } Jul 13 22:53:15 vmware02 Hostd: vmware server bad pass: Jul 13 22:51:47 vmware02 Hostd: [2010-07-13 22:51:47.215 'ha-eventmgr' 1086609728 info] Event 1 : Failed login attempt for root@127.0.0.1 Jul 13 22:51:47 vmware02 Hostd: [2010-07-13 22:51:47.215 'PropertyProvider' 1086609728 verbose] RecordOp ASSIGN: latestEvent, ha-eventmgr Jul 13 22:51:47 vmware02 Hostd: Rejected password for user root from 127.0.0.1 Jul 13 22:51:47 vmware02 Hostd: [2010-07-13 22:51:47.216 'Vmomi' 1086609728 info] Activation [N5Vmomi10ActivationE:0xe5e3a80] : Invoke done [login] on [vim.SessionManager:ha-sessionmgr] Jul 13 22:51:47 vmware02 Hostd: [2010-07-13 22:51:47.216 'Vmomi' 1086609728 info] Throw vim.fault.InvalidLogin Jul 13 22:51:47 vmware02 Hostd: [2010-07-13 22:51:47.216 'Vmomi' 1086609728 info] Result: Jul 13 22:51:47 vmware02 Hostd: (vim.fault.InvalidLogin) { dynamicType = <unset>, msg = "" } Jul 13 22:51:47 vmware02 Hostd: vmware server no permissions: Jul 13 22:54:27 vmware02 Hostd: Accepted password for user phemmer from 127.0.0.1 Jul 13 22:54:27 vmware02 Hostd: [2010-07-13 22:54:27.905 'Vimsvc' 1098688832 info] [Auth]: User phemmer Jul 13 22:54:27 vmware02 Hostd: [2010-07-13 22:54:27.906 'ha-eventmgr' 1098688832 info] Event 4 : Failed to login user phemmer@127.0.0.1: No permission Jul 13 22:54:27 vmware02 Hostd: [2010-07-13 22:54:27.906 'PropertyProvider' 1098688832 verbose] RecordOp ASSIGN: latestEvent, ha-eventmgr Jul 13 22:54:27 vmware02 Hostd: [2010-07-13 22:54:27.906 'Vmomi' 1098688832 info] Activation [N5Vmomi10ActivationE:0xe86bd80] : Invoke done [login] on [vim.SessionManager:ha-sessionmgr] Jul 13 22:54:27 vmware02 Hostd: [2010-07-13 22:54:27.907 'Vmomi' 1098688832 info] Throw vim.fault.NoPermission Jul 13 22:54:27 vmware02 Hostd: [2010-07-13 22:54:27.907 'Vmomi' 1098688832 info] Result: Jul 13 22:54:27 vmware02 Hostd: (vim.fault.NoPermission) { dynamicType = <unset>, object = 'vim.Folder:ha-folder-root', privilegeId = "System.View", msg = "" } Jul 13 22:54:27 vmware02 Hostd:
On Tue, 2010-07-13 at 17:12 -0600, Patrick H. wrote:
Sent: Tuesday, July 13, 2010 5:25:13 AM From: Balazs Scheidler <bazsi@balabit.hu> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] patterndb: collect login/logout samples
Hi,
After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone.
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
I took a look at that pdb format and was lost. I'll probably learn it eventually, but would just make a mess of it if I tried now. But here are a lot of examples that havent been provided yet. All messages were generated from RHEL 5 servers
ssh netgroup restricted login (user is valid): Jul 13 22:58:35 slider.dev.usa.net sshd[16563]: Invalid user phemmer from 165.212.225.134 Jul 13 22:58:35 slider.dev.usa.net sshd[16563]: Failed none for invalid user phemmer from 165.212.225.134 port 49528 ssh2
we're using the 2nd log message to identify the login event, the first is just additional information, that would need to be associated with the 2nd via correllation, that we don't have right now. The 2nd form however is covered with the already existing rules.
ssh tcpwrapper (/etc/hosts.deny) restricted login: Jul 13 23:02:57 admin02.cms.usa.net sshd[7442]: refused connect from 165.212.15.221 (165.212.15.221)
This is interesting, however it is not a login event. It is more like a firewall event (e.g. flowevt + secevt in the current schema model), however port information is missing, so it doesn't contain the complete tuple. Anyway, it could perhaps be possible to categorize this under the flowevt schema, but I don't want to open that can of worms yet :)
-------------------
su valid login: Jul 13 22:47:07 admin02.cms.usa.net su: pam_unix(su:session): session opened for user root by phemmer(uid=8129)
Jul 13 22:54:27 vmware02 Hostd:
thanks, these are useful, I just need to get some sleep now. Will get these marked up tomorrow. -- Bazsi
On Tue, 2010-07-13 at 17:12 -0600, Patrick H. wrote:
Sent: Tuesday, July 13, 2010 5:25:13 AM From: Balazs Scheidler <bazsi@balabit.hu> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] patterndb: collect login/logout samples
Hi,
After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone.
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
I took a look at that pdb format and was lost. I'll probably learn it eventually, but would just make a mess of it if I tried now. But here are a lot of examples that havent been provided yet. All messages were generated from RHEL 5 servers
ssh netgroup restricted login (user is valid): Jul 13 22:58:35 slider.dev.usa.net sshd[16563]: Invalid user phemmer from 165.212.225.134 Jul 13 22:58:35 slider.dev.usa.net sshd[16563]: Failed none for invalid user phemmer from 165.212.225.134 port 49528 ssh2
ssh tcpwrapper (/etc/hosts.deny) restricted login: Jul 13 23:02:57 admin02.cms.usa.net sshd[7442]: refused connect from 165.212.15.221 (165.212.15.221)
-------------------
su valid login: Jul 13 22:47:07 admin02.cms.usa.net su: pam_unix(su:session): session opened for user root by phemmer(uid=8129)
su bad pass: Jul 13 22:31:07 admin02.cms.usa.net su: pam_unix(su:auth): authentication failure; logname=phemmer uid=8129 euid=0 tty=pts/13 ruser=phemmer rhost= user=root
su bad user generates no message
su log out: Jul 13 23:07:13 admin02.cms.usa.net su: pam_unix(su:session): session closed for user root
Thanks for your submission. I've added su events to: commit 5e38f9dab2a89e8839829f7740485784accb3baa Author: Balazs Scheidler <bazsi@balabit.hu> Date: Mon Jul 26 18:01:27 2010 +0200 su: added su login/logout/failure rules This patch covers su on Linux with PAM. Submitted-By: Patrick H. The others I still have to mark up. Anyone who could perhaps give a hand at marking up the patterns that Patrick submitted? Would be appreciated. -- Bazsi
On Tue, 2010-07-13 at 17:12 -0600, Patrick H. wrote:
Sent: Tuesday, July 13, 2010 5:25:13 AM From: Balazs Scheidler <bazsi@balabit.hu> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] patterndb: collect login/logout samples
Hi,
After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone.
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
I took a look at that pdb format and was lost. I'll probably learn it eventually, but would just make a mess of it if I tried now. But here are a lot of examples that havent been provided yet. All messages were generated from RHEL 5 servers
[snip]
VMware server messages are the exact same for both remote console application and web UI.
Thanks for all the samples that you sent, I've finished marked up all of these, except for the vmware stuff, but I hope to finish that in the coming days. This means that I've almost finished with everything that anyone posted to this list so far. Thanks for everyone. -- Bazsi
Hi, I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package. I have fixed the sshd.pdb example messages and extended them to check for name/value pairs as well. I have also modified to patterns to use the ESTRING/ANYSTRING parsers instead of the STRING/IPv4/NUMBER parsers as the previous ones are faster, and they should be used when possible. The STRING/IPv4/NUMBER parser parse the message char by char, while the ESTRING/QSTRING parsers are looking for an stop character/string and parse everything till than. The ANYSTRING parser on the other hand simply parses everything till the end of the message, so it is handy to parse the rest of the message into a name/value pair. The QSTRING/ESTRING parsers are especially useful when the type of the parsed part is not important, eg: we do not want to handle an ip address or a number specially later. (This was the case in the sshd messages, though it might make sense to extend the policy to define which parser should be used in some cases...) Bazsi, please pull my tree: Marton Illes (2): added test-patterns.py script to test the patterns with the example log messages access/sshd.pdb: fixed example messages and added test_values access/sshd.pdb: use ESTRING/ANYSTRING parser instead of STRING/IPv4/NUMBER Marci On Tue, 2010-07-13 at 13:25 +0200, Balazs Scheidler wrote:
Hi,
After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone.
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events.
I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random).
If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated.
The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself.
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
On Thu, 2010-07-15 at 14:16 +0200, ILLES, Marton wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
I have fixed the sshd.pdb example messages and extended them to check for name/value pairs as well.
I have also modified to patterns to use the ESTRING/ANYSTRING parsers instead of the STRING/IPv4/NUMBER parsers as the previous ones are faster, and they should be used when possible. The STRING/IPv4/NUMBER parser parse the message char by char, while the ESTRING/QSTRING parsers are looking for an stop character/string and parse everything till than. The ANYSTRING parser on the other hand simply parses everything till the end of the message, so it is handy to parse the rest of the message into a name/value pair. The QSTRING/ESTRING parsers are especially useful when the type of the parsed part is not important, eg: we do not want to handle an ip address or a number specially later. (This was the case in the sshd messages, though it might make sense to extend the policy to define which parser should be used in some cases...)
Bazsi, please pull my tree:
Marton Illes (2): added test-patterns.py script to test the patterns with the example log messages access/sshd.pdb: fixed example messages and added test_values access/sshd.pdb: use ESTRING/ANYSTRING parser instead of STRING/IPv4/NUMBER
I've just did that. Thanks Marci. -- Bazsi
On Thu, 2010-07-15 at 14:46 +0200, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:16 +0200, ILLES, Marton wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
Could you be more specific which XML package is needed? I've tried everything, but none of them worked with the script. -- Bazsi
On 07/15/2010 03:01 PM, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:46 +0200, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:16 +0200, ILLES, Marton wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
Could you be more specific which XML package is needed? I've tried everything, but none of them worked with the script.
The old python-xml package from jaunty would be needed but is not available for recent distros. I refactored it to use python-lxml, please find the patch attached. I also ran into some UTF-8 related output trouble that the other patch fixes. Both tested on Ubuntu Lucid and strictly WORKSFORME. Balint
It worked for me as well, once I installed python-lxml (on Ubuntu lucid). Anyone wishing to test for the embedded examples, just use: bin/test-patterns.py <pathtopdb> <pathtopdbtool> I've used the pdbtool in 3.2, but it should work with the one in 3.1 as well. Applied both. Thanks Bálint. On Thu, 2010-07-15 at 15:51 +0200, Balint Kovacs wrote:
On 07/15/2010 03:01 PM, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:46 +0200, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:16 +0200, ILLES, Marton wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
Could you be more specific which XML package is needed? I've tried everything, but none of them worked with the script.
The old python-xml package from jaunty would be needed but is not available for recent distros. I refactored it to use python-lxml, please find the patch attached. I also ran into some UTF-8 related output trouble that the other patch fixes. Both tested on Ubuntu Lucid and strictly WORKSFORME.
-- Bazsi
Hi, As this python xml is a real problem, i re-implemented the script in pdbtool, so no extra script or library is required. You can grab it from my git tree: http://git.balabit.hu/?p=marci/syslog-ng-3.2.git;a=commit;h=fc01838593d18225... Bazsi please grab it. Now you can simply use "pdbtool test" to test your favourite patterndb! :) An example output: $ pdbtool test -p sshd.pdb -v ; echo $? Testing message program='sshd' message='Failed password for bazsi from 127.0.1.1 port 44637 ssh2' Match name='.classifier.rule_id', value='aecda233-3d80-48cd-a72b-4896f58069c8', expected='aecda233-3d80-48cd-a72b-4896f58069c8' Match name='usracct.username', value='bazsi', expected='bazsi' Match name='usracct.authmethod', value='password', expected='password' Match name='usracct.device', value='127.0.1.1', expected='127.0.1.1' Match name='usracct.service', value='ssh2', expected='ssh2' Testing message program='sshd' message='Accepted password for bazsi from 127.0.0.1 port 48650 ssh2' Match name='.classifier.rule_id', value='4dd5a329-da83-4876-a431-ddcb59c2858c', expected='4dd5a329-da83-4876-a431-ddcb59c2858c' Match name='usracct.username', value='bazsi', expected='bazsi' Match name='usracct.authmethod', value='password', expected='password' Match name='usracct.device', value='127.0.0.1', expected='127.0.0.1' Match name='usracct.service', value='ssh2', expected='ssh2' 0 Hope it helps. M On Thu, 2010-07-15 at 22:06 +0200, Balazs Scheidler wrote:
It worked for me as well, once I installed python-lxml (on Ubuntu lucid).
Anyone wishing to test for the embedded examples, just use:
bin/test-patterns.py <pathtopdb> <pathtopdbtool>
I've used the pdbtool in 3.2, but it should work with the one in 3.1 as well.
Applied both. Thanks Bálint.
On Thu, 2010-07-15 at 15:51 +0200, Balint Kovacs wrote:
On 07/15/2010 03:01 PM, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:46 +0200, Balazs Scheidler wrote:
On Thu, 2010-07-15 at 14:16 +0200, ILLES, Marton wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
Could you be more specific which XML package is needed? I've tried everything, but none of them worked with the script.
The old python-xml package from jaunty would be needed but is not available for recent distros. I refactored it to use python-lxml, please find the patch attached. I also ran into some UTF-8 related output trouble that the other patch fixes. Both tested on Ubuntu Lucid and strictly WORKSFORME.
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
Hi, On Thursday, July 15, 2010 14:16 CEST, "ILLES, Marton" <illes.marton@balabit.hu> wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
I have fixed the sshd.pdb example messages and extended them to check for name/value pairs as well.
I have also modified to patterns to use the ESTRING/ANYSTRING parsers instead of the STRING/IPv4/NUMBER parsers as the previous ones are faster, and they should be used when possible. The STRING/IPv4/NUMBER parser parse the message char by char, while the ESTRING/QSTRING parsers are looking for an stop character/string and parse everything till than. The ANYSTRING parser on the other hand simply parses everything till the end of the message, so it is handy to parse the rest of the message into
I was wondering about how the ANYSTRING parser would play together with the multiline message handling introduced in 3.2. Would it parse the message to the end of the message, or only to the end of the line? If it goes all the way to the end of the message, then another parser (or an optional parameter for ANYSTRING) that parses only to the end of the current line might be useful to properly handle multiline messages. Just a thought. Robert
a name/value pair. The QSTRING/ESTRING parsers are especially useful when the type of the parsed part is not important, eg: we do not want to handle an ip address or a number specially later. (This was the case in the sshd messages, though it might make sense to extend the policy to define which parser should be used in some cases...)
Bazsi, please pull my tree:
Marton Illes (2): added test-patterns.py script to test the patterns with the example log messages access/sshd.pdb: fixed example messages and added test_values access/sshd.pdb: use ESTRING/ANYSTRING parser instead of STRING/IPv4/NUMBER
Marci
On Tue, 2010-07-13 at 13:25 +0200, Balazs Scheidler wrote:
Hi,
After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone.
My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication.
As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events.
I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random).
If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated.
The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself.
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2010-07-15 at 20:37 +0200, Fekete Róbert wrote:
Hi, On Thursday, July 15, 2010 14:16 CEST, "ILLES, Marton" <illes.marton@balabit.hu> wrote:
Hi,
I took the liberty and created my own patterndb git tree, so i can track my patches there and Bazsi can easily pull from there. You can check it at git.balabit.hu: http://git.balabit.hu/?p=marci/syslog-ng-patterndb.git;a=summary
I have added a small python script test-patterns.py which can be used to automatically check the example messages against the patterns, while it also verifies the parsed name/value pairs. It is kind of a handy tool when you poke with the patterns and want to run automatic tests. It requires pdbtool and python xml package.
I have fixed the sshd.pdb example messages and extended them to check for name/value pairs as well.
I have also modified to patterns to use the ESTRING/ANYSTRING parsers instead of the STRING/IPv4/NUMBER parsers as the previous ones are faster, and they should be used when possible. The STRING/IPv4/NUMBER parser parse the message char by char, while the ESTRING/QSTRING parsers are looking for an stop character/string and parse everything till than. The ANYSTRING parser on the other hand simply parses everything till the end of the message, so it is handy to parse the rest of the message into
I was wondering about how the ANYSTRING parser would play together with the multiline message handling introduced in 3.2. Would it parse the message to the end of the message, or only to the end of the line? If it goes all the way to the end of the message, then another parser (or an optional parameter for ANYSTRING) that parses only to the end of the current line might be useful to properly handle multiline messages.
Just a thought.
It'd eat the string till the end of the file, you are right, it could be useful, but I'd like to wait for the first occurence of such a pattern. -- Bazsi
participants (9)
-
Anton Chuvakin
-
Balazs Scheidler
-
Balint Kovacs
-
Clayton Dukes
-
Fekete Róbert
-
ILLES, Marton
-
Martin Holste
-
Patrick H.
-
Siem Korteweg