Solving parsing problem
Hi guys, I have done parser for the authentication logs of HPUX. Here is the situation: A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng. The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=xxxx SucceedLogin_SOURCE.PORT=1691 The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown Here is the parser from xml file: <patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns> and here is the command pdbtool: /opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2" Please, advice me how to proceed. Thank you. Kosta
On Thu, 2010-09-02 at 13:12 +0000, otgovorete@gmail.com wrote:
Hi guys,
I have done parser for the authentication logs of HPUX. Here is the situation:
A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2
The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng.
The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=x.x.x.x SucceedLogin_SOURCE.PORT=1691
The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown
Here is the parser from xml file:
<patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns>
and here is the command pdbtool:
/opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2"
Please, advice me how to proceed. Thank you.
With -M you need to use pass only the $MSG portion of the syslog message. Alternatively you can use pdbtool match -f <filename> which will read and _parse_ syslog messages from the given file and only pass the payload for parsing. So, in your rule, you don't need the day/month macros. And if you really wanted to parse the date, you could perhaps use the @NUMBER@ parser. -- Bazsi
Cool, thanks for the pattern. So what tags would we use for this? I'm thinking you could tag with net, ssh, login, success. Also, what would be the conventional format for the class, rule, and field names? I think "useracct" had been the prefix before. On Thu, Sep 2, 2010 at 10:44 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2010-09-02 at 13:12 +0000, otgovorete@gmail.com wrote:
Hi guys,
I have done parser for the authentication logs of HPUX. Here is the situation:
A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2
The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng.
The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=x.x.x.x SucceedLogin_SOURCE.PORT=1691
The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown
Here is the parser from xml file:
<patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns>
and here is the command pdbtool:
/opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2"
Please, advice me how to proceed. Thank you.
With -M you need to use pass only the $MSG portion of the syslog message. Alternatively you can use pdbtool match -f <filename> which will read and _parse_ syslog messages from the given file and only pass the payload for parsing.
So, in your rule, you don't need the day/month macros.
And if you really wanted to parse the date, you could perhaps use the @NUMBER@ parser.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2010-09-02 at 10:51 -0500, Martin Holste wrote:
Cool, thanks for the pattern. So what tags would we use for this? I'm thinking you could tag with net, ssh, login, success. Also, what would be the conventional format for the class, rule, and field names? I think "useracct" had been the prefix before.
This pattern is already in patterndb's git tree, I just tried to help out with the specific problem. $ pdbtool match -p sshd.pdb -M "Accepted keyboard-interactive/pam for username from 1.2.3.4 port 1345 ssh2" -P sshd MESSAGE=Accepted keyboard-interactive/pam for username from 1.2.3.4 port 1345 ssh2 PROGRAM=sshd .classifier.class=system .classifier.rule_id=4dd5a329-da83-4876-a431-ddcb59c2858c usracct.authmethod=keyboard-interactive/pam usracct.username=username usracct.device=1.2.3.4 usracct.service=ssh2 usracct.type=login usracct.sessionid= usracct.application=sshd secevt.verdict=ACCEPT
On Thu, Sep 2, 2010 at 10:44 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2010-09-02 at 13:12 +0000, otgovorete@gmail.com wrote:
Hi guys,
I have done parser for the authentication logs of HPUX. Here is the situation:
A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2
The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng.
The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=x.x.x.x SucceedLogin_SOURCE.PORT=1691
The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown
Here is the parser from xml file:
<patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns>
and here is the command pdbtool:
/opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from x.x.x.x port 1691 ssh2"
Please, advice me how to proceed. Thank you.
With -M you need to use pass only the $MSG portion of the syslog message. Alternatively you can use pdbtool match -f <filename> which will read and _parse_ syslog messages from the given file and only pass the payload for parsing.
So, in your rule, you don't need the day/month macros.
And if you really wanted to parse the date, you could perhaps use the @NUMBER@ parser.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
On Thu, Sep 02, 2010 at 01:12:40PM +0000, otgovorete@gmail.com wrote:
The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng.
I have this problem pretty often myself unfortunately. If you just want to get it to work ASAP even if it's not the most elegant situation, just add @ANYSTRING::@ to the end of your parser, right after the last non whitespace character of the message and it will "eat up" the trailing chars to successfully process the string.
Please, advice me how to proceed. Thank you.
Hope this will help you out.
Kosta
Matthew.
On Thu, 2010-09-02 at 10:31 -0700, Matthew Hall wrote:
On Thu, Sep 02, 2010 at 01:12:40PM +0000, otgovorete@gmail.com wrote:
The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng.
I have this problem pretty often myself unfortunately. If you just want to get it to work ASAP even if it's not the most elegant situation, just add @ANYSTRING::@ to the end of your parser, right after the last non whitespace character of the message and it will "eat up" the trailing chars to successfully process the string.
Please, advice me how to proceed. Thank you.
Hope this will help you out.
Hi, Actually you do not have to use ANYSTRING for matching as the parses does a longest prefix search. ANYSTRING is useful only in cases when you want to store the rest of the message in a variable for accessing it latter. Normally you can simply end you pattern and it should match. BR, Marton
participants (5)
-
Balazs Scheidler
-
ILLES Marton
-
Martin Holste
-
Matthew Hall
-
otgovorete＠gmail.com