pdbtool parse with success but syslog-ng.conf NOT
Hi guys, I have following situation - I can parse with pdbtool bellow message successfully but when I parse it through syslog-ng.conf and put .classifier.class in DB I got "unknown" in the field where is placed .classifier.class or nothing when put the value FailedLogin_USERNAME in the table. Please see my syslog-ng.conf file below. Please help me to find the mistake. Thank you. Kosta Values: MESSAGE=Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2 PROGRAM=ssh .classifier.class=violation .classifier.rule_id=ssh-failed FailedLogin_MONTH=Sep FailedLogin_DATE=13 FailedLogin_TIME=17:34:00 FailedLogin_SERVER=server1 FailedLogin_SERVICE.ID2=sshd[20981]: FailedLogin_USERNAME=dfgdf FailedLogin_SOURCE_IP=xxxx FailedLogin_SOURCE.PORT=3602 kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2" <rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule> ################ # DESTINATIONS # ################ destination d_mssql_unix { sql(type(mssql) host("medea.mobiltel.bg") port("1433") username("syslog_ng") password("nglogp@ss") database("SysLog") table("SYSLOG_unix1")columns("Date varchar(40)", "Time varchar(16)", "SourceServer varchar(8)", "SendingProgram varchar(5)", "PID varchar(6)$ values("$DATE_", "$TIME_", "$HOST", "$PROGRAM", "$PID", "${.classifier.class}")); }; ############### # PARSER # ############### parser pattern_db { db_parser(file("/opt/syslog-ng/var/login.parser.new.xml")); }; parser DateTime { csv-parser(columns("DATE_", "TIME_") delimiters("T") flags(escape-none) template("${ISODATE}")); }; ############### # LOG # ############### log { source(s_net); parser (DateTime); parser(pattern_db); destination(d_mssql_unix); destination(d_messages_successful); };
Hi, The patterndb does only check the message (and optionally the program) part of a log, while in your example with pdbtool you matched on a whole log with date+time and host, which is not used in case of syslog-ng. Your example pattern is broken (probably bad copy&paste), but you should try pattern something like this: <pattern>Failed keyboard-interactive/pam for invalid user @ESTRING:FailedLogin_USERNAME: @from @ESTRING:FailedLogin_SOURCE_IP: @port @ESTRING:FailedLogin_SOURCE.PORT: @ssh2</pattern> Hope it helps, Marton On Fri, 2010-09-03 at 15:07 +0000, otgovorete@gmail.com wrote:
Hi guys,
I have following situation - I can parse with pdbtool bellow message successfully but when I parse it through syslog-ng.conf and put .classifier.class in DB I got "unknown" in the field where is placed .classifier.class or nothing when put the value FailedLogin_USERNAME in the table. Please see my syslog-ng.conf file below.
Please help me to find the mistake. Thank you.
Kosta
Values: MESSAGE=Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from x.x.x.x port 3602 ssh2 PROGRAM=ssh .classifier.class=violation .classifier.rule_id=ssh-failed FailedLogin_MONTH=Sep FailedLogin_DATE=13 FailedLogin_TIME=17:34:00 FailedLogin_SERVER=server1 FailedLogin_SERVICE.ID2=sshd[20981]: FailedLogin_USERNAME=dfgdf FailedLogin_SOURCE_IP=x.x.x.x FailedLogin_SOURCE.PORT=3602 kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from x.x.x.x port 3602 ssh2"
<rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule>
################ # DESTINATIONS # ################
destination d_mssql_unix { sql(type(mssql) host("medea.mobiltel.bg") port("1433") username("syslog_ng") password("nglogp@ss") database("SysLog") table("SYSLOG_unix1")columns("Date varchar(40)", "Time varchar(16)", "SourceServer varchar(8)", "SendingProgram varchar(5)", "PID varchar(6)$ values("$DATE_", "$TIME_", "$HOST", "$PROGRAM", "$PID", "${.classifier.class}")); };
############### # PARSER # ###############
parser pattern_db { db_parser(file("/opt/syslog-ng/var/login.parser.new.xml")); };
parser DateTime { csv-parser(columns("DATE_", "TIME_") delimiters("T") flags(escape-none) template("${ISODATE}")); };
############### # LOG # ###############
log { source(s_net); parser (DateTime); parser(pattern_db); destination(d_mssql_unix); destination(d_messages_successful); }; ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Fri, Sep 03, 2010 at 03:07:03PM +0000, otgovorete@gmail.com wrote:
kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2"
<rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule>
I had this problem before as well. It's important to know that certain headers are stripped off the message before they are parsed. "Sep 13 17:34:00 server1 " should get stripped off before the match. There's a thread from a while ago I started when I had this issue: https://lists.balabit.hu/pipermail/syslog-ng/2010-August/014588.html
On Fri, 2010-09-03 at 10:56 -0700, Matthew Hall wrote:
On Fri, Sep 03, 2010 at 03:07:03PM +0000, otgovorete@gmail.com wrote:
kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2"
<rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule>
I had this problem before as well. It's important to know that certain headers are stripped off the message before they are parsed.
"Sep 13 17:34:00 server1 " should get stripped off before the match.
There's a thread from a while ago I started when I had this issue:
And before going any further, you could also use the patterndb patterns which already covers this and is already tested. http://git.balabit.hu/bazsi/syslog-ng-patterndb.git -- Bazsi
participants (4)
-
Balazs Scheidler
-
Marton ILLES
-
Matthew Hall
-
otgovorete＠gmail.com