I can't understand how work db-parser, i want to parse a string: m-56767-1333854 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]') I wanto to have m-56767-1333854 on $ID_MESSAGE and 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]') on $MSG i try with: <patterndb> <ruleset name='assp'> <pattern>assp</pattern> <rules> <rule provider='balabit' id='1' class='system'> <patterns> <pattern>@QSTRING:id_message: @ @QSTRING:msg@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> But i have the field on db empty. I read link about db-parser usage but i can't resolve... Thanks, Jacopo -- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh
Some documentation is here: http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog-ng-d... . Try this: <pattern>@ESTRING:id_message: @@QSTRING:msg:@</pattern> I'm not sure about the msg part (didn't test it) but I'm sure that you want an ESTRING for the beginning since there is no starting quote char and you have special chars in what you are extracting. Marton's blog post has a lot more explanation, but in the end it will take a bit of trial and error for you to get proficient at it. It's worth it, though--the db-parser module is extremely efficient and will add a lot of depth to your analysis capabilities. I'm working on a Javascript front-end for point-and-click creation of db-parser templates from example logs, but it won't be ready for awhile. --Martin On Tue, Jul 7, 2009 at 3:56 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
I can't understand how work db-parser, i want to parse a string: m-56767-1333854 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]')
I wanto to have m-56767-1333854 on $ID_MESSAGE and 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]') on $MSG
i try with:
<patterndb> <ruleset name='assp'> <pattern>assp</pattern> <rules> <rule provider='balabit' id='1' class='system'> <patterns> <pattern>@QSTRING:id_message: @ @QSTRING:msg@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
But i have the field on db empty. I read link about db-parser usage but i can't resolve...
Thanks, Jacopo -- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
There is something in my configuration because with your string not log the id... parser p_assp { # db-parser(file("/opt/syslog-ng/etc/patterndb.xml")); db-parser(); }; I try twice but not work. log { ... ... parser(p_assp); ... }; Thanks, Jacopo 2009/7/7 Martin Holste <mcholste@gmail.com>:
Some documentation is here: http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog-ng-d... .
Try this:
<pattern>@ESTRING:id_message: @@QSTRING:msg:@</pattern>
I'm not sure about the msg part (didn't test it) but I'm sure that you want an ESTRING for the beginning since there is no starting quote char and you have special chars in what you are extracting. Marton's blog post has a lot more explanation, but in the end it will take a bit of trial and error for you to get proficient at it. It's worth it, though--the db-parser module is extremely efficient and will add a lot of depth to your analysis capabilities.
I'm working on a Javascript front-end for point-and-click creation of db-parser templates from example logs, but it won't be ready for awhile.
--Martin
On Tue, Jul 7, 2009 at 3:56 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
I can't understand how work db-parser, i want to parse a string: m-56767-1333854 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]')
I wanto to have m-56767-1333854 on $ID_MESSAGE and 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]') on $MSG
i try with:
<patterndb> <ruleset name='assp'> <pattern>assp</pattern> <rules> <rule provider='balabit' id='1' class='system'> <patterns> <pattern>@QSTRING:id_message: @ @QSTRING:msg@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
But i have the field on db empty. I read link about db-parser usage but i can't resolve...
Thanks, Jacopo -- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh
Hi, First you should simply try a pattern like this: <pattern>@ESTRING:id_message: @</pattern> This would match your line and would extract the message id. Than you can work on extending it. Also probably the easiest option is to use the @ANYSTRING@ parser which would match everything till the end of the message. It is available in the 3.1 git tree: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=commit;h=c22ee8dad59b56b9... So your pattern would look something like this: <pattern>@ESTRING:id_message: @@ANYSTRING:rest@</pattern> In the sql statement you can than use the ${id_message} and ${rest} macros. (Note that ANYSTRING is available only in the 3.1 tree which uses the newer patterndb format!) let me know if it works. best, Marton On Tue, 2009-07-07 at 11:10 -0500, Martin Holste wrote:
Some documentation is here: http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog-ng-d... .
Try this:
<pattern>@ESTRING:id_message: @@QSTRING:msg:@</pattern>
I'm not sure about the msg part (didn't test it) but I'm sure that you want an ESTRING for the beginning since there is no starting quote char and you have special chars in what you are extracting. Marton's blog post has a lot more explanation, but in the end it will take a bit of trial and error for you to get proficient at it. It's worth it, though--the db-parser module is extremely efficient and will add a lot of depth to your analysis capabilities.
I'm working on a Javascript front-end for point-and-click creation of db-parser templates from example logs, but it won't be ready for awhile.
--Martin
On Tue, Jul 7, 2009 at 3:56 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
I can't understand how work db-parser, i want to parse a string: m-56767-1333854 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]')
I wanto to have m-56767-1333854 on $ID_MESSAGE and 79.127.28.54 <mfdesigner@diggitgraphics.com> MessageScore is now 30, after adding 30 (Suspicious HELO - contains IP: '[79.127.28.54]') on $MSG
i try with:
<patterndb> <ruleset name='assp'> <pattern>assp</pattern> <rules> <rule provider='balabit' id='1' class='system'> <patterns> <pattern>@QSTRING:id_message: @ @QSTRING:msg@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
But i have the field on db empty. I read link about db-parser usage but i can't resolve...
Thanks, Jacopo -- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
On Mon, 2009-07-13 at 19:59 +0200, ILLES, Marton wrote:
Hi,
First you should simply try a pattern like this:
<pattern>@ESTRING:id_message: @</pattern>
This would match your line and would extract the message id. Than you can work on extending it. Also probably the easiest option is to use the @ANYSTRING@ parser which would match everything till the end of the message. It is available in the 3.1 git tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=commit;h=c22ee8dad59b56b9...
So your pattern would look something like this:
<pattern>@ESTRING:id_message: @@ANYSTRING:rest@</pattern>
In the sql statement you can than use the ${id_message} and ${rest} macros. (Note that ANYSTRING is available only in the 3.1 tree which uses the newer patterndb format!)
let me know if it works.
I didn't have time to completely integrate your patterndb v2 patches, so it still sits in a local branch and not on master. But ANYSTRING is already there. -- Bazsi
Ok i must use ANYSTRING but for use it i need the 3.1 version but i can't compile it... I download the snapshot from git-web but when i try to "make" but afsql.c:36:21: error: dbi/dbi.h: No such file or directory i download the wrong version? Thank, Jacopo 2009/7/14 Balazs Scheidler <bazsi@balabit.hu>:
On Mon, 2009-07-13 at 19:59 +0200, ILLES, Marton wrote:
Hi,
First you should simply try a pattern like this:
<pattern>@ESTRING:id_message: @</pattern>
This would match your line and would extract the message id. Than you can work on extending it. Also probably the easiest option is to use the @ANYSTRING@ parser which would match everything till the end of the message. It is available in the 3.1 git tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=commit;h=c22ee8dad59b56b9...
So your pattern would look something like this:
<pattern>@ESTRING:id_message: @@ANYSTRING:rest@</pattern>
In the sql statement you can than use the ${id_message} and ${rest} macros. (Note that ANYSTRING is available only in the 3.1 tree which uses the newer patterndb format!)
let me know if it works.
I didn't have time to completely integrate your patterndb v2 patches, so it still sits in a local branch and not on master.
But ANYSTRING is already there.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh
You probably need to install libdbi (libdbi.sourceforge.net) and probably some of the drivers for libdbi as well. It should compile pretty easily with the standard configure make make install. On Tue, Jul 14, 2009 at 8:48 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
Ok i must use ANYSTRING but for use it i need the 3.1 version but i can't compile it... I download the snapshot from git-web but when i try to "make" but afsql.c:36:21: error: dbi/dbi.h: No such file or directory
i download the wrong version?
Thank, Jacopo
2009/7/14 Balazs Scheidler <bazsi@balabit.hu>:
On Mon, 2009-07-13 at 19:59 +0200, ILLES, Marton wrote:
Hi,
First you should simply try a pattern like this:
<pattern>@ESTRING:id_message: @</pattern>
This would match your line and would extract the message id. Than you can work on extending it. Also probably the easiest option is to use the @ANYSTRING@ parser which would match everything till the end of the message. It is available in the 3.1 git tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=commit;h=c22ee8dad59b56b9...
So your pattern would look something like this:
<pattern>@ESTRING:id_message: @@ANYSTRING:rest@</pattern>
In the sql statement you can than use the ${id_message} and ${rest} macros. (Note that ANYSTRING is available only in the 3.1 tree which uses the newer patterndb format!)
let me know if it works.
I didn't have time to completely integrate your patterndb v2 patches, so it still sits in a local branch and not on master.
But ANYSTRING is already there.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Work :) Another thing :P it's possible to log @ESTRING:id_message: @ only if contains specific word? Thanks, Jacopo 2009/7/14 Martin Holste <mcholste@gmail.com>:
You probably need to install libdbi (libdbi.sourceforge.net) and probably some of the drivers for libdbi as well. It should compile pretty easily with the standard configure make make install.
On Tue, Jul 14, 2009 at 8:48 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
Ok i must use ANYSTRING but for use it i need the 3.1 version but i can't compile it... I download the snapshot from git-web but when i try to "make" but afsql.c:36:21: error: dbi/dbi.h: No such file or directory
i download the wrong version?
Thank, Jacopo
2009/7/14 Balazs Scheidler <bazsi@balabit.hu>:
On Mon, 2009-07-13 at 19:59 +0200, ILLES, Marton wrote:
Hi,
First you should simply try a pattern like this:
<pattern>@ESTRING:id_message: @</pattern>
This would match your line and would extract the message id. Than you can work on extending it. Also probably the easiest option is to use the @ANYSTRING@ parser which would match everything till the end of the message. It is available in the 3.1 git tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=commit;h=c22ee8dad59b56b9...
So your pattern would look something like this:
<pattern>@ESTRING:id_message: @@ANYSTRING:rest@</pattern>
In the sql statement you can than use the ${id_message} and ${rest} macros. (Note that ANYSTRING is available only in the 3.1 tree which uses the newer patterndb format!)
let me know if it works.
I didn't have time to completely integrate your patterndb v2 patches, so it still sits in a local branch and not on master.
But ANYSTRING is already there.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh
ESTRING will match all characters up until the match character given and store them in the macro provided, in this case "id_message." The match character is the last character before the @ symbol, in this case, a space (ASCII 0x20). On Wed, Jul 15, 2009 at 4:34 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
Work :) Another thing :P it's possible to log @ESTRING:id_message: @ only if contains specific word?
Thanks, Jacopo
2009/7/14 Martin Holste <mcholste@gmail.com>:
You probably need to install libdbi (libdbi.sourceforge.net) and probably some of the drivers for libdbi as well. It should compile pretty easily with the standard configure make make install.
On Tue, Jul 14, 2009 at 8:48 AM, Jacopo Cappelli<jacopo89@gmail.com> wrote:
Ok i must use ANYSTRING but for use it i need the 3.1 version but i can't compile it... I download the snapshot from git-web but when i try to "make" but afsql.c:36:21: error: dbi/dbi.h: No such file or directory
i download the wrong version?
Thank, Jacopo
2009/7/14 Balazs Scheidler <bazsi@balabit.hu>:
On Mon, 2009-07-13 at 19:59 +0200, ILLES, Marton wrote:
Hi,
First you should simply try a pattern like this:
<pattern>@ESTRING:id_message: @</pattern>
This would match your line and would extract the message id. Than you can work on extending it. Also probably the easiest option is to use the @ANYSTRING@ parser which would match everything till the end of the message. It is available in the 3.1 git tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=commit;h=c22ee8dad59b56b9...
So your pattern would look something like this:
<pattern>@ESTRING:id_message: @@ANYSTRING:rest@</pattern>
In the sql statement you can than use the ${id_message} and ${rest} macros. (Note that ANYSTRING is available only in the 3.1 tree which uses the newer patterndb format!)
let me know if it works.
I didn't have time to completely integrate your patterndb v2 patches, so it still sits in a local branch and not on master.
But ANYSTRING is already there.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Linux, Windows Xp ed MS-DOS (anche conosciuti come il Bello, il Brutto ed il Cattivo). -- Matt Welsh ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2009-07-15 at 08:03 -0500, Martin Holste wrote:
ESTRING will match all characters up until the match character given and store them in the macro provided, in this case "id_message." The match character is the last character before the @ symbol, in this case, a space (ASCII 0x20).
Please note that I've just pushed out some 3.1 patterndb updates, mostly those that were sitting in Marci's branch. This means that the latest rev already uses the revised v2 patterndb format. I had some xslt that converted the old one to the new one, but I'll have to dig that out. -- Bazsi
participants (4)
-
Balazs Scheidler
-
ILLES, Marton
-
Jacopo Cappelli
-
Martin Holste