Please help getting patterndb working.
Hello, By following the documentation examples I think I have recreated a custom parser, but it doesn't work, please can you advise what I have missed? To create my patterndb I have used the example in link [1], to load it and output to a file I have kinda copied the example in link [2]. The issue I have is that my output file is mostly empty, it's like the variables are not being read properly. In case the files do not render properly , please see this gist -> https://gist.github.com/linickx/8002981 I have built a clean fedora 19 box which ships with syslog-ng-3.4.6-1. My /etc/syslog-ng/syslog-ng conf looks like: <PASTE> @version:3.2 # Default options copied from distribution options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); owner("nick"); group("nick"); perm(0640); }; source test_logfile { file("/home/nick/testfile.log"); }; parser test_pattern { db_parser( file("/home/nick/example.xml") ); }; destination test_output { file("/home/nick/output.log" template("${SSH_USERNAME}; ${SSH_CLIENT_ADDRESS}; \n") template_escape(no) ); }; log { source (test_logfile); parser(test_pattern); destination (test_output); }; </PASTE> The /home/nick/example.xml looks like <PASTE> <patterndb version='4' pub_date='2010-10-17'> <ruleset name='ssh' id='123456678'> <pattern>ssh</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern> </patterns> <examples> <example> <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message> <test_values> <test_value name="SSH.AUTH_METHOD">password</test_value> <test_value name="SSH_USERNAME">sampleuser</test_value> <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value> <test_value name="SSH_PORT_NUMBER">42156</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> </PASTE> For /home/nick/testfile.log I have used <PASTE> Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2 Accepted password for user from 10.51.0.27 port 4256 ssh2 </PASTE> after starting syslog-ng I get the following in /home/nick/output.log PASTE ; ; ; ; </PASTE> Any pointers on what I'm doing wrong would be very much appreciated. This is a cross-post from http://communities.balabit.com/balabit/topics/please_help_me_get_patterndb_w..., apologies if this offends, the balabit.com website is not clear on where the most active community can be found. Thanks in Advance, Nick [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guide...
Hi, Your pattern is wrong, and doesn't match your example. First of all, add a "program" attribute to your test_message. Second, use 'pdbtool' to test your pdb. Third, correct your pattern :) Fourth, use 'pdbtool' to parse your logfile for you. Hint: look up the documnentation for QSTRING pattern, and also look at @ESTRING cheers
Hello Fabien, Thanks for your mail, I am n00b and just following that which balabit documented, please bare with me :) [1] I have added a program attribute. <PASTE> <patterndb version='4' pub_date='2010-10-17'> <ruleset name='ssh' id='123456678'> <pattern>ssh</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER :SSH_PORT_NUMBER:@ ssh2</pattern> </patterns> <examples> <example> <test_message program="example">Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message> <test_values> <test_value name="SSH.AUTH_METHOD">password</test_value> <test_value name="SSH_USERNAME">sampleuser</test_value> <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value> <test_value name="SSH_PORT_NUMBER">42156</test_value> </test_values> </example> </examples> </rule> </rules> </ruleset> </patterndb> </PASTE> [2] prior to adding the program, pdbtool just responded with "example.xml validates", now I get something a bit more verbose. [nick@localhost ~]$ pdbtool test --validate example.xml example.xml validates Testing message program='example' message='Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2' Wrong match name='.classifier.rule_id', value='', expected='182437592347598' Wrong match name='SSH.AUTH_METHOD', value='', expected='password' Wrong match name='SSH_USERNAME', value='', expected='sampleuser' Wrong match name='SSH_CLIENT_ADDRESS', value='', expected='10.50.0.247' Wrong match name='SSH_PORT_NUMBER', value='', expected='42156' [nick@localhost ~]$ [3] I agree that the pattern is wrong, the output above shows that but the example I'm following if from the balabit documentation, is there a better reference I should be following? Following your hint, I've tried changing "<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for" with "<pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for", according to [link*] the "space" between the final : (colon) and @ (at) should act as a stop char, but clearly not. Am I trying to run before walking? [4] I assume you mean "pdbtool patternize -f testfile.log" ? I'm not sure how that helps... [nick@localhost ~]$ pdbtool patternize -f testfile.log [Tue Dec 17 11:57:22 2013] Searching clusters; input lines='4' [Tue Dec 17 11:57:23 2013] Finding frequent words; phase='caching' [Tue Dec 17 11:57:23 2013] Finding frequent words; phase='searching' <patterndb version='3' pub_date='2013-12-17'> <ruleset name='patternize' id='8a7a3c95-af22-894b-942b-d4517c389175'> <rules> <rule id='ed85a798-8440-4044-97e2-ba23753188e5' class='system' provider='patternize'> <!-- support: 2 --> <patterns> <pattern>password for user from 10.51.0.27 port 4256 ssh2</pattern> </patterns> <examples> <example> <test_message program='patternize'>password for user from 10.51.0.27 port 4256 ssh2</test_message> </example> </examples> </rule> <rule id='44834044-fda5-2040-ae58-048bbc039d3d' class='system' provider='patternize'> <!-- support: 2 --> <patterns> <pattern>password for sampleuser from 10.50.0.247 port 42156 ssh2</pattern> </patterns> <examples> <example> <test_message program='patternize'>password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message> </example> </examples> </rule> </rules> </ruleset> </patterndb> [nick@localhost ~]$ I would have expected pdbtool to create three variables for 'sampleuser' , the IP addresses and port numbers as they are the things which change on each line of the file. I tried updating testlogfile to have four unique entries to give it a bit more chance to spot the changes but still no luck. Thanks in Advance, Nick [link*]http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... On 17 December 2013 11:31, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
Your pattern is wrong, and doesn't match your example. First of all, add a "program" attribute to your test_message. Second, use 'pdbtool' to test your pdb. Third, correct your pattern :) Fourth, use 'pdbtool' to parse your logfile for you.
Hint: look up the documnentation for QSTRING pattern, and also look at @ESTRING
cheers
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Shameless plug for google Juice: http://www.linickx.com
Hi, On Tue, Dec 17, 2013 at 12:07:41PM +0000, Nick wrote:
[1] I have added a program attribute.
Note that the "program" attribute of "test_message" needs to match the rule's "pattern" text if you want a match.
[3] I agree that the pattern is wrong, the output above shows that but
Try the following: https://gist.github.com/faxm0dem/b2c87efb098b4aba1969
[4] I assume you mean "pdbtool patternize -f testfile.log" ? I'm not sure how that helps...
Actually I meant 'pdbtool match' Sadly, 'patternize' failed to help me in the past, maybe someone else can comment on that.
Hello Fabien, Thank you for the example, I can now see what you have done and changes respond as I expect. Now on to start my own :) Thanks Again! rgds, Nick On 17 December 2013 12:19, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Tue, Dec 17, 2013 at 12:07:41PM +0000, Nick wrote:
[1] I have added a program attribute.
Note that the "program" attribute of "test_message" needs to match the rule's "pattern" text if you want a match.
[3] I agree that the pattern is wrong, the output above shows that but
Try the following: https://gist.github.com/faxm0dem/b2c87efb098b4aba1969
[4] I assume you mean "pdbtool patternize -f testfile.log" ? I'm not sure how that helps...
Actually I meant 'pdbtool match'
Sadly, 'patternize' failed to help me in the past, maybe someone else can comment on that.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Shameless plug for google Juice: http://www.linickx.com
I would also like to thank Fabien for his help. Such assistance on the mailing list is really useful for the project as a whole. And although its important I can't always make it in a timely manner. Kudos, Fabien. On Dec 18, 2013 4:53 PM, "Nick" <linickx@gmail.com> wrote:
Hello Fabien,
Thank you for the example, I can now see what you have done and changes respond as I expect.
Now on to start my own :)
Thanks Again! rgds, Nick
On 17 December 2013 12:19, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Tue, Dec 17, 2013 at 12:07:41PM +0000, Nick wrote:
[1] I have added a program attribute.
Note that the "program" attribute of "test_message" needs to match the rule's "pattern" text if you want a match.
[3] I agree that the pattern is wrong, the output above shows that but
Try the following: https://gist.github.com/faxm0dem/b2c87efb098b4aba1969
[4] I assume you mean "pdbtool patternize -f testfile.log" ? I'm not sure how that helps...
Actually I meant 'pdbtool match'
Sadly, 'patternize' failed to help me in the past, maybe someone else can comment on that.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Shameless plug for google Juice: http://www.linickx.com
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hello, Apologies - I thought I had closed this thread :-/ I've just realised that when I use Fabien's example in syslog-ng the output file is still empty. I've updated the https://gist.github.com/linickx/8002981 in case the text below doesn't render. example.xml is now the correct version... [nick@localhost ~]$ pdbtool test --validate example.xml example.xml validates Testing message program='ssh' message='Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2' [nick@localhost ~]$ The syslog-ng.conf is the same (https://gist.github.com/linickx/8002981#file-syslog-ng-conf) and the net result is the output is still the same (https://gist.github.com/linickx/8002981#file-output-log) Following the previous 'pdbtool match' ... I get the following? [nick@localhost ~]$ pdbtool match -p example.xml -f testfile.log --template "${SSH_USERNAME}; ${SSH_CLIENT_ADDRESS}; \n" ; ; ; ; [nick@localhost ~]$ pdbtool match -p example.xml -f testfile.log --template "${SSH_USERNAME}; ${SSH_CLIENT_ADDRESS}; \n" -D -v Module loaded and initialized successfully; module='syslogformat' Module loaded and initialized successfully; module='basicfuncs' Pattern matching part: password for sampleuser from 10.50.0.247 port 42156 ssh2 Matching part: Values: MESSAGE=password for sampleuser from 10.50.0.247 port 42156 ssh2 PROGRAM=Accepted LEGACY_MSGHDR=Accepted .classifier.class=unknown TAGS=.classifier.unknown Pattern matching part: password for user from 10.51.0.27 port 4256 ssh2 Matching part: Values: MESSAGE=password for user from 10.51.0.27 port 4256 ssh2 PROGRAM=Accepted LEGACY_MSGHDR=Accepted .classifier.class=unknown TAGS=.classifier.unknown Closing log transport fd; fd='3' [nick@localhost ~]$ In my syslog-ng.conf or template, how should I be using the variables defined in the patterndb? Thanks in Advance, Nick On 17 December 2013 12:19, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Tue, Dec 17, 2013 at 12:07:41PM +0000, Nick wrote:
[1] I have added a program attribute.
Note that the "program" attribute of "test_message" needs to match the rule's "pattern" text if you want a match.
[3] I agree that the pattern is wrong, the output above shows that but
Try the following: https://gist.github.com/faxm0dem/b2c87efb098b4aba1969
[4] I assume you mean "pdbtool patternize -f testfile.log" ? I'm not sure how that helps...
Actually I meant 'pdbtool match'
Sadly, 'patternize' failed to help me in the past, maybe someone else can comment on that.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Shameless plug for google Juice: http://www.linickx.com
participants (3)
-
Balazs Scheidler
-
Fabien Wernli
-
Nick