Syslog-NG to parse HTTP query strings?
Hello Is Syslog-NG capable of parsing HTTP query string key=value pairs? I've read through both the CSV and pattern DB parser documentation and it seems they both presume a consistent ordering in the $MSG field. Neither seem appropriate when if the order of query string pairs changes, such as: http://somehost.com/somefile.cgi?keyA=valueA&keyB=valueB or http://somehost.com/somefile.cgi?keyB=valueB&keyA=valueA Basically, I want to grab keyA's value irrespective of which order it appears and use it as a value in an SQL insert statement. Is this possible with Syslog-NG's native parsers or am I going to have to handle this in a Perl script/program? Cheers Steve
On Fri, 2010-07-09 at 16:51 +1000, Steve Barnes wrote:
Hello
Is Syslog-NG capable of parsing HTTP query string key=value pairs? I've read through both the CSV and pattern DB parser documentation and it seems they both presume a consistent ordering in the $MSG field. Neither seem appropriate when if the order of query string pairs changes, such as:
http://somehost.com/somefile.cgi?keyA=valueA&keyB=valueB
or
http://somehost.com/somefile.cgi?keyB=valueB&keyA=valueA
Basically, I want to grab keyA's value irrespective of which order it appears and use it as a value in an SQL insert statement. Is this possible with Syslog-NG's native parsers or am I going to have to handle this in a Perl script/program?
Not right now, however syslog-ng 3.2 (only in its git form right now), features plugins and it is quite easy to implement a new parser using a plugin. using the internal API of syslog-ng is _way_ faster than using an external script. -- Bazsi
On Fri, 2010-07-09 at 16:51 +1000, Steve Barnes wrote:
Hello
Is Syslog-NG capable of parsing HTTP query string key=value pairs? I've read through both the CSV and pattern DB parser documentation and it seems they both presume a consistent ordering in the $MSG field. Neither seem appropriate when if the order of query string pairs changes, such as:
http://somehost.com/somefile.cgi?keyA=valueA&keyB=valueB
or
http://somehost.com/somefile.cgi?keyB=valueB&keyA=valueA
Basically, I want to grab keyA's value irrespective of which order it appears and use it as a value in an SQL insert statement. Is this possible with Syslog-NG's native parsers or am I going to have to handle this in a Perl script/program?
Not right now, however syslog-ng 3.2 (only in its git form right now), features plugins and it is quite easy to implement a new parser using a plugin.
using the internal API of syslog-ng is _way_ faster than using an external script.
Bazsi Agreed - handling parsing within Syslog-NG is definitely my preferred choice. I'll take a look at 3.2 and see what's involved in writing a plugin. Thanks for the pointer. Cheers Steve
On Fri, 2010-07-09 at 19:16 +1000, Steve Barnes wrote:
On Fri, 2010-07-09 at 16:51 +1000, Steve Barnes wrote:
Hello
Is Syslog-NG capable of parsing HTTP query string key=value pairs? I've read through both the CSV and pattern DB parser documentation and it seems they both presume a consistent ordering in the $MSG field. Neither seem appropriate when if the order of query string pairs changes, such as:
http://somehost.com/somefile.cgi?keyA=valueA&keyB=valueB
or
http://somehost.com/somefile.cgi?keyB=valueB&keyA=valueA
Basically, I want to grab keyA's value irrespective of which order it appears and use it as a value in an SQL insert statement. Is this possible with Syslog-NG's native parsers or am I going to have to handle this in a Perl script/program?
Not right now, however syslog-ng 3.2 (only in its git form right now), features plugins and it is quite easy to implement a new parser using a plugin.
using the internal API of syslog-ng is _way_ faster than using an external script.
Bazsi
Agreed - handling parsing within Syslog-NG is definitely my preferred choice. I'll take a look at 3.2 and see what's involved in writing a plugin. Thanks for the pointer.
Let me know if I can give you help. I've now split the db-parser() and csv-parser() code into separate plugins. The tricky part is the configuration parser, since bison doesn't support modular parsers, but if you copy the makefiles, *-parser.{c,h} files of the other modules and modify them accordingly should be relatively straightforward. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Steve Barnes