[syslog-ng] Logging all message metadata

[Martin Holste]

This is a bit of a problem if you want to get it into a traditional
RDBMS as you have to know the columns ahead of time.  There are a
couple of ways you can handle this:
 - use a NoSQL database like MongoDB (which is awesome)
 - use a traditional RMDBS but run your output through a (you guessed
it) Perl script which will format it into blobs in XML or JSON to get
a SQL/NoSQL hybrid
  - record the "real" column names in a separate DB table and use
aliases for the patterns (this is what I do now).  For instance, I
have a set table with six integer columns and six string columns, and
extract the names i0-i5 and s0-s5 so they can go directly to the DB.
When it comes time to query, I use the class_id to dictate what the
context of "i1" means for a given row.  So i1 could be an IP address
for class 1 and an event ID for class 2.  The point is that I don't
need to worry about altering the DB schema for each class type.  I
have just one destination driver with one template, and it logs the
macros i0-s5 whether they are present or not, which is fine, because
they'll just go in as nulls in the DB.

On Mon, Oct 25, 2010 at 2:58 PM, Lars Kellogg-Stedman <lars at oddbit.com> wrote:
> As I work with the classification engine, I wish there were a way for
> me to log *all* metadata associated with a log message.  That is, I
> would like to record any data parsed out of the message by the parsing
> engine, as well as all the default metadata that syslog-ng generates
> about a message.  Is there any way to do this other than writing my
> own output driver?  As far as I can tell, all of the available drivers
> -- including the sql() driver -- require me to explicitly list which
> attributes I want to log.
>
> While my ultimate target would be sending this data into a database, I
> would be happy if I were able to dump it all to a file in some sort of
> structured format that I could parse with my own tools (or maybe even
> feed back into syslog-ng).
>
> Thanks,
>
> -- Lars
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>