This is a bit of a problem if you want to get it into a traditional RDBMS as you have to know the columns ahead of time. There are a couple of ways you can handle this: - use a NoSQL database like MongoDB (which is awesome) - use a traditional RMDBS but run your output through a (you guessed it) Perl script which will format it into blobs in XML or JSON to get a SQL/NoSQL hybrid - record the "real" column names in a separate DB table and use aliases for the patterns (this is what I do now). For instance, I have a set table with six integer columns and six string columns, and extract the names i0-i5 and s0-s5 so they can go directly to the DB. When it comes time to query, I use the class_id to dictate what the context of "i1" means for a given row. So i1 could be an IP address for class 1 and an event ID for class 2. The point is that I don't need to worry about altering the DB schema for each class type. I have just one destination driver with one template, and it logs the macros i0-s5 whether they are present or not, which is fine, because they'll just go in as nulls in the DB. On Mon, Oct 25, 2010 at 2:58 PM, Lars Kellogg-Stedman <lars@oddbit.com> wrote:
As I work with the classification engine, I wish there were a way for me to log *all* metadata associated with a log message. That is, I would like to record any data parsed out of the message by the parsing engine, as well as all the default metadata that syslog-ng generates about a message. Is there any way to do this other than writing my own output driver? As far as I can tell, all of the available drivers -- including the sql() driver -- require me to explicitly list which attributes I want to log.
While my ultimate target would be sending this data into a database, I would be happy if I were able to dump it all to a file in some sort of structured format that I could parse with my own tools (or maybe even feed back into syslog-ng).
Thanks,
-- Lars ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html