On 01/02/2011 12:51 AM, Gergely Nagy wrote:
This is why I asked in my earlier mail if it's possible to set up the mongo driver to log all vars in a message or a subset of vars in a message. I was hoping it'd be possible for the schema to change somewhat dynamically based on what's present in the messages. You can set it up to log a set of vars, and it will only actually insert the non-empty values.
Say, if you have something like this:
destination d_mongo { mongodb( keys("host", "program", "pid", "message") values("$HOST", "$PROGRAM", "$PID", "$MSGONLY") ); };
If a message does not contain a PID, then that will not be added to the document, only the rest.
Thus, if you set a maximum of vars, that'll do just what you need, and only add those that do have a value.
To the best of my knowledge it is not possible to log all available variables (that would be bad too, since there are overlapping macros), but you can set up a selected maximum set, and the driver will Do The Right Thing, and only store those parts of it, that are set. Hi,
first of all, thanks for the great work. I agree with Matthew, that it would be really important to make this driver "dynamic", as it would be a great tool combined with patterndb for reporting without the need to pre-define fields and a dozen of destination statements. It is actually not that hard to achieve (again, syslog-ng is a breeze), pdbtool does quite the same when emitting all variables, the nv_table_foreach() function is there to iterate over all of the name-value pairs. However the NVTable struct stores the builtin and dynamic values separately and with a small copy-paste coding in nvtable.c you can grab only the dynamic values. Please find a patch attached that introduces the flags() option for the mongodb driver and the auto_nvpairs flag, that inserts all dynamic name-value pairs into the DB as well. I'm sure that there's a better way to implement some parts of it, so please somebody review and clean up if possible :) Usage would look something like this: destination d_mongo { mongodb( database("logs") keys("host", "program", "pid", "message") values("$HOST", "$PROGRAM", "$PID", "$MSGONLY") flags(auto_nvpairs) ); }; No performance measurements were done yet, I would be glad to see it on the same box and same settings as the previous ones. Of course this will be a bit slower, ad it makes sense only if you use it in conjunction with patterndb, but I expect no drastic drop in performance. (Disclaimer: I am not a developer, this code is far from being ready for production, may leak, etc, etc) Balint