Balazs Scheidler <bazsi@balabit.hu> writes:
On Sun, 2011-05-01 at 09:32 +0200, Gergely Nagy wrote:
Balazs Scheidler <bazsi@balabit.hu> writes:
On Sun, 2011-04-24 at 11:29 +0200, Gergely Nagy wrote:
Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded.
The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database.
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
I've rebased this patch and adapted it to my value-pairs changes. Can you please check if it works for you?
It's not on your master branch, neither in 3.3 nor in 3.4, as far as I see. I can reach the patch on git.balabit.hu via gitweb, but a cloned repository does not have it.
hmm, I don't understand, it is on the master branch, like all the others. Are you sure you properly cloned the repo? Can you try to explicitly check out origin/master ?
Yep, I'm pretty sure: $ git clone git://git.balabit.hu/bazsi/syslog-ng-3.3.git Cloning into syslog-ng-3.3... remote: Generating pack... remote: Done counting 9638 objects. remote: Deltifying 9638 objects... remote: 100% (9638/9638) done remote: Total 9638 (delta 7421), reused 8274 (delta 6348) Receiving objects: 100% (9638/9638), 2.32 MiB | 1.67 MiB/s, done. Resolving deltas: 100% (7421/7421), done. $ cd syslog-ng-3.3/ $ git log --pretty=oneline | head -n 30 7efc9d72a74f78f7ae7ce426c36174301af2224c Merge remote branch '3.2/master' 9245389973dbc78f14f79c2e091e3234eb0a6652 preparations for 3.2.3 c844bef80554d2b4dacd3796e0522f90b3721702 make systemd support a conditional feature a8e298f300ad29442cd6f0aee7de5956438658e7 affile: don't attempt to remember the position for follow-freq(0) files 88a884a52bfc975335a05ffa0107684b6655ef29 tfjson: Template function for outputting JSON. ae01dddab39926c86c0651d4a3aac46284393ef1 value-pairs: fixed macro expansion 0c62650f7a2dffed3a110c6589c136806105d000 afmongodb: Make the collection name a simple string. dd65482f75270c40ac5f2f0cf948a9f772a2e9b7 affile: don't attempt to remember the position for follow-freq(0) files 2b65655028da7cc216aa747d0bb206fe28b93955 logwriter: initialize the queue_filled event earlier 2e243ea62396e6305ada53862ff4f0a3800b7443 libsyslog-ng-crypto.so: This patch introduces a new shared object where openssl dependent core goes adff96aa401c8ef398a617c14daf7dc45f85cab7 logreader: fixed parsing flags containing dashes or underscores be4f2fc5efdf637ef29e1216dd301db3a810f7bd The following options now can use with their default values (in file and in pipe destination): fe958c6a486b33996fca8e6595b2c2995ee943e2 test_msgparse: added testcase to check unescaped quote in structured data 451071b928805322aed742be6e6f813c1468fa80 loggen: fixed compile error caused by the ignored SIGPIPE signal dfdb9b17dff33e11019b46184e54420c35432a8f syslogformat: make sure all partially initialized NV pairs are removed if parsing the message fails d329a86c1b2b94d9b8df4853ca458a4dcec47b08 [tags] fix log_tags_deinit() to properly free all tags related variables 1ba9335884d2fe44dcd51ad40d0f8e4afa9314bf value-pairs: A framework to filter key=value pairs. 5a2274a4f3e392537505313b2035bab22cbd6c6e loggen: handle SIGPIPE be36e3c31a32cefce1fab0610b433f273f5b12a6 loggen: call shutdown on the socket 63f50e6abdeb6e859b22ff76a229688dd0b2cd5f [loggen] multithreaded loggen handles if file source is given as log source and more than one active-connection is used. and other improvments fe4548bb3d985a22b636910641d1a8b925dd623a loggen: Added dont-parse option and restructured help menu bf5bcb649c39130a869636373239052c32c5f7a3 loggen: added support for IPv6 3e9a0df3b6319c69d42a33c73f232838b762761f mainloop: fixed a possible use-after-free ebde08e1efd3a53b5c30161edfbac11950821b81 fixed possible segmentation fault at exit 379420e5b0a4add464c6d8b03667dc5c4a244075 fixed 100% CPU usage if SSL is enabled on a tcp() source 3abdd8773662f9d779429262262a1ed9229e98e6 logproto: Handle EAGAIN and EINTR correctly in _text_client_flush(). 78ea4d8aeba41a89a932e93b8b555480b2ebe130 fixed __thread keyword usage 8ce7ad823d450e3ceee93696b6ddbc491783c64f fixed refcache in consumer threads in case flow-control is not in use c892da8491e65803e54c7bb1fd7b5a61f8810ef8 SCL: implement uninstall target 745111ea007c8ffa9a3d083757a6c47c78ddf0cd LogWriter: handle epoll/kqueue failures better The collection name patch is there, but the value-pairs port isn't, as far as I can see.
I looked through the patch on the web though, and I believe it's correct, and should work. One comment though, regarding the commit message: you moved the collection name un-templating to a separate patch, so you might want to delete that part of the commit message :)
Well, you submitted that portion in a separate email, so I've just committed them separately.
It was in both patches: one separate, and it was included as part of the value-pairs() update too. (It's confusing, and I shouldn't have done that, but I was dead tired at the time).
I'll download & apply the patch by hand later today, if time allows, and test it properly. But I'm fairly sure it'll work.
would be great. I just compile tested it.
Can you point me to a simple description for the simplest ever usecase for the mongodb destination? So that I can test it when I apply a change.
The easiest way to test the driver is to install mongodb (for Debian & Ubuntu systems, see http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages), and once done, set up a mongodb destination: destination d_mongo { mongodb(); }; Then send a few messages to this destination, and from the shell, you can do the following: $ mongo localhost/syslog
db.messages.find()
This will dump all the messages in the syslog.messages collection (the default for the mongodb destination). The output should be reasonably straightforward, I believe: all the stuff set by the default scope (selected_macros + nvpairs, minus a few stuff like R_* and S_*), in JSON-ish format. -- |8]