[syslog-ng] [PATCH 2/3] afmongodb: Port to value-pairs().

[Gergely Nagy]

Balazs Scheidler <bazsi at balabit.hu> writes:

> On Sun, 2011-05-01 at 09:32 +0200, Gergely Nagy wrote:
>> Balazs Scheidler <bazsi at balabit.hu> writes:
>> 
>> > On Sun, 2011-04-24 at 11:29 +0200, Gergely Nagy wrote:
>> >> Ported from custom, explicit key/value pairs to the far more flexible
>> >> value-pairs() solution. By default, the driver uses a custom scope:
>> >> selected_macros and nv_pairs, with a few patterns excluded.
>> >> 
>> >> The patch also turns the collection() parameter into a plain string
>> >> (from a templatable string), to considerably reduce the work needed to
>> >> insert messages into the database.
>> >> 
>> >> Signed-off-by: Gergely Nagy <algernon at balabit.hu>
>> >
>> > I've rebased this patch and adapted it to my value-pairs changes. Can
>> > you please check if it works for you?
>> 
>> It's not on your master branch, neither in 3.3 nor in 3.4, as far as I
>> see. I can reach the patch on git.balabit.hu via gitweb, but a cloned
>> repository does not have it.
>
> hmm, I don't understand, it is on the master branch, like all the
> others. Are you sure you properly cloned the repo? Can you try to
> explicitly check out origin/master ?

Yep, I'm pretty sure:

$ git clone git://git.balabit.hu/bazsi/syslog-ng-3.3.git
Cloning into syslog-ng-3.3...
remote: Generating pack...
remote: Done counting 9638 objects.
remote: Deltifying 9638 objects...
remote:  100% (9638/9638) done
remote: Total 9638 (delta 7421), reused 8274 (delta 6348)
Receiving objects: 100% (9638/9638), 2.32 MiB | 1.67 MiB/s, done.
Resolving deltas: 100% (7421/7421), done.
$ cd syslog-ng-3.3/
$ git log --pretty=oneline | head -n 30
7efc9d72a74f78f7ae7ce426c36174301af2224c Merge remote branch '3.2/master'
9245389973dbc78f14f79c2e091e3234eb0a6652 preparations for 3.2.3
c844bef80554d2b4dacd3796e0522f90b3721702 make systemd support a conditional feature
a8e298f300ad29442cd6f0aee7de5956438658e7 affile: don't attempt to remember the position for follow-freq(0) files
88a884a52bfc975335a05ffa0107684b6655ef29 tfjson: Template function for outputting JSON.
ae01dddab39926c86c0651d4a3aac46284393ef1 value-pairs: fixed macro expansion
0c62650f7a2dffed3a110c6589c136806105d000 afmongodb: Make the collection name a simple string.
dd65482f75270c40ac5f2f0cf948a9f772a2e9b7 affile: don't attempt to remember the position for follow-freq(0) files
2b65655028da7cc216aa747d0bb206fe28b93955 logwriter: initialize the queue_filled event earlier
2e243ea62396e6305ada53862ff4f0a3800b7443 libsyslog-ng-crypto.so: This patch introduces a new shared object where openssl dependent core goes
adff96aa401c8ef398a617c14daf7dc45f85cab7 logreader: fixed parsing flags containing dashes or underscores
be4f2fc5efdf637ef29e1216dd301db3a810f7bd The following options now can use with their default values (in file and in pipe destination):
fe958c6a486b33996fca8e6595b2c2995ee943e2 test_msgparse: added testcase to check unescaped quote in structured data
451071b928805322aed742be6e6f813c1468fa80 loggen: fixed compile error caused by the ignored SIGPIPE signal
dfdb9b17dff33e11019b46184e54420c35432a8f syslogformat: make sure all partially initialized NV pairs are removed if parsing the message fails
d329a86c1b2b94d9b8df4853ca458a4dcec47b08 [tags] fix log_tags_deinit() to properly free all tags related variables
1ba9335884d2fe44dcd51ad40d0f8e4afa9314bf value-pairs: A framework to filter key=value pairs.
5a2274a4f3e392537505313b2035bab22cbd6c6e loggen: handle SIGPIPE
be36e3c31a32cefce1fab0610b433f273f5b12a6 loggen: call shutdown on the socket
63f50e6abdeb6e859b22ff76a229688dd0b2cd5f [loggen] multithreaded loggen handles if file source is given as log source and more than one active-connection is used. and other improvments
fe4548bb3d985a22b636910641d1a8b925dd623a loggen: Added dont-parse option and restructured help menu
bf5bcb649c39130a869636373239052c32c5f7a3 loggen: added support for IPv6
3e9a0df3b6319c69d42a33c73f232838b762761f mainloop: fixed a possible use-after-free
ebde08e1efd3a53b5c30161edfbac11950821b81 fixed possible segmentation fault at exit
379420e5b0a4add464c6d8b03667dc5c4a244075 fixed 100% CPU usage if SSL is enabled on a tcp() source
3abdd8773662f9d779429262262a1ed9229e98e6 logproto: Handle EAGAIN and EINTR correctly in _text_client_flush().
78ea4d8aeba41a89a932e93b8b555480b2ebe130 fixed __thread keyword usage
8ce7ad823d450e3ceee93696b6ddbc491783c64f fixed refcache in consumer threads in case flow-control is not in use
c892da8491e65803e54c7bb1fd7b5a61f8810ef8 SCL: implement uninstall target
745111ea007c8ffa9a3d083757a6c47c78ddf0cd LogWriter: handle epoll/kqueue failures better

The collection name patch is there, but the value-pairs port isn't, as
far as I can see.

>> I looked through the patch on the web though, and I believe it's
>> correct, and should work. One comment though, regarding the commit
>> message: you moved the collection name un-templating to a separate
>> patch, so you might want to delete that part of the commit message :)
>
> Well, you submitted that portion in a separate email, so I've just
> committed them separately.

It was in both patches: one separate, and it was included as part of the
value-pairs() update too. (It's confusing, and I shouldn't have done
that, but I was dead tired at the time).

>> I'll download & apply the patch by hand later today, if time allows, and
>> test it properly. But I'm fairly sure it'll work.
>> 
>
> would be great. I just compile tested it.
>
> Can you point me to a simple description for the simplest ever usecase
> for the mongodb destination? So that I can test it when I apply a
> change.

The easiest way to test the driver is to install mongodb (for Debian &
Ubuntu systems, see
http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages), and
once done, set up a mongodb destination:

destination d_mongo {
  mongodb();
};

Then send a few messages to this destination, and from the shell, you
can do the following:

$ mongo localhost/syslog
> db.messages.find()

This will dump all the messages in the syslog.messages collection (the
default for the mongodb destination).

The output should be reasonably straightforward, I believe: all the
stuff set by the default scope (selected_macros + nvpairs, minus a few
stuff like R_* and S_*), in JSON-ish format.

-- 
|8]