[syslog-ng] Writing JSON message to a MongoDB destination

[Gergely Nagy]

Rajalakshmi Iyer <raj at blismedia.com> writes:

> Hello,
>
> I have a log message in JSON format that needs to be written to the MongoDB
> database, such  that I should be able to later query the database based on
> fields in the JSON record.
>
> However, when I try to use the syslog-ng mongodb destination driver using :
>
> destination d_mongodb {
>   mongodb(
>     servers("localhost:27017")
>     database("syslog")
>     collection("messages")
>     value-pairs(key("MESSAGE"))
>   );
> };
>
>
> The record appears in the mongodb database as:
>
> {"MESSAGE":<My JSON formatted message>}
>
> I do not want the MESSAGE macro to be part of the record in the database, I
> only want to it's JSON-encoded value to be entered in the database. Is this
> possible?

If I understand you correctly, you want to *parse* the JSON message, and
put whatever fileds are in it, into MongoDB. That is not possible with
syslog-ng 3.3, because it does not have a json parser (unless you
backport it from 3.4, which isn't that hard, and I can help if so need
be).

With syslog-ng 3.4, you could do something along these lines:

source s_json {
 tcp("0.0.0.0" port(8192) flags(no-parse));
};

parser p_json {
  json-parser(prefix(".json."));
};

destination d_mongodb {
  mongodb(
    servers("localhost:27017")
    database("syslog")
    collection("messages")
    value-pairs(key(".json.*" rekey(shift(6))))
  );
};

log {
 source(s_json);
 parser(p_json);
 destination(d_mongodb);
};

-- 
|8]