Re: [syslog-ng] Writing JSON message to a MongoDB destination

29 May 2012

      Rajalakshmi Iyer <raj@blismedia.com> writes:
...
Hello,
I have a log message in JSON format that needs to be written to the MongoDB
database, such  that I should be able to later query the database based on
fields in the JSON record.
However, when I try to use the syslog-ng mongodb destination driver using :
destination d_mongodb {
  mongodb(
    servers("localhost:27017")
    database("syslog")
    collection("messages")
    value-pairs(key("MESSAGE"))
  );
};
The record appears in the mongodb database as:
{"MESSAGE":<My JSON formatted message>}
I do not want the MESSAGE macro to be part of the record in the database, I
only want to it's JSON-encoded value to be entered in the database. Is this
possible?
If I understand you correctly, you want to *parse* the JSON message, and
put whatever fileds are in it, into MongoDB. That is not possible with
syslog-ng 3.3, because it does not have a json parser (unless you
backport it from 3.4, which isn't that hard, and I can help if so need
be).

With syslog-ng 3.4, you could do something along these lines:

source s_json {
 tcp("0.0.0.0" port(8192) flags(no-parse));
};

parser p_json {
  json-parser(prefix(".json."));
};

destination d_mongodb {
  mongodb(
    servers("localhost:27017")
    database("syslog")
    collection("messages")
    value-pairs(key(".json.*" rekey(shift(6))))
  );
};

log {
 source(s_json);
 parser(p_json);
 destination(d_mongodb);
};

-- 
|8]

Re: [syslog-ng] Writing JSON message to a MongoDB destination

Gergely Nagy