[syslog-ng] Trying to get json out of 3.7beta2

[Balazs Scheidler]

Hi,

I meant the c source code for json-parser(), which is
modules/json/json-parser.c, more specifically the json_parser_process()
function:

```
static gboolean
json_parser_process(LogParser *s, LogMessage **pmsg, const LogPathOptions
*path_options, const gchar *input, gsize input_len)
{
  JSONParser *self = (JSONParser *) s;
  struct json_object *jso;
  struct json_tokener *tok;

  if (self->marker)
    {
      if (strncmp(input, self->marker, self->marker_len) != 0)
        return FALSE;
      input += self->marker_len;

      while (isspace(*input))
        input++;
    }

  tok = json_tokener_new();
  jso = json_tokener_parse_ex(tok, input, input_len);
  if (tok->err != json_tokener_success || !jso)
    {
      msg_error("Unparsable JSON stream encountered",
                evt_tag_str ("input", input),
                tok->err != json_tokener_success ? evt_tag_str ("error",
json_tokener_error_desc(tok->err)) : NULL,
                NULL);
      json_tokener_free (tok);
      return FALSE;
    }
  json_tokener_free(tok);

  log_msg_make_writable(pmsg, path_options);
  if (!json_parser_extract(self, jso, *pmsg))
    {
      msg_error("Error extracting JSON members into LogMessage as the
top-level JSON object is not an object",
                evt_tag_str ("input", input),
                NULL);
      json_object_put(jso);
      return FALSE;
    }
  json_object_put(jso);

  return TRUE;
}

```

As it seems, there are three cases where json-parser() drops messages:
1) a marker() option is specified and the message doesn't start with that
value
2) json parse error, in which case an error is printed
3) if the top-level json object in the input is not an Object (but a list
or a single value), but again an error message is printed in this case too

I didn't see the marker option in your configuration, so either of the
error cases should be logged as an internal message, at the error level.

Can you check that?

Cheers,
Bazsi



On Tue, Jul 7, 2015 at 8:48 PM, Russell Fulton <r.fulton at auckland.ac.nz>
wrote:

>
> > On 7 Jul 2015, at 08:57, Scheidler, Balázs <balazs.scheidler at balabit.com>
> wrote:
> >
> > Seems that jsonparser drops messages for some reason.
> > I don't have the code handy but perhaps you could check the error cases.
>
> sorry, not sure what you mean by 'check the error cases’.  There is
> nothing I can find in /var/log
>
> BTW This is a corporate managed RHEL box that runs rsyslog for ‘normal’
> logging.  I am running syslog-ng listening on non standard ports as I have
> done on other systems.
>
> >
> > It should report failures though.
>
> Let me know if there is anything I can do to help diagnose this.  I am
> keen to get this going.
>
> Is there any docs apart from the section in the user manual?  If anyone
> can point me to a non trivial example (including using pattern parsing)
> that would be really helpful.
>
> Russell
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>


-- 
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150707/14135d96/attachment.htm