[syslog-ng] [PATCH] jsonparser: New parser module for JSON parsing.

[Balazs Scheidler]

On Sat, 2011-10-29 at 13:38 +0200, Balazs Scheidler wrote:
> This is cool stuff, but again doesn't compile with a stock json-c.
> 
> Hmm.. this time I've checked, newer json-c versions do install the
> referenced headers. Then I guess it's it needs an updated version number
> in configure.in.

I've integrated this, but I may have been a bit too fast handed. There's
no locking in json-parser(), which will cause things to go bad if
threaded(yes) is enabled in the configuration.

And while looking further, $(format-json) is not either, as I've noticed
that value-pairs() is not thread safe either, because it uses a similar
GString instance that can be used from multiple threads.

Can you look into making them thread safe?

One way to do that is to allocate a new GString instance at every
invocation, which is not nice as that increases malloc() load. 


The alternative idea I was playing with was to create a new API for
managing per-thread scratch buffers that could be reused even between
multiple independent portions of syslog-ng.

The way it would work:
  - scratch buffers are a numbered array of GString buffers, allocated
    for _each_ thread.

  - as the message is processed various parts of syslog-ng allocate such
    buffers, do their thing with it.

  - in order to make it possible for a thread to work with scratch
    buffers, scratch_buffers_reset() would need to be called, once for
    each independent task (e.g. log message).
    


GString *scratch_buffer_acquire()

  Allocate a new scratch buffer, used to hold a temp value, that _will_
  automatically be freed in the future, when the current function call
  returns. The returned GString instance should not be saved to
  non-local variables.

void scratch_buffer_release(GString *scratch)

  Free the specified scratch buffer to be useful further in the call
  chain. May not be needed at all, I guesstimate that processing a
  single message would need 4-8 scratch buffers, which get reset at the
  end of the current message processing. Probably complicated free-alloc
  mechanism is not needed.

scratch_buffers_reset()

  Free all per-thread scratch buffers previously allocated.

With this API, the required number of GString buffers would be allocated
on-demand, but still they would not increase the malloc() load too much,
as the scratch buffers subsystem can keep GString instances around. The
code using GString buffers become simpler too, as the allocation to
store it somewhere vanishes.

What do you think?


-- 
Bazsi