We need to order the keys to generate json from our flat name value structure. There are a lot of things that make format-json slow, and could be sped up easily, at least in my opinion. Sorting doesn't help either, but unfortunately that part is needed. On Oct 31, 2015 3:52 PM, "Evan Rempel" <erempel@uvic.ca> wrote:

I am surprised that it orders them at all. The json object is key name specific, with ordering of list elements the only place where order could be important.

I would think that sorting the keys is just a waste of time. The json format output is currently performance limited to around 15,000 events per second on the latest generation of CPU.

I would be in favor of doing ANYTHING that could speed that up. Perhaps the ability to sort, and the ascending/descending order should be options to the json output template.

Evan

"Scheidler, Balázs" <balazs.scheidler@balabit.com> wrote:

Hi,

I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that?

This is the only patch needed to fix the order, but I've figured there may have been some reason behind the ordering.

static gint vp_walk_cmp(const gchar *s1, const gchar *s2) { - return strcmp(s2, s1); + return strcmp(s1, s2); }

Thanks in advance,

-- Bazsi

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Hi, -- Bazsi On Mon, Nov 2, 2015 at 6:39 AM, Gergely Nagy <algernon@madhouse-project.org> wrote:

On Sat, Oct 31, 2015 at 1:28 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:

...

Hi,

I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that?

It's for the flat format => structured format conversion. Consider you have keys like a.b.c, a.b.d, a.b.e, a.c.a, a.c.f. With reverse sorting, you get a.c.f first, and generate an f key, then a, and collect that into c. Then you get the b stuff, and then wrap them all in a. It may be possible to do it the other way around, generating a first, and extending it, but this order is more straightforward in my opinion.

It seems to have worked by flipping the order, but maybe I wasn't testing it enough. I don't understand your example, though, "f", then "a", then "c", the original example was a.c.f But since there's a definite reason for the ordering, I'm dropping the reordering patch.

I also seem to remember needing this for array support, but I'm not exactly sure of that.

this makes sense at least as long as the index is alphabetically sortable (e.g. the same length), with stuff where indexes contain variable length numbers (10 sorts before 9), it probably wouldn't work.

As for getting rid of sorting: that'd just make things slower in the end, unless syslog-ng starts storing its key-value pairs in an already structured format. But even then, my gut feeling is that sorting is still faster. If the data is unsorted, you'd have to hunt down where to insert the new keys, possibly deep into other structures. That sounds very inefficient. Also considerably more code

Yup, sorting is an architectural must at least now.