Gergely Nagy <algernon@madhouse-project.org> writes:

"Lucas, Sascha" <Sascha.Lucas@gisa.de> writes:

...

I'm again looking for a way to store patterndb classified / tagged events in mongodb. My greatest wish is to store the event tags as an array (list) in mongodb. This would greatly enhance querying for different combinations of tags. ATM TAGS is a string. According to the documentation, there seems no way to "type hint" that tags is an array. Or do I miss something?

There is currently no way to do that. I have a work-in-progress branch[1] to address that in a fairly crude manner, but it only works for $(format-json) at the moment. It should be fairly easy to take that branch forward and add MongoDB support too. I may do that later today, but I only have resources to prepare a proof of concept patch, can't clean it up and submit it properly.

[1]: https://github.com/algernon/syslog-ng/tree/f/3.6/type-hint/list

I just updated the patch[1], and with that, MongoDB supports things like this: destination d_mongo_tagged { mongodb(database("algernon") collection("devel") value-pairs(key(MESSAGE) pair("TAGS" list("${TAGS}")))); }; Resulting in something like this:

db.devel.find() { "_id" : ObjectId("541bd0122149cd40ff95a6fe"), "tags" : [ ".source.#anon-source0" ], "MESSAGE" : "seq: 0000000000, thread: 0000, runid: 1411108882, ..." }

[1]: https://github.com/algernon/syslog-ng/commit/611c4e7aba6c7814be0dc2d96332a77... -- |8]

Hi Gergely, From: Gergely Nagy Date: Fri, 19. Sep 2014 08:49

I just updated the patch[1], and with that, MongoDB supports things [1]: https://github.com/algernon/syslog-ng/commit/611c4e7aba6c7814be0dc2d96332a77...

It seems, that storing a list-type-hinted $TAGS in mongoDB effects other keys stored also as list. I.E. SOURCEIP and SEQNUM are mostly, HOST sometimes, lists. Does this make sense to you? A workaround ATM is, to type hint every other key as string. I've applied your patch on syslog-ng-3.5.6. The following is an example showing miss-typed keys SOURCEIP and HOST: { "_id" : ObjectId("5429630324ad83f366000055"), "owner" : { "group" : "XXX" }, "_SDATA" : { "origin" : { "ip" : "Y.Y.Y.Y" } }, "TAGS" : [ ".source.s_network", ".classifier.assigned", ".classifier.not_interesting" ], "SOURCEIP" : [ "Z.Z.Z.Z" ], "SEQNUM" : "85", "RCVDATE" : "2014-09-29T15:47:47+02:00", "PROGRAM" : "SAPAAA_00", "PRIORITY" : "err", "PID" : "5222", "MESSAGE" : "AAA Basis System: Stop Workp. 19, PID 5222", "ISODATE" : "2014-09-29T15:47:47+02:00", "HOST" : [ "test" ], "FACILITY" : "local0", "DATE" : "Sep 29 15:47:47" } Thanks, Sascha. Aufsichtsratsvorsitzender: Herbert Vogel Geschäftsführung: Michael Krüger Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.