syslog-ng with mongodb user survey
Hello, Mongodb support is available in syslog-ng for more than a year now. We are aware, that there are many people using it, but now we would like to ask for some feedback about the use cases and environments. Here are just a few sample questions, but any feedback regarding mongodb support is very welcome: - average and peak message rates sent to the mongodb destination - what is your related syslog-ng.conf - how did you tune your mongodb server (if you did) - how and how often do you query your logs in mongodb - etc. Thanks for your help! Bye, -- Peter Czanik (CzP) <czanik@balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/
Hi, thanks a lot for interest. So here is my feedback: We have 15-20e6 messages per day and keep them stored for 20 days in a mongodb collection. Here are my mongodb stats:
db.messages.stats() { "ns" : "test.messages", "count" : 374862071, "size" : 208169470372, "avgObjSize" : 555.322841323149, "storageSize" : 320632751840, "numExtents" : 170, "nindexes" : 9, "lastExtentSize" : 2146426864, "paddingFactor" : 1, "systemFlags" : 1, "userFlags" : 1, "totalIndexSize" : 255933041840, "indexSizes" : { "_id_" : 12162503136, "HOST_1" : 24961180832, "RCVDATE_-1" : 30299888080, "FACILITY_1" : 14748686400, "PRIORITY_1" : 13601961520, "PROGRAM_1" : 13947577392, "HOST_1_RCVDATE_-1" : 48404094816, "HOST_1_RCVDATE_-1_PRIORITY_1_PROGRAM_1_FACILITY_1" : 64082384240, "RCVDATE_1_FACILITY_1" : 33724765424 }, "ok" : 1 }
A nightly cron job does expiration, export to file (CSV XZ-compressed) and generates a report (atm only based on priority). We have a custom web-UI (mojology is nice, but does not fit our query needs). Logs a searched non regularly (by hand, a few times a day). This number of logs is searchable (mostly the cache is polluted after some time of writing logs and doing no queries). mongo tuning was limited to do the right query and to have the right index. The server specs are: 6-Core-Xeon, 64GB RAM, 6x10k SAS-RAID5. OS is SLES11-SP2 with a self compiled syslog-ng-3.4.1. syslog-ng.conf has nothing special, expect that I use pair("RCVDATE" "$R_ISODATE") and pair("ISODATE" "$ISODATE") instead of unixtime. The nightly crons forces me to have log_fifo_size(500000), else I'm losing logs. A nagios check parses the output of "syslog-ng-ctl stats" and watches for dropped logs. There are some mongo specific features we like to use, but they are missing atm: * use TTL collection (the nightly deletion is slow) -> store data type DATE * store data type array for tags and classes (patterndb) -> makes it searchable by mongo And there are problems with syslog-ng itself (some reported earlier on this list): * our network destination: Invalid byte sequence or other error while converting input, skipping character; encoding='UTF-8', char='0xf0' * a unknown problem, where syslog-ng stops logging/scrambles logs after (frequent) SIGHUP * a unknown problem, where syslog-ng stops working and blocks most of the system: i.e. sshd or sudo waiting to log to /dev/log Thanks, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.
Hi! "Lucas, Sascha" <Sascha.Lucas@gisa.de> writes:
thanks a lot for interest. So here is my feedback:
Thanks a lot for your feedback, it's very valuable!
There are some mongo specific features we like to use, but they are missing atm: * use TTL collection (the nightly deletion is slow) -> store data type DATE
This is being worked on, and will likely be part of syslog-ng 3.5.
* store data type array for tags and classes (patterndb) -> makes it searchable by mongo
I have not started working on this yet, but once the type hinting support is in, adding this should be fairly painless.
And there are problems with syslog-ng itself (some reported earlier on this list):
* our network destination: Invalid byte sequence or other error while converting input, skipping character; encoding='UTF-8', char='0xf0'
I'm afraid I have nothing to offer for this problem at this time, will look into it as soon as I can find some time.
* a unknown problem, where syslog-ng stops logging/scrambles logs after (frequent) SIGHUP
Are you using file sources, by any chance?
* a unknown problem, where syslog-ng stops working and blocks most of the system: i.e. sshd or sudo waiting to log to /dev/log
We've seen something similar when the supress() option was used, that dead-locked syslog-ng in a few cases. There's also a race condition in afmongodb that was recently fixed in git, but that should not result in syslog-ng blocking (rather, it would lead to a crash, rarely). -- |8]
Hi Gergely, From: Gergely Nagy Date: Tue, 28. May 2013 15:31
Thanks a lot for your feedback, it's very valuable!
Your welcome. I just missed to tell a imported mongo setting: Beside the right query and index I have to set: db.runCommand({collMod:"messages", usePowerOf2Sizes:"true"}) This does some kind of padding to avoid "fragmentation" of deleted space. Else mongo won't be able to reuse deleted space and your DB just grows... Another syslog-setting I have set is encoding(UTF-8) in the TCP/UDP source. Else non UTF-8 chars will be inserted into mongodb. This is really bad. Because all queries stop at the document containing the invalid char.
* our network destination: Invalid byte sequence or other error while converting input, skipping character; encoding='UTF-8', char='0xf0'
Is this somehow related to my setting encoding(UTF-8)?
* a unknown problem, where syslog-ng stops logging/scrambles logs after (frequent) SIGHUP
Are you using file sources, by any chance?
No. Just TCP and UDP.
* a unknown problem, where syslog-ng stops working and blocks most of the system: i.e. sshd or sudo waiting to log to /dev/log We've seen something similar when the supress() option was used, that
That's a good point. I used suppress() for internal() going to a file and even on a pipe() destination getting all logs from the network. For now I removed suppress(). Also I had threaded(yes) in global options(), which I also disable for now. Using strace on the "hanging, threaded, suppressing" syslog-ng process just shows syscall futex() hanging... Thanks, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.
participants (3)
-
Gergely Nagy
-
Lucas, Sascha
-
Peter Czanik