May 2012 - syslog-ng - lists.balabit.hu

Connection pooling in syslog-ng for database destinations
by Rajalakshmi Iyer 28 May '12

28 May '12

Hello, I want to know if the syslog-ng destination plug-ins for writing logs directly into SQL databases or MongoDB use any connection pooling mechanism for connections to the back-end database? Regards, Raj -- Twitter: @Blismedia <http://twitter.com/#%21/blismobile> BlisMedia 32 Percy Street, London W1T 2DE www.blismedia.com <http://www.blismobile.com/> <http://www.blismobile.com/> [image: Follow on Twitter] <http://twitter.com/#%21/blismobile>[image: Blis Website] <http://www.blismobile.com/> This communication is from BlisMedia, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify info(a)blismedia.com<info(a)blismobile.com> by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it.

2 1

System Requirements for Syslog-ng
by Jerome Yanga 25 May '12

25 May '12

What are the system requirements for each mode? I have been trying to look for this piece of information in the site but I have not found it. Regards, j

2 1

remove header
by Richard F. Hart II 23 May '12

23 May '12

I have a juniper srx that is sending its syslogs to a syslog-ng server. Then I am having the syslog-ng server forward those juniper syslog messages to NItroSecurity ESM. However, I am having some difficutly. The syslog-ng server is appending a header to the beginning of the juniper syslog message when it sends it to the NitroSecurity ESM. How can I tell syslog-ng not to append its header to the beginning of the message? Here is a sample: *<14>May 22 08:31:03 syslog-ng.example.com* 2012-05-22T08:28:48.548 Juniper-SRX RT_FLOW - RT_FLOW_SESSION_CREATE [junos(a)2636.1.1.1.2.34source-address="192.168.1.34" source-port="40944" destination-address="4.2.2.2" destination-port="1984" service-name="None" nat-source-address="192.168.1.34" nat-source-port="40944" nat-destination-address="4.2.2.2" nat-destination-port="1984" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="6" policy-name="175008" source-zone-name="legacy" destination-zone-name="t-a" session-id-32="220372159" username="N/A" roles="N/A" packet-incoming-interface="reth4.0"] How do I stop syslog-ng from appending the underlined section? Thank you, Richard

3 2

.
by pedro castro 22 May '12

22 May '12

You can earn today money using your computer http://www.lekhajokha.in/hfdno.php?iriIDCid=855 Tue, 22 May 2012 13:59:18 ______________ That birds which fly with flapping wings derivetheir support from the air, is undoubtedly true,and that the lift produced is due, not to the form,or shape, or area of the wing, is also beyond question. jalyne aescleah

1 0

...
by pedro castro 20 May '12

20 May '12

Make aIot of money right from your PC http://kacchemicals.com/uuioino.php?ynenumberID=309 _______________ 126 86 36. emmett winefrith Sun, 20 May 2012 23:37:06

1 0

Rewriting CEF messages
by carlopmart 18 May '12

18 May '12

Hi all, I am trying to rewrite CEF messages that arrives to a syslog-ng instance. I am using the following rewrite rule: rewrite r_cef_logs { set("mydevice_logs", value("PROGRAM") condition(message("Device"))); }; Test message is: CEF:0|Device|Firewall|0|.... What am I doing wrong?? -- CL Martinez carlopmart {at} gmail {d0t} com

1 0

Introducing myself and GSoC syslog-ng MongoDB driver project
by Eun Kyung 18 May '12

18 May '12

Greetings, fellow developers. I recently joined the syslog-ng community because of Google Summer of Code, and I thought this would be a good time to briefly introduce myself. I am studying Information Technology in Dublin, Ireland, and I got accepted for GSoC 2012 to work on the MongoDB destination driver project. Syslog-ng, the leading logging platform for log collection and management, has been storing logs in MongoDB for over a year, but the technical documentation on the implementation is rudimentary, which impedes the improvement of the driver's functionality and speed. The goal of this project is to enhance the MongoDB destination driver for syslog-ng, to make it more robust, scalable and performant. Specific tasks and deliverables will include: - modifying syslog-ng's internal value pairs subsystem so that it will be able to emit a tree of values, instead of just a flat name:value list; - teaching the mongodb driver to walk the tree, and constructing BSON objects out of it, and using 'insert' instead of 'upsert'; - adding support for bulk inserts into a MongoDB, instead of inserting one by one My GSoC project proposal can be found at https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2012/eky… This is my first time working on an open source project, and I've liked a lot so far. My mentor has been tremendously friendly and helpful. I'm looking forward to a great summer of code with the syslog-ng community, and I would certainly like to keep you posted on the discussions and patches I will be making. Meanwhile, your comments and feedback would always be helpful. Best regards, Eun Kyung

1 0

receiving and forwarding snmp data
by Jason Kojro-Badziak 18 May '12

18 May '12

Hello all: I would like to use Syslog-NG to forward SNMP formatted data, in particular SNMP traps and some other SNMP data streams. I've been doing a lot of looking around today, and I've a few sites that have some recommended ways of doing things, but I'm wondering if anyone on this list has a tried and true way of doing it. Thank you! Jason Kojro-Badziak Monolith Software Staff Engineer 311 North 2nd Street, Suite #302 St. Charles, IL 60174 Office: 312-957-6470 x3010 Email: jbadziak(a)monolith-software.com<mailto:jbadziak@monolith-software.com/jb.022411>

2 1

[PATCH] jsonparser: Handle every kind of invalid input gracefully
by Gergely Nagy 18 May '12

18 May '12

Previously, we used json_tokener_parse_ex(), which always returned NULL in case of an error. A while ago, we switched to json_tokener_parse() instead, which never returns NULL, and instead, modifies the returned object to be an integer casted to pointer in case of an error. This resulted in the json-parser segfaulting whenever it encountered invalid input. The best course of action is to switch back to json_tokener_parse_ex(), because that way we can easily see when an error happened, and we even have easy access to the error message. In the long run, we should probably move back the tokener to the LogJSONParser object, and protect it with a mutex, for a bit better performance. Reported-by: Eun Kyung <ekyung01(a)googlemail.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/jsonparser/jsonparser.c | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/modules/jsonparser/jsonparser.c b/modules/jsonparser/jsonparser.c index 7331ad9..d95b0e2 100644 --- a/modules/jsonparser/jsonparser.c +++ b/modules/jsonparser/jsonparser.c @@ -166,6 +166,7 @@ log_json_parser_process (LogParser *s, LogMessage **pmsg, const LogPathOptions * { LogJSONParser *self = (LogJSONParser *) s; struct json_object *jso; + struct json_tokener *tok; if (self->marker) { @@ -177,7 +178,16 @@ log_json_parser_process (LogParser *s, LogMessage **pmsg, const LogPathOptions * input++; } - jso = json_tokener_parse (input); + tok = json_tokener_new (); + jso = json_tokener_parse_ex (tok, input, -1); + if (tok->err != json_tokener_success) + { + msg_error ("Unparsable JSON stream encountered", + evt_tag_str ("error", json_tokener_errors[tok->err]), NULL); + json_tokener_free (tok); + return FALSE; + } + json_tokener_free (tok); if (!jso) { -- 1.7.9

1 0

[PATCH] afmongodb: Fix a queue notify<->wakeup deadlock
by Gergely Nagy 18 May '12

18 May '12

As it turns out, there's a race condition between afmongodb_dd_queue_notify() and the worker thread: one is trying to lock queue_mutex to send a wakeup signal and clear the notification callback, and the other is using g_cond_wait() with the same queue_mutex locked around it. There appears to be a race between the two threads, and it can happen that both are trying to g_mutex_lock() the same mutex at the same time. To work around this problem, the patch below removes the queue_mutex locking from dd_queue_notify(): it's not required anyway. But it also moves the log_queue_reset_parallel_push() from the queue notify function to the other thread, where we're already holding the mutex after the g_cond_wait() anyway. (And do something similar in the suspend wakeup case too, where locking queue_mutex is also safe). This gets rid of the deadlock, and the code becomes easier to follow aswell. Reported-by: Eun Kyung <ekyung01(a)googlemail.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 5af34c9..7a1b6e2 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -388,6 +388,9 @@ afmongodb_worker_thread (gpointer arg) self->suspend_mutex, &self->writer_thread_suspend_target); self->writer_thread_suspended = FALSE; + g_mutex_lock(self->queue_mutex); + log_queue_reset_parallel_push(self->queue); + g_mutex_unlock(self->queue_mutex); g_mutex_unlock(self->suspend_mutex); } else @@ -398,6 +401,7 @@ afmongodb_worker_thread (gpointer arg) if (log_queue_get_length(self->queue) == 0) { g_cond_wait(self->writer_thread_wakeup_cond, self->queue_mutex); + log_queue_reset_parallel_push(self->queue); } g_mutex_unlock(self->queue_mutex); } @@ -560,10 +564,7 @@ afmongodb_dd_queue_notify(gpointer user_data) { MongoDBDestDriver *self = (MongoDBDestDriver *)user_data; - g_mutex_lock(self->queue_mutex); g_cond_signal(self->writer_thread_wakeup_cond); - log_queue_reset_parallel_push(self->queue); - g_mutex_unlock(self->queue_mutex); } static void -- 1.7.9

1 0