- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog_ng 3.3.4 make error on HPUX HPPA 11.11
by Lennert Buytenhek 31 May '12

31 May '12

FWIW, I'm not sure what's going on here. On Tue, May 29, 2012 at 02:34:06PM +0530, Rakesh Rajasekharan wrote: > I have attached my libpthread.a file (under /usr/lib). > I though , have the "libpthread.a" file under all the below locations, but, > I am no tsure which one exactly gets picked up or needs to be picked up > while compiling. > > # find / -name 'libpthread.a' > /usr/lib/pa20_64/libp/libpthread.a > /usr/lib/pa20_64/libpthread.a > /usr/lib/libp/libpthread.a > /usr/lib/libpthread.a > /var/adm/sw/save/PHCO_26466/C-MIN-32ALIB/usr/lib/libpthread.a > /var/adm/sw/save/PHCO_26466/C-MIN-64ALIB/usr/lib/pa20_64/libpthread.a > > On Tue, May 29, 2012 at 2:10 PM, Lennert Buytenhek > <buytenh(a)wantstofly.org>wrote: > > > On Tue, May 29, 2012 at 12:18:10PM +0530, Rakesh Rajasekharan wrote: > > > > > /bin/sh ../../libtool --tag=CC --mode=link gcc -Wall -g -O2 -o client > > > client.o ../libivykis.la -lpthread > > > libtool: link: gcc -Wall -g -O2 -o client client.o ../.libs/libivykis.a > > > -lpthread > > > /usr/ccs/bin/ld: Unsatisfied symbols: > > > pthread_sigmask (first referenced in ../.libs/libivykis.a(iv_main.o)) > > > (data) > > > pthread_mutex_init (first referenced in > > ../.libs/libivykis.a(iv_main.o)) > > > (data) > > > pthread_mutex_destroy (first referenced in > > > ../.libs/libivykis.a(iv_main.o)) (data) > > > pthread_mutex_unlock (first referenced in > > > ../.libs/libivykis.a(iv_main.o)) (data) > > > pthread_mutex_lock (first referenced in > > ../.libs/libivykis.a(iv_main.o)) > > > (data) > > > > Can you send me a copy of your actual libpthread.{a,so} ? > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

3 3

Cancle
by Weerawat Chawakhiew 30 May '12

30 May '12

1 0

[PATCH 0/3] WIP: Convert syslog-ng to use upstream ivykis
by Gergely Nagy 30 May '12

30 May '12

A long long time ago, upstream and BalaBit's version of ivykis started to diverge, which - at the time - was necessary, but the intent always was to eventually get things upstream and merged, to allow syslog-ng to use upstream ivykis, as-is. With the help of Lennert Buytenhek (ivykis upstream) and Balazs Scheidler, I present you the first work in progress patch set against syslog-ng that enables it to work with an unmodified ivykis! The patch set is not 100% complete yet however, I'm only submitting it for review and possible testing. I will probably port it to 3.3 too (it applies almost cleanly, there's only one trivial conflict in lib/Makefile.am) and build the Debian packages using upstream ivykis to get wider testing. What is missing, is the old lib/ivykis/configure.gnu script, which we had there to force a static build of ivykis. This will be replaced by a modification to autogen.sh, that will automatically create the configure.gnu scripts in the various submodules, so we can point the submodule links to upstream repos instead of our own forks. This is not included in this set yet, and one will need to copy that file over from the BalaBit fork, or things won't work too well. Another thing is that the first patch stops libsyslog-ng-crypto from being directly linked to ivykis. This is not an issue when ivykis is statically compiled into libsyslog-ng, since all those symbols will be exported to other modules, including libsyslog-ng-crypto. However, it might become a problem when using a dynamically linked ivykis. I haven't tested that case yet. The reason I remove the link is that if ivykis is statically compiled in to both shared objects, the constructors will run twice, as dlopen()ing anything linked against libsyslog-ng-crypto will trigger them again. This leads to an abort() inside ivykis. This can be done on the ivykis side too: separate the constructors from the init functions, and make the constructors no-op if they ran already: this allows the init functions to continue aborting if called more than once, but allows the constructors to run twice. I have to discuss this with upstream, and see if we need this at all, or if my workaround of removing the linkage from libsyslog-ng-crypto is enough and without side effects. These were the easy issues, which I could solve right now if I had some coffee left. But alas, there is another issue! With the first two patches applied, we'd have a syslog-ng that compiles fine against upstream ivykis, and even works. Except it doesn't: whenever we try to set an io handler for an fd, we'd end up in an abort, because for some reason, we already had a handler set. I do not understand it quite yet why this happens - the old BalaBit fork of ivykis was similarly picky and abort-happy and they did not trigger. As a workaround, I created the third patch, because I wanted to have something that shows signs of working. The third patch is *WRONG*, but it's an acceptable workaround until I have time to debug the problem further. What it does, is that it *always* clears the io handlers before registering new ones. This results in marginally more work being done each time we need to set a handler, but not by much, and as far as I saw, it has no side effects. But since the part of ivykis that aborts before the third patch, did not change between BalaBit's fork and upstream, I'm certain that the second patch introduces a bug that wasn't there before. Nevertheless! The good news still is, that we're *very* close to being able to build with upstream ivykis! A few issues to fix, and then comes the next part: going over all the changes that were made to the BalaBit fork of ivykis, and send everything upstream that has not been sent yet. That will be another funny adventure, but at least, by that time, I will understand what the hell I am doing =) -- |8]

3 16

Howto configure grep to parse multiple mail recipients
by C. L. Martinez 30 May '12

30 May '12

Hi all, I am trying to parse some Ironport logs to cacth mutliple destination recipients using grep in value option: recipient: $(grep ("${to_address}" != "") ${to_address}) but it doesn't works ... What am I doing wrong?? I am using syslog-ng 3.3.5 under CentOS6. Thanks.

2 4

Completing an empty field with specific value
by C. L. Martinez 30 May '12

30 May '12

Hi all, I am trying to complete an empty field with a specific value when I need to correlate postfix logs. Sometimes From filed comes empty: May 29 08:18:04 testsrv postfix/smtp[1569]: E2B653F2: from=<> ... I have tried to do this using "value" options in my patterndb.xml: a) <value name="nosender">$(if ("${from_address}" == "") "no-sender(a)no-domain.com" "${from_address}@7")</value> b) <value name="nosender">$(grep "${from_address}" == "") "no-sender(a)no-domain.com"</value> ... but it doesn't works ... Any help??

1 1

file("/proc/kmsg" program_override("kernel: "));
by Tom 29 May '12

29 May '12

Dear all, I just started using syslog-ng to pipe log files to Loggly.com. I have a couple of questions. First of all, why is there a `: ` after kernel here? file("/proc/kmsg" program_override("kernel: ")); I have seen people do it differently, like so: file("/proc/kmsg" program_override("kernel)); Does this matter? And more general, what does the entire rule do exactly? The following is my syslog-ng.conf file: https://gist.github.com/2819768 As you can see, I have a lot of log files. Is this the proper way of doing this? Doesn't it cost a lot of resources for syslog-ng to check all these files for changes? Finally, where can I find the documentation? Eg. I would like to learn about the specifics of follow_freq and what the parameter actually implies. Thanks a lot for your help. Kind regards, Tom

3 2

Writing JSON message to a MongoDB destination
by Rajalakshmi Iyer 29 May '12

29 May '12

Hello, I have a log message in JSON format that needs to be written to the MongoDB database, such that I should be able to later query the database based on fields in the JSON record. However, when I try to use the syslog-ng mongodb destination driver using : destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") collection("messages") value-pairs(key("MESSAGE")) ); }; The record appears in the mongodb database as: {"MESSAGE":<My JSON formatted message>} I do not want the MESSAGE macro to be part of the record in the database, I only want to it's JSON-encoded value to be entered in the database. Is this possible? Thanks in advance for your help. Regards, Raj -- Twitter: @Blismedia <http://twitter.com/#%21/blismobile> BlisMedia 32 Percy Street, London W1T 2DE www.blismedia.com <http://www.blismobile.com/> <http://www.blismobile.com/> [image: Follow on Twitter] <http://twitter.com/#%21/blismobile>[image: Blis Website] <http://www.blismobile.com/> This communication is from BlisMedia, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify info(a)blismedia.com<info(a)blismobile.com> by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it.

2 1

(no subject)
by pedro castro 28 May '12

28 May '12

http://cabinetbianchi.re/tenyun.php?top=288 Learn h0w t0 turn successful fr0m h0me ___ Indestructibility.Not capable of being injured or destroyed. damita aekerman

1 0

Connection pooling in syslog-ng for database destinations
by Rajalakshmi Iyer 28 May '12

28 May '12

Hello, I want to know if the syslog-ng destination plug-ins for writing logs directly into SQL databases or MongoDB use any connection pooling mechanism for connections to the back-end database? Regards, Raj -- Twitter: @Blismedia <http://twitter.com/#%21/blismobile> BlisMedia 32 Percy Street, London W1T 2DE www.blismedia.com <http://www.blismobile.com/> <http://www.blismobile.com/> [image: Follow on Twitter] <http://twitter.com/#%21/blismobile>[image: Blis Website] <http://www.blismobile.com/> This communication is from BlisMedia, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify info(a)blismedia.com<info(a)blismobile.com> by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it.

2 1

System Requirements for Syslog-ng
by Jerome Yanga 25 May '12

25 May '12

What are the system requirements for each mode? I have been trying to look for this piece of information in the site but I have not found it. Regards, j

2 1

remove header
by Richard F. Hart II 23 May '12

23 May '12

I have a juniper srx that is sending its syslogs to a syslog-ng server. Then I am having the syslog-ng server forward those juniper syslog messages to NItroSecurity ESM. However, I am having some difficutly. The syslog-ng server is appending a header to the beginning of the juniper syslog message when it sends it to the NitroSecurity ESM. How can I tell syslog-ng not to append its header to the beginning of the message? Here is a sample: *<14>May 22 08:31:03 syslog-ng.example.com* 2012-05-22T08:28:48.548 Juniper-SRX RT_FLOW - RT_FLOW_SESSION_CREATE [junos(a)2636.1.1.1.2.34source-address="192.168.1.34" source-port="40944" destination-address="4.2.2.2" destination-port="1984" service-name="None" nat-source-address="192.168.1.34" nat-source-port="40944" nat-destination-address="4.2.2.2" nat-destination-port="1984" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="6" policy-name="175008" source-zone-name="legacy" destination-zone-name="t-a" session-id-32="220372159" username="N/A" roles="N/A" packet-incoming-interface="reth4.0"] How do I stop syslog-ng from appending the underlined section? Thank you, Richard

3 2

.
by pedro castro 22 May '12

22 May '12

You can earn today money using your computer http://www.lekhajokha.in/hfdno.php?iriIDCid=855 Tue, 22 May 2012 13:59:18 ______________ That birds which fly with flapping wings derivetheir support from the air, is undoubtedly true,and that the lift produced is due, not to the form,or shape, or area of the wing, is also beyond question. jalyne aescleah

1 0

...
by pedro castro 20 May '12

20 May '12

Make aIot of money right from your PC http://kacchemicals.com/uuioino.php?ynenumberID=309 _______________ 126 86 36. emmett winefrith Sun, 20 May 2012 23:37:06

1 0

Rewriting CEF messages
by carlopmart 18 May '12

18 May '12

Hi all, I am trying to rewrite CEF messages that arrives to a syslog-ng instance. I am using the following rewrite rule: rewrite r_cef_logs { set("mydevice_logs", value("PROGRAM") condition(message("Device"))); }; Test message is: CEF:0|Device|Firewall|0|.... What am I doing wrong?? -- CL Martinez carlopmart {at} gmail {d0t} com

1 0

Introducing myself and GSoC syslog-ng MongoDB driver project
by Eun Kyung 18 May '12

18 May '12

Greetings, fellow developers. I recently joined the syslog-ng community because of Google Summer of Code, and I thought this would be a good time to briefly introduce myself. I am studying Information Technology in Dublin, Ireland, and I got accepted for GSoC 2012 to work on the MongoDB destination driver project. Syslog-ng, the leading logging platform for log collection and management, has been storing logs in MongoDB for over a year, but the technical documentation on the implementation is rudimentary, which impedes the improvement of the driver's functionality and speed. The goal of this project is to enhance the MongoDB destination driver for syslog-ng, to make it more robust, scalable and performant. Specific tasks and deliverables will include: - modifying syslog-ng's internal value pairs subsystem so that it will be able to emit a tree of values, instead of just a flat name:value list; - teaching the mongodb driver to walk the tree, and constructing BSON objects out of it, and using 'insert' instead of 'upsert'; - adding support for bulk inserts into a MongoDB, instead of inserting one by one My GSoC project proposal can be found at https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2012/eky… This is my first time working on an open source project, and I've liked a lot so far. My mentor has been tremendously friendly and helpful. I'm looking forward to a great summer of code with the syslog-ng community, and I would certainly like to keep you posted on the discussions and patches I will be making. Meanwhile, your comments and feedback would always be helpful. Best regards, Eun Kyung

1 0

receiving and forwarding snmp data
by Jason Kojro-Badziak 18 May '12

18 May '12

Hello all: I would like to use Syslog-NG to forward SNMP formatted data, in particular SNMP traps and some other SNMP data streams. I've been doing a lot of looking around today, and I've a few sites that have some recommended ways of doing things, but I'm wondering if anyone on this list has a tried and true way of doing it. Thank you! Jason Kojro-Badziak Monolith Software Staff Engineer 311 North 2nd Street, Suite #302 St. Charles, IL 60174 Office: 312-957-6470 x3010 Email: jbadziak(a)monolith-software.com<mailto:jbadziak@monolith-software.com/jb.022411>

2 1

[PATCH] jsonparser: Handle every kind of invalid input gracefully
by Gergely Nagy 18 May '12

18 May '12

Previously, we used json_tokener_parse_ex(), which always returned NULL in case of an error. A while ago, we switched to json_tokener_parse() instead, which never returns NULL, and instead, modifies the returned object to be an integer casted to pointer in case of an error. This resulted in the json-parser segfaulting whenever it encountered invalid input. The best course of action is to switch back to json_tokener_parse_ex(), because that way we can easily see when an error happened, and we even have easy access to the error message. In the long run, we should probably move back the tokener to the LogJSONParser object, and protect it with a mutex, for a bit better performance. Reported-by: Eun Kyung <ekyung01(a)googlemail.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/jsonparser/jsonparser.c | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/modules/jsonparser/jsonparser.c b/modules/jsonparser/jsonparser.c index 7331ad9..d95b0e2 100644 --- a/modules/jsonparser/jsonparser.c +++ b/modules/jsonparser/jsonparser.c @@ -166,6 +166,7 @@ log_json_parser_process (LogParser *s, LogMessage **pmsg, const LogPathOptions * { LogJSONParser *self = (LogJSONParser *) s; struct json_object *jso; + struct json_tokener *tok; if (self->marker) { @@ -177,7 +178,16 @@ log_json_parser_process (LogParser *s, LogMessage **pmsg, const LogPathOptions * input++; } - jso = json_tokener_parse (input); + tok = json_tokener_new (); + jso = json_tokener_parse_ex (tok, input, -1); + if (tok->err != json_tokener_success) + { + msg_error ("Unparsable JSON stream encountered", + evt_tag_str ("error", json_tokener_errors[tok->err]), NULL); + json_tokener_free (tok); + return FALSE; + } + json_tokener_free (tok); if (!jso) { -- 1.7.9

1 0

[PATCH] afmongodb: Fix a queue notify<->wakeup deadlock
by Gergely Nagy 18 May '12

18 May '12

As it turns out, there's a race condition between afmongodb_dd_queue_notify() and the worker thread: one is trying to lock queue_mutex to send a wakeup signal and clear the notification callback, and the other is using g_cond_wait() with the same queue_mutex locked around it. There appears to be a race between the two threads, and it can happen that both are trying to g_mutex_lock() the same mutex at the same time. To work around this problem, the patch below removes the queue_mutex locking from dd_queue_notify(): it's not required anyway. But it also moves the log_queue_reset_parallel_push() from the queue notify function to the other thread, where we're already holding the mutex after the g_cond_wait() anyway. (And do something similar in the suspend wakeup case too, where locking queue_mutex is also safe). This gets rid of the deadlock, and the code becomes easier to follow aswell. Reported-by: Eun Kyung <ekyung01(a)googlemail.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 5af34c9..7a1b6e2 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -388,6 +388,9 @@ afmongodb_worker_thread (gpointer arg) self->suspend_mutex, &self->writer_thread_suspend_target); self->writer_thread_suspended = FALSE; + g_mutex_lock(self->queue_mutex); + log_queue_reset_parallel_push(self->queue); + g_mutex_unlock(self->queue_mutex); g_mutex_unlock(self->suspend_mutex); } else @@ -398,6 +401,7 @@ afmongodb_worker_thread (gpointer arg) if (log_queue_get_length(self->queue) == 0) { g_cond_wait(self->writer_thread_wakeup_cond, self->queue_mutex); + log_queue_reset_parallel_push(self->queue); } g_mutex_unlock(self->queue_mutex); } @@ -560,10 +564,7 @@ afmongodb_dd_queue_notify(gpointer user_data) { MongoDBDestDriver *self = (MongoDBDestDriver *)user_data; - g_mutex_lock(self->queue_mutex); g_cond_signal(self->writer_thread_wakeup_cond); - log_queue_reset_parallel_push(self->queue); - g_mutex_unlock(self->queue_mutex); } static void -- 1.7.9

1 0

LogZilla v4.0
by Clayton Dukes 17 May '12

17 May '12

Hi Everyone, Just wanted to let you all know that I have released v4.0. This new version includes some features that many of the users on the syslog-ng list asked for, so I wanted to share the good news :-) Here's more info on this release: http://www.logzilla.pro/news/releases/4.0 ______________________________________________________________ Clayton Dukes ______________________________________________________________

1 0

Re: [syslog-ng] issue during start up
by Lay, James 16 May '12

16 May '12

Hello all: I am afraid I am having another issue which I would like some help with. When I try to start syslog-ng, I am getting this error: Starting syslog-ng: Error initializing raw socket, spoof-source support disabled; Syslog-ng then starts, but it doesn't appear to be forwarding messages. This is extremely strange as I have syslog-ng installed and running with the exact same config file on another system without any issues. I am sure the problem is that I'm an idiot, but I would greatly appreciate any help with the error I am having. Thank you! Try doing a: ldd `which syslog-ng` on both boxes....see if they are different. James

3 4

Problem with Syslog-NG 3.2.5 on Aix 7.1 ... It coredumps..
by Jonathan Kaufman 16 May '12

16 May '12

Hello All, I am hoping someone out there has a working installation of 3.2.5 on Aix 7.1 and can help me get mine working. I am currently trying to use a self-compiled version of 3.2.5 on an Power7 CPU Aix 7.1 TL1 SP3 box. I was able to get Syslog-NG to compile on Aix 7.1 after I mangled a new libtool release into the source so it would compile shared objects. Unfortunately, when I start syslog-ng it core dumps repeatedly. Core dumps start filling up /sftw/syslog-ng/var and a ps -ef shows only the "supervising syslog-ng process". a dbx syslog-ng gives this output... [root@mlwitt71]:[/sftw/syslog-ng/sbin] > dbx syslog-ng Type 'help' for help. [using memory image in core.10092546.01150742] reading symbolic information ... Segmentation fault in alloca at 0xd2970930 ($t1) 0xd2970930 (alloca+0x8) 800c0000 lwz r0,0x0(r12) (dbx) where alloca() at 0xd2970930 evt_str_append_escape_bs() at 0xd2970520 evtrec_format_plain() at 0xd296feb4 evt_format() at 0xd2970104 msg_event_send(e = 0x00000005), line 166 in "messages.c" main_loop_run(cfg = (nil)), line 148 in "main.c" main(argc = 1, argv = 0x2ff227f4), line 470 in "main.c" (dbx) I have also tried using the Syslog-ng rpm's from perzl & Bull. Both of them core dump as well, so I went back to trying to get a self-compiled release to work as I have some control over that. I used IBM VisualAge C/C++ v11.1.0.9 as the C compiler. [root@mlwitt71]:[/sftw/syslog-ng/sbin] > lslpp -L | grep vac vac.Bnd 11.1.0.1 C F XL C for AIX Media Defined vac.C 11.1.0.9 C F IBM XL C Compiler vac.aix53.lib 11.1.0.9 C F XL C for AIX Libraries for AIX vac.include 11.1.0.9 C F IBM XL C Compiler Include vac.lib 11.1.0.9 C F XL C for AIX Libraries vacpp.Bnd 11.1.0.1 C F IBM XL C/C++ Media Defined vacpp.cmp.aix53.lib 11.1.0.9 C F IBM XL C/C++ Libraries for AIX vacpp.cmp.core 11.1.0.9 C F IBM XL C/C++ Compiler vacpp.cmp.include 11.1.0.9 C F IBM XL C/C++ Compiler Include vacpp.cmp.lib 11.1.0.9 C F IBM XL C/C++ Libraries vacpp.cmp.rte 11.1.0.9 C F IBM XL C/C++ Compiler vacpp.cmp.tools 11.1.0.9 C F IBM XL C/C++ Tools vacpp.tnb 11.1.0.1 C F IBM XL C/C++ Evaluation I mangled libtool 2.4.2 into the source directories so it would detect/compile shared libraries (it wouldn't otherwise). I compiled and installed eventlog 0.2.12 in /sftw/syslog-ng, I also compiled OpenSSL 1.0.0g into it staticly. I am initially trying to start it using the default configuration files. There were no errors during the compile, and a syslog-ng -s did NOT coredump. I believe syslog-ng is using the following libraries and their locations (dynamically linked). [root@mlwitt71]:[/sftw/syslog-ng/sbin] > ldd syslog-ng syslog-ng needs: /sftw/syslog-ng/lib/libsyslog-ng.a(libsyslog-ng.so.0) /usr/lib/libnsl.a(shr.o) /opt/freeware/lib/libgmodule-2.0.so /opt/freeware/lib/libglib-2.0.so /usr/lib/libpthread.a(shr_xpg5.o) /usr/lib/libc.a(shr.o) /sftw/syslog-ng/lib/libevtlog.a(libevtlog.so.0) /usr/lib/librtl.a(shr.o) /opt/freeware/lib/libpcre.a(libpcre.so.0) /usr/lib/libthread.a(shr.o) /usr/lib/libpthreads_compat.a(shr.o) /usr/lib/libpthreads.a(shr_xpg5.o) /usr/lib/libtli.a(shr.o) /opt/freeware/lib/libglib-2.0.a(libglib-2.0.so.0) /opt/freeware/lib/libintl.a(libintl.so.1) /usr/lib/libiconv.a(shr4.o) /usr/lib/libpthreads.a(shr_comm.o) /unix /usr/lib/libcrypt.a(shr.o) /usr/lib/libpthreads.a(shr.o) /usr/lib/libc.a(pse.o) and the configure command when compiling from source was: ./configure --prefix=/sftw/syslog-ng \ --disable-spoof-source \ --enable-dynamic-linking \ --enable-debug \ --enable-ssl and I added -g to the CFLAGS I am using glib2-2.28.6-1 from perzl as well.. I plan on removing the --enable-debug if I can get it working Any ideas? Jonathan Kaufman

8 10

issue during start up
by Jason Kojro-Badziak 15 May '12

15 May '12

Hello all: I am afraid I am having another issue which I would like some help with. When I try to start syslog-ng, I am getting this error: Starting syslog-ng: Error initializing raw socket, spoof-source support disabled; Syslog-ng then starts, but it doesn't appear to be forwarding messages. This is extremely strange as I have syslog-ng installed and running with the exact same config file on another system without any issues. I am sure the problem is that I'm an idiot, but I would greatly appreciate any help with the error I am having. Thank you! Jason Kojro-Badziak Monolith Software Staff Engineer 311 North 2nd Street, Suite #302 St. Charles, IL 60174 Office: 312-957-6470 x3010 Email: jbadziak(a)monolith-software.com<mailto:jbadziak@monolith-software.com/jb.022411>

1 0

(no subject)
by pedro castro 13 May '12

13 May '12

Learn how to turn successfuI from home http://www.icarografica.it/kkimom.php?jdolpin=349 Sun, 13 May 2012 12:17:24 ____________ We DIDset him free--me and Tom. venize addyson

1 0

RFC: Applying transformations to a whole log message
by Gergely Nagy 10 May '12

10 May '12

Hi! In the GeoIP thread[1], I started to play with the idea of introducing another way to modify messages. So far, we have rewrite, which can set new values associated with a message, or change existing ones - one at a time. We also have template functions, which one can use how a specific value will be formatted. Again, pretty much one at a time. What syslog-ng lacks right now, is a way to apply a transformation to a message as a whole, a transformation that will take effect right there, right then, instead of making a modified copy like value-pairs() does. (value-pairs() also suffers from the problem that to be useful, it needs explicit support elsewhere: among the template functions, or within the destination driver). What I wish for, is to be able to apply any number of transformation functions to a whole LogMessage. Whether the transformations rewire the key names, or change values, I'd love to be able to just tell syslog-ng, that "here, take this message, go out and prosper, make it better, whatever it takes!" - and it would do just that. To give a few examples, I'd love to be able to do any and all of the following: * Ask syslog-ng to take a message, and look up every IP address associated with it (for simplicity's sake, lets assume every such address is stored within a key that ends with "_IP", and no other keys end with the same suffix) in a GeoIP database, and put the result in keys that have a "geo." prefix, followed by the original key name. * Ask syslog-ng to take a message, and rename the keys according to various rules I set - similar how value-pairs()' rekeying works, possibly following the same syntax. For example, I want to take all ".json.*" keys, remove the prefix, and uppercase the names. Then, I want to replace all leading dots with an underscore in whatever keys remain. * Ask syslog-ng to remove keys completely. I don't care about the DATE field, because I receive CEE-enabled messages only, and they come with a high-precision date field anyway, called a "timestamp". * I also want to drop every key where the value matches a certain pattern. Or perhaps not drop them, but anonymize the value.. For example, I might not like the word "plasson", so much so, that whatever key contains it, I never want to see it. I also want to pull a prank, because it just happens to be April 1st, so I want to replace every occurence of "Linux" with "Emacs" within a LogMessage, in every single key. * Since we're applying transformations, might aswell do what rewrite does too, and be able to set stuff - we already do subst. For example, I want to anonymize all the IP addresses. I don't mind the country-codes exposed, but I don't want the IPs in my logs. * I want to be able to compose all of the above, chain them together, so one gets executed after the other, and in the end, the LogMessage will end up with their combined result. For the above, I propose the following syntax: ,---- | map m_do_stuff { | geoip("*_IP", target-prefix("geo.")) | rekey(".json.*", | shift(6) uppercase()) | rekey(".*", replace(".", "_")) | filter-out(key("DATE")) | filter-out(value("plasson", type(substring))) | subst(value("Linux", "Emacs", type(substring))) | set(key("*_IP", "<anonymized>")) | }; `---- Of course, this differs a bit from the syntax used in rewrite, and to be honest, intentionally so. I could never learn to love rewrite's way of set("new-value", value("key-name")). Nevertheless, the syntax can be changed to be similar to rewrite, the functionality would remain the same even then. And how to use this? destination d_something { source(s_something); map(m_do_stuff); ... } And in case we want to tie it to a condition, then: map(m_do_stuff, condition(filter(f_filter_condition))) I don't think I'd want to support specifying maps in-line, but I suppose that could be done aswell. Basically, this would be rewrite on steroids, with the ability to modify keys and values in bulk aswell. The major advantage this would have is the ability to work not only on a single key, but apply transformations to any number of key-value pairs, changing either of them or both. If architectured well, it could even be fast, on par with rewrite if it has to do similar things. I mean, the following two should be equally fast: ,---- | map m_set_host { | set(key("HOST", "myhost")); | }; `---- ,---- | rewrite r_set_host { | set("myhost", value("HOST")); | }; `---- For this to work, and for optimisations to be made possible, the implementation will have to be clever, and able to take shortcuts. I have a few ideas about that too, but that's a topic for a later time: let's see first if the idea is deemed useful, and if the syntax I came up with makes sense to anyone else. What do YOU think? Would you have a use for a way to configure bulk transformations? If so, what other transformations would you find interesting? -- |8]

2 2

Connections not closing on syslog-ng 2.1.4 server
by Robert Nickel 10 May '12

10 May '12

I have a busy syslog-ng server that is collecting a large volume of logs. This problem is causing me grief and I am wondering if it is related to having flow_control enabled on the host. Any help would be greatly appreciated. The connection limit on the inbound tcp connection is slowly being exhausted by multiple connections from each client slowly building until the server stops working. The source is configured as such: source s_remote { tcp(ip(0.0.0.0) port(5140) log_iw_size(204800) max-connections(2048)); }; The bulk of the logs are being sent to other syslog-ng servers running on the same host. e.g.: destination child1 { tcp("127.0.0.1" port (5148)); }; All of the log lines have the same setup: log { source (s_remote); filter(f_q1); destination(d_q1); flags(flow-control,final); }; This setup is working very well with throughput in excess of 35000 messages per second but the whole thing blows up every couple of days due to running out of connections on the source. Before it dies, for a couple of days, I see a number of these errors in the logs: Number of allowed concurrent connections exceeded; num='2048', max='2048' Usually, the actual number of connections is a couple of hundred above the 2048 number listed here. (netstat -an | fgrep -v ESTABLISHED | fgrep :5140). The actual server count hovers around 1300 of which there are ~1000 actively logging. Options in use are: create_dirs(yes); dir_perm (0755); dns_cache_expire(28800); dns_cache (yes); flush_lines(10); flush_timeout(2048); frac_digits(3); keep_hostname(no); log_fetch_limit(100); log_fifo_size (2048); log_iw_size(100); long_hostnames(off); perm(0644); stats_freq(300); time_reopen(10); time_sleep(10); ts_format("iso"); use_dns(yes); use_fqdn(no); use_time_recvd(no); Thank you! --Robert

2 2