May 2012 - syslog-ng - lists.balabit.hu

[Bug 143] New: syslog-ng supervisor dies on SIGHUP
by bugzilla＠bugzilla.balabit.com 31 May '12

31 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=143 Summary: syslog-ng supervisor dies on SIGHUP Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 The process which supervises syslog-ng and restarts it in the event of a crash dies when it receives a SIGHUP. This is a problem as when sending a HUP to syslog-ng to have it reload its config, the HUP will get sent to both the supervisor process and the worker process if doing something like `pkill -HUP syslog-ng`. I attached a gdb to the supervisor and its not a segfault or anything, it just quits. I'm guessing the supervisor doesnt have a signal handler for HUP, so it uses the default action which is to quit. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 175] New: syslog-ng doesnt perform fsync() with fsync option enabled on file destinations
by bugzilla＠bugzilla.balabit.com 31 May '12

31 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=175 Summary: syslog-ng doesnt perform fsync() with fsync option enabled on file destinations Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 While trying to track down an issue on our production system where our box would experience hanging issues every 30 seconds, I discovered this was because syslog-ng wasnt flushing its data, and so when the linux pdflush woke up, it would flush out a ton of data to NFS causing syslog and other applications to hang until the flush completed. Ultimately this was causing messages to be dropped as input buffers were filling up while syslog-ng was hung. This shouldnt have been happening as I have syslog-ng configured with the flush option enabled on the file destination which writes to NFS. I have a workaround in place by forcing pdflush to run more frequently, and have also increased several of the input buffers so they dont fill up so fast. I confirmed this issue by building syslog-ng-3.3.5 and putting a gdb break point on lib/logtransport.c:170, and the break point never triggers. I do have a question about the fsync option as well. Does this option tell syslog-ng to flush() after every write, or after all lines have been written (like from the flush_lines or flush_timeout() options)? If it flushes after every write, would it be possible to get an option to flush only once all the messages in the batch have been written? This is the stripped down configuration I have duplicated the issue with: @version: 3.3 options { keep_hostname(yes); chain_hostnames(no); normalize_hostnames(yes); use_dns(no); use_fqdn(yes); stats_freq(0); mark_freq(0); create_dirs(yes); dir_perm(0755); perm(0644); }; source s_sys { unix-dgram("/dev/log" max-connections(256)); internal(); file("/proc/kmsg"); }; destination d_default { file("/var/log/$FACILITY" flush_lines(1) fsync(yes)); }; log { source(s_sys); destination(d_default); flags(fallback); }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

Re: [syslog-ng] syslog_ng 3.3.4 make error on HPUX HPPA 11.11
by Lennert Buytenhek 31 May '12

31 May '12

FWIW, I'm not sure what's going on here. On Tue, May 29, 2012 at 02:34:06PM +0530, Rakesh Rajasekharan wrote: > I have attached my libpthread.a file (under /usr/lib). > I though , have the "libpthread.a" file under all the below locations, but, > I am no tsure which one exactly gets picked up or needs to be picked up > while compiling. > > # find / -name 'libpthread.a' > /usr/lib/pa20_64/libp/libpthread.a > /usr/lib/pa20_64/libpthread.a > /usr/lib/libp/libpthread.a > /usr/lib/libpthread.a > /var/adm/sw/save/PHCO_26466/C-MIN-32ALIB/usr/lib/libpthread.a > /var/adm/sw/save/PHCO_26466/C-MIN-64ALIB/usr/lib/pa20_64/libpthread.a > > On Tue, May 29, 2012 at 2:10 PM, Lennert Buytenhek > <buytenh(a)wantstofly.org>wrote: > > > On Tue, May 29, 2012 at 12:18:10PM +0530, Rakesh Rajasekharan wrote: > > > > > /bin/sh ../../libtool --tag=CC --mode=link gcc -Wall -g -O2 -o client > > > client.o ../libivykis.la -lpthread > > > libtool: link: gcc -Wall -g -O2 -o client client.o ../.libs/libivykis.a > > > -lpthread > > > /usr/ccs/bin/ld: Unsatisfied symbols: > > > pthread_sigmask (first referenced in ../.libs/libivykis.a(iv_main.o)) > > > (data) > > > pthread_mutex_init (first referenced in > > ../.libs/libivykis.a(iv_main.o)) > > > (data) > > > pthread_mutex_destroy (first referenced in > > > ../.libs/libivykis.a(iv_main.o)) (data) > > > pthread_mutex_unlock (first referenced in > > > ../.libs/libivykis.a(iv_main.o)) (data) > > > pthread_mutex_lock (first referenced in > > ../.libs/libivykis.a(iv_main.o)) > > > (data) > > > > Can you send me a copy of your actual libpthread.{a,so} ? > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

3 3

Cancle
by Weerawat Chawakhiew 30 May '12

30 May '12

1 0

[PATCH 0/3] WIP: Convert syslog-ng to use upstream ivykis
by Gergely Nagy 30 May '12

30 May '12

A long long time ago, upstream and BalaBit's version of ivykis started to diverge, which - at the time - was necessary, but the intent always was to eventually get things upstream and merged, to allow syslog-ng to use upstream ivykis, as-is. With the help of Lennert Buytenhek (ivykis upstream) and Balazs Scheidler, I present you the first work in progress patch set against syslog-ng that enables it to work with an unmodified ivykis! The patch set is not 100% complete yet however, I'm only submitting it for review and possible testing. I will probably port it to 3.3 too (it applies almost cleanly, there's only one trivial conflict in lib/Makefile.am) and build the Debian packages using upstream ivykis to get wider testing. What is missing, is the old lib/ivykis/configure.gnu script, which we had there to force a static build of ivykis. This will be replaced by a modification to autogen.sh, that will automatically create the configure.gnu scripts in the various submodules, so we can point the submodule links to upstream repos instead of our own forks. This is not included in this set yet, and one will need to copy that file over from the BalaBit fork, or things won't work too well. Another thing is that the first patch stops libsyslog-ng-crypto from being directly linked to ivykis. This is not an issue when ivykis is statically compiled into libsyslog-ng, since all those symbols will be exported to other modules, including libsyslog-ng-crypto. However, it might become a problem when using a dynamically linked ivykis. I haven't tested that case yet. The reason I remove the link is that if ivykis is statically compiled in to both shared objects, the constructors will run twice, as dlopen()ing anything linked against libsyslog-ng-crypto will trigger them again. This leads to an abort() inside ivykis. This can be done on the ivykis side too: separate the constructors from the init functions, and make the constructors no-op if they ran already: this allows the init functions to continue aborting if called more than once, but allows the constructors to run twice. I have to discuss this with upstream, and see if we need this at all, or if my workaround of removing the linkage from libsyslog-ng-crypto is enough and without side effects. These were the easy issues, which I could solve right now if I had some coffee left. But alas, there is another issue! With the first two patches applied, we'd have a syslog-ng that compiles fine against upstream ivykis, and even works. Except it doesn't: whenever we try to set an io handler for an fd, we'd end up in an abort, because for some reason, we already had a handler set. I do not understand it quite yet why this happens - the old BalaBit fork of ivykis was similarly picky and abort-happy and they did not trigger. As a workaround, I created the third patch, because I wanted to have something that shows signs of working. The third patch is *WRONG*, but it's an acceptable workaround until I have time to debug the problem further. What it does, is that it *always* clears the io handlers before registering new ones. This results in marginally more work being done each time we need to set a handler, but not by much, and as far as I saw, it has no side effects. But since the part of ivykis that aborts before the third patch, did not change between BalaBit's fork and upstream, I'm certain that the second patch introduces a bug that wasn't there before. Nevertheless! The good news still is, that we're *very* close to being able to build with upstream ivykis! A few issues to fix, and then comes the next part: going over all the changes that were made to the BalaBit fork of ivykis, and send everything upstream that has not been sent yet. That will be another funny adventure, but at least, by that time, I will understand what the hell I am doing =) -- |8]

3 16

Howto configure grep to parse multiple mail recipients
by C. L. Martinez 30 May '12

30 May '12

Hi all, I am trying to parse some Ironport logs to cacth mutliple destination recipients using grep in value option: recipient: $(grep ("${to_address}" != "") ${to_address}) but it doesn't works ... What am I doing wrong?? I am using syslog-ng 3.3.5 under CentOS6. Thanks.

2 4

Completing an empty field with specific value
by C. L. Martinez 30 May '12

30 May '12

Hi all, I am trying to complete an empty field with a specific value when I need to correlate postfix logs. Sometimes From filed comes empty: May 29 08:18:04 testsrv postfix/smtp[1569]: E2B653F2: from=<> ... I have tried to do this using "value" options in my patterndb.xml: a) <value name="nosender">$(if ("${from_address}" == "") "no-sender(a)no-domain.com" "${from_address}@7")</value> b) <value name="nosender">$(grep "${from_address}" == "") "no-sender(a)no-domain.com"</value> ... but it doesn't works ... Any help??

1 1

file("/proc/kmsg" program_override("kernel: "));
by Tom 29 May '12

29 May '12

Dear all, I just started using syslog-ng to pipe log files to Loggly.com. I have a couple of questions. First of all, why is there a `: ` after kernel here? file("/proc/kmsg" program_override("kernel: ")); I have seen people do it differently, like so: file("/proc/kmsg" program_override("kernel)); Does this matter? And more general, what does the entire rule do exactly? The following is my syslog-ng.conf file: https://gist.github.com/2819768 As you can see, I have a lot of log files. Is this the proper way of doing this? Doesn't it cost a lot of resources for syslog-ng to check all these files for changes? Finally, where can I find the documentation? Eg. I would like to learn about the specifics of follow_freq and what the parameter actually implies. Thanks a lot for your help. Kind regards, Tom

3 2

Writing JSON message to a MongoDB destination
by Rajalakshmi Iyer 29 May '12

29 May '12

Hello, I have a log message in JSON format that needs to be written to the MongoDB database, such that I should be able to later query the database based on fields in the JSON record. However, when I try to use the syslog-ng mongodb destination driver using : destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") collection("messages") value-pairs(key("MESSAGE")) ); }; The record appears in the mongodb database as: {"MESSAGE":<My JSON formatted message>} I do not want the MESSAGE macro to be part of the record in the database, I only want to it's JSON-encoded value to be entered in the database. Is this possible? Thanks in advance for your help. Regards, Raj -- Twitter: @Blismedia <http://twitter.com/#%21/blismobile> BlisMedia 32 Percy Street, London W1T 2DE www.blismedia.com <http://www.blismobile.com/> <http://www.blismobile.com/> [image: Follow on Twitter] <http://twitter.com/#%21/blismobile>[image: Blis Website] <http://www.blismobile.com/> This communication is from BlisMedia, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify info(a)blismedia.com<info(a)blismobile.com> by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it.

2 1

(no subject)
by pedro castro 28 May '12

28 May '12

http://cabinetbianchi.re/tenyun.php?top=288 Learn h0w t0 turn successful fr0m h0me ___ Indestructibility.Not capable of being injured or destroyed. damita aekerman

1 0