- syslog-ng - lists.balabit.hu

syslog-ng uses 100% cpu
by Yakup Kaya 17 Apr '14

17 Apr '14

Hello everyone, My problem is syslog-ng is using 100% CPU. When I trace the process for about 1 minute I get output as follows. As you can see epoll_wait and clock_gettime system calls are causing the problem. My syslog-ng version is 3.3.4. In the older versions I did not have such problems (e.g. 2.0.9). I would like to ask what exactly these system calls are used for and is there a configuration option or parameter to disable these calls or tune them? *top command output;* top - 15:28:46 up 4 days, 16:44, 7 users, load average: 1.33, 1.35, 1.33 Tasks: 245 total, 3 running, 240 sleeping, 2 stopped, 0 zombie Cpu(s): 9.2%us, 16.5%sy, 0.0%ni, 72.4%id, 1.9%wa, 0.1%hi, 0.0%si, 0.0%st Mem: 4029172k total, 3958124k used, 71048k free, 322556k buffers Swap: 4192928k total, 479076k used, 3713852k free, 1613548k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10192 root 20 0 8304 3908 2452 R 99.4 0.1 850:33.93 syslog-ng *strace command output:* strace -f -c -p `pgrep -f '/sbin/syslog-ng'` Process 10192 attached - interrupt to quit % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 54.37 0.073938 0 5740561 epoll_wait 45.54 0.061930 0 5740562 clock_gettime 0.03 0.000035 0 1884 fcntl64 0.01 0.000020 0 1488 370 read 0.01 0.000018 0 408 writev 0.01 0.000015 0 2166 epoll_ctl 0.01 0.000008 0 172 78 accept 0.01 0.000007 0 797 write 0.01 0.000007 0 848 182 setsockopt 0.00 0.000006 0 489 gettimeofday 0.00 0.000006 0 408 _llseek 0.00 0.000000 0 92 close 0.00 0.000000 0 190 alarm 0.00 0.000000 0 15 stat64 ------ ----------- ----------- --------- --------- ---------------- 100.00 0.135990 11490080 630 total -- Yakup KAYA (B.Sc., CCNA) K?demli Sistem Destek Uzman?/Senior System Support Specialist Labris Teknoloji A.S,. Silikon Blok 1 NK 24 ODTÜ-Teknokent / Ankara, TURKEY Tel: +90 312 210 11 13 Fax: +90 312 210 14 92 yakup.kaya(a)labrisnetworks.com yakup.kaya(a)labris.eu www.labrisnetworks.com

2 2

[Bug 261] New: DOCBUG: --rekey
by bugzilla＠bugzilla.balabit.com 15 Apr '14

15 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=261 Summary: DOCBUG: --rekey Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 In Example 2.3. (Using the rekey() option) from §2.10.1. (Specifying data types in value-pairs) the documentation claims: "The rekey() option can be used with the format-json template-function as well, using the following syntax: $(format-json --key .cee.* --rekey --shift 4 --add-prefix events.)" However, when using this syntax, the config parser fails with the following error: "Error parsing config, Error compiling template (Error parsing value-pairs: expected an equal sign in key=value pair)" The correct syntax (which works, as suggested by algeron on IRC) would be the following: $(format-json --rekey .cee.* --add-prefix events. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Fix several memory leaks caused by configuration reloading
by Xufeng Zhang 14 Apr '14

14 Apr '14

Hello all, Patch 1 is try to resolve below valgrind memory leak: ==25354== 26,112 bytes in 32 blocks are definitely lost in loss record 619 of 619 ==25354== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25354== by 0x3A05A478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25354== by 0x3A036416D2: log_writer_flush (logwriter.c:1020) ==25354== by 0x3A03641812: log_writer_deinit (logwriter.c:1138) ==25354== by 0x523AF4A: affile_dw_deinit (logpipe.h:268) ==25354== by 0x523B7EB: affile_dd_deinit (logpipe.h:268) ==25354== by 0x3A0362961E: cfg_tree_stop (logpipe.h:268) ==25354== by 0x3A036420DF: main_loop_reload_config_apply (mainloop.c:498) ==25354== by 0x3A036613FA: iv_signal_event (iv_signal.c:170) ==25354== by 0x3A0365FE48: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==25354== by 0x3A03660511: iv_fd_poll_and_run (iv_fd.c:163) ==25354== by 0x3A03660C93: iv_main (iv_main_posix.c:117) Patch 3 is try to resolve below valgrind memory leak: ==25354== 1,107 (176 direct, 931 indirect) bytes in 1 blocks are definitely lost in loss record 594 of 619 ==25354== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25354== by 0x3A05A478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25354== by 0x3A0362FFEF: g_process_set_argv_space (gprocess.c:502) ==25354== by 0x40164C: main (main.c:196) The memory leak problem which resolved by patch 2 is very clear. The only concern from me is patch 1, would it cause any side effect? Thanks!

1 3

Syslog-ng build issue (Autoconf?)
by Bendler, Ehren 11 Apr '14

11 Apr '14

Our build environment: Autoconf 2.68 Automake 1.11.3 m4 1.4.16 libtool 2.4.2 gcc 4.6.3 (cross compiling to PPC) We get this error when building syslog-ng 3.5.4.1: ./configure: line 12794: syntax error near unexpected token `-Wno-pointer-sign,' ./configure: line 12794: `AX_CFLAGS_GCC_OPTION(-Wno-pointer-sign, CFLAGS_NOWARN_POINTER_SIGN)' When I comment out that line in configure.ac (since we don't use that flag), I get an error from libmongo complaining about not having automake 1.14+ and that it only tries to build it because I changed the AC file. I tried to get around that by disabling mongodb support in the configure flags, but it still tries to build that library. I'd prefer to just get rid of the initial error. At the moment I am thinking it is related to our GNU Autotools versions, but I need confirmation of that before I can go ask the powers that be for updates. Or if that isn't the problem, alternate suggestions are appreciated. -Ehren Bendler

1 1

insider 2014-04: incubator packaged; tailing logs; PCI-DSS; eCSI;
by Czanik Péter 10 Apr '14

10 Apr '14

Dear syslog-ng users, This is the 33th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng incubator packaged and documented ------------------------------------------- Last month we had a nice list of new syslog-ng incubator features. By that time it was not much documented and was packaged only for a few platforms. Luckily both of these changed, as new features are now well documented in blogs and additional platforms are now supported. https://czanik.blogs.balabit.com/2014/03/syslog-ng-incubator-packaged/ Tailing SSB logs remotely ------------------------- Perhaps the most important new feature introduced in syslog-ng Store Box 3F2 is the REST-based API to access stored logs and SSB’s indexing engine. It can be used for “tail -f logfile.txt | grep foo” functionality by creating a simple shell script. https://gyp.blogs.balabit.com/2014/03/stuff-to-do-with-the-rest-api-tailing… Google Summer of Code (GsoC) ---------------------------- Student application deadline is over and we are happy to announce, that there are many promising applications. Right now we don't know yet how many of these we can accept, the number of slots will only be known by the end of the month. Stay tuned! PCI-DSS compliance and log management ------------------------------------- After the new version of PCI DSS was published in November, we decided to revisit our very popular Regulatory Compliance and System Logging white paper. Since PCI DSS is such an important standard, we decided to dedicate a white paper exclusively to this topic. In addition to highlighting the changes in version 3.0, we have changed the format of the paper to include information on how log management can help you meet each requirement – not just number 10 – as well as detailed information on how syslog-ng Premium Edition and the syslog-ng Store Box can help you fulfill PCI-DSS requirements. https://pages.balabit.com/pci-dss-compliance-and-log-management.html Become an eCSI Officer - free of charge --------------------------------------- Unknown hackers, industrial espionage, or malicious codes spread by zombie networks are not the greatest risks today: your own users do most of the real damage. To help you overcome these challenges, we have created an information-packed education series. The eCSI video series consists of separate modules that you can watch one-by-one, covering various aspects of logging, log management, and privileged activity management from the ground up. https://www.balabit.com/lp/ecsi-training-camp SHORT NEWS * LOADays 2014 experiences: https://czanik.blogs.balabit.com/2014/04/loadays-2014/ * Why somebody chose syslog-ng over rival logging solutions: http://sharknet.us/2014/04/04/a-review-of-syslog-ng/ SYSLOG-NG AT CONFERENCES The next confirmed events are: * Infosecurity Europe, 29 April - 1 May, London, United Kingdom: http://www.infosec.co.uk/, Giving a talk titled "Finding method in the madness: the challenges of the automatic classification of log messages". The talk will be given by Balazs Scheidler (Bazsi) who you will also be able to find at the BalaBit booth at the event. * LinuxTag, 8-10 May, Berlin, Germany: http://linuxtag.org, Giving a talk titled "Finding method in the madness: the challenges of the automatic classification of log messages". BalaBit will also be sponsoring the event so you'll be able to find us and get T-shirts and talk with our engineers. The talk will be a revised version of the talk given a week before at Infosecurity Europe. NEW RELEASES * Check git if you are impatient :) Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Stats on destinations with macro's ?
by Scot 09 Apr '14

09 Apr '14

Is there a trick to get stats on destinations with macros ? I get stats on my FIFO, local, net work destinations but not on the destinations with macros. Also I don't see any difference in the stats output using any level level 1,2,3. Syslog-ng restarted after each conf change Sent from my iPad

6 11

[PATCH] patterndb: Fix test_patterndb crash when testing outside-of-rule db lookups.
by Nick Alcock 08 Apr '14

08 Apr '14

First test_patterndb_tags_outside_of_rule() nulls out the messages pointer, then it calls clean_pattern_db(), which unconditionally dereferences it. This will never work. Guard the use of messages in clean_pattern_db() with a nullity check. Signed-off-by: Nick Alcock <nix(a)esperi.org.uk> --- modules/dbparser/tests/test_patterndb.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/modules/dbparser/tests/test_patterndb.c b/modules/dbparser/tests/test_patterndb.c index 7ad7f09..ea7c576 100644 --- a/modules/dbparser/tests/test_patterndb.c +++ b/modules/dbparser/tests/test_patterndb.c @@ -69,8 +69,11 @@ create_pattern_db(gchar *pdb) void clean_pattern_db(void) { - g_ptr_array_foreach(messages, (GFunc) log_msg_unref, NULL); - g_ptr_array_free(messages, TRUE); + if (messages) + { + g_ptr_array_foreach(messages, (GFunc) log_msg_unref, NULL); + g_ptr_array_free(messages, TRUE); + } pattern_db_free(patterndb); patterndb = NULL; -- 1.9.1.171.gc28c581

3 2

[Bug 276] New: bundled rabbitmq-c does not compile on Solaris
by bugzilla＠bugzilla.balabit.com 07 Apr '14

07 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=276 Summary: bundled rabbitmq-c does not compile on Solaris Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.hocke(a)nyu.edu Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=94) --> (https://bugzilla.balabit.com/attachment.cgi?id=94) patch for Makefile.in and amqp_private.h The bundled rabbitmq-c library needs htonll and ntohll that do not exist in Solaris 10. I added a couple defines to modules/afamqp/rabbitmq-c/librabbitmq/amqp_private.h that implement those two functions. I also had to force the linker to add -lsocket and -lnsl for a successful compile. This is probably more of a problem with rabbitmq-c but since it is bundled with syslog-ng I thought I should report it here. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

AUTO: James B Horwath is out of the office (returning 04/14/2014)
by James B Horwath 07 Apr '14

07 Apr '14

I am out of the office until 04/14/2014. I will be out of the office until 04/14 with limited access to e-mail. I will do my best to return any urgent communications. If you have an urgent issue please contact Sunil Gupta at (610)-807-6710. Regards Jim Horwath IT Security Services & Governance, Corporate BTS and IT Security/Risk Tel 610-807-8795 Mobile 610-533-6972 3900 Burgess Place, Bethlehem, Pa. 18017 Jim_Horwath(a)glic.com Note: This is an automated response to your message "syslog-ng Digest, Vol 108, Issue 5" sent on 4/7/2014 6:00:02. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[Bug 277] New: test_basicfuncs crashes, apparently due to $(env ...) of a nonexistent variable
by bugzilla＠bugzilla.balabit.com 07 Apr '14

07 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=277 Summary: test_basicfuncs crashes, apparently due to $(env ...) of a nonexistent variable Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: nix(a)esperi.org.uk Type of the Report: bug Estimated Hours: 0.0 This is with syslog-ng 3.5.4.1, fresh out of git, glib 2.38.2, and gcc 4.8.2. Backtrace: #0 0x00007f4209096d1a in strlen () from /lib/libc.so.6 #1 0x00007f4209631550 in g_string_insert_len (len=-1, val=0x0, pos=-1, string=0x219d380) at gstring.c:440 #2 g_string_insert_len (string=0x219d380, pos=-1, val=0x0, len=-1) at gstring.c:428 #3 0x00007f4209b82563 in tf_env (msg=<optimized out>, argc=1, argv=<optimized out>, result=0x219d380) at modules/basicfuncs/misc-funcs.c:54 #4 0x00007f4209942788 in log_template_append_format_with_context (self=self@entry=0x2196d40, messages=messages@entry=0x7fff01a9f1e8, num_messages=num_messages@entry=1, opts=0x2196828, opts@entry=0x0, tz=tz@entry=0, seq_num=999, seq_num@entry=32578, context_id=0x40c143 "test-context-id", context_id@entry=0x0, result=result@entry=0x219d380) at lib/template/templates.c:1385 #5 0x00007f4209942b9f in log_template_append_format (self=self@entry=0x2196d40, lm=lm@entry=0x219c3c0, opts=opts@entry=0x0, tz=tz@entry=0, seq_num=seq_num@entry=32578, context_id=context_id@entry=0x0, result=0x219d380) at lib/template/templates.c:1412 #6 0x00007f4209942c0b in log_template_format (self=self@entry=0x2196d40, lm=lm@entry=0x219c3c0, opts=opts@entry=0x0, tz=tz@entry=0, seq_num=32578, seq_num@entry=999, context_id=0x0, context_id@entry=0x40c143 "test-context-id", result=result@entry=0x219d380) at lib/template/templates.c:1419 #7 0x0000000000409457 in assert_template_format_with_escaping (template=template@entry=0x40b408 "$(env OHHELLO)", escaping=escaping@entry=0, expected=expected@entry=0x40c55e "") at libtest/template_lib.c:108 #8 0x00000000004094ba in assert_template_format (template=template@entry=0x40b408 "$(env OHHELLO)", expected=expected@entry=0x40c55e "") at libtest/template_lib.c:93 #9 0x0000000000409161 in test_misc_funcs () at modules/basicfuncs/tests/test_basicfuncs.c:101 #10 0x0000000000408c4a in main (argc=<optimized out>, argv=<optimized out>) at modules/basicfuncs/tests/test_basicfuncs.c:115 Somehow, between assert_template_format_with_escaping() and tf_env(), we end up with a 1-entry argc and an argv containing an empty string; this then turns into a request to insert a null pointer at position -1. I'd say glib is right to coredump when faced with that :) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3