- syslog-ng - lists.balabit.hu

Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 29 Apr '14

29 Apr '14

Hi there, I'm trying to process ~800 UDP messages second, which I don't think is much. The current setup worked fine in Ubuntu 10.04 (Lucid) and syslog-ng 2.6 (I believe). I'm running into what I believe is receive buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX Via tcpdump, I can attest all messages are getting to the server. I'm listening to port 514 UDP. From syslog-ng stats, I don't see any "dropped" statistics (always 0), but I know for a fact that when the message load is more than 150, I start seeing message loss. All messages received are sent to files on local disk after matching some filters. I have tested this with loggen and no filters -- more messages are sent than written to file. netstat -su shows increasing number of receive errors when going beyond 150 msgs per second. It also shows the receive queue hover in the 20000's. I have tried setting the so_rcvbuf to a relatively large value as described in the admin guide (2 097 152 bytes). The default net.core.rmem_max on ubuntu server 12.04 is 8388608. I have played with this number, up and down with no luck -- messages are still lost. I have spent hours reading documentation, doing packet captures, playing with values in -ng and the host, etc, but I'm at a loss. Any hints as to what I should look into? And more direct questions, Will so_rcvbuf declared on the src modify the system's rmem_max or do I have to do it on both? Why is it recommended in the admin guide to not modify the rmem_default? Thanks any advice would be helpful.

2 1

[Ask for help] How to resolve these two memory leak problem? thanks!
by Xufeng Zhang 23 Apr '14

23 Apr '14

Hello all, I still met below two memory leaks problem when using remote udp connection: ==11004== 20 bytes in 2 blocks are definitely lost in loss record 18 of 596 ==11004== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==11004== by 0x305927C6B1: strdup (strdup.c:43) ==11004== by 0x305E6529F2: _cfg_lexer_lex (cfg-lex.l:199) ==11004== by 0x305E6284C1: cfg_lexer_lex (cfg-lexer.c:759) ==11004== by 0x305E65BF76: rewrite_expr_parse (rewrite-expr-grammar.c:2959) ==11004== by 0x305E654732: T.99 (cfg-parser.h:83) ==11004== by 0x305E655294: main_parse (cfg-grammar.y:584) ==11004== by 0x305E625EE2: cfg_run_parser (cfg-parser.h:83) ==11004== by 0x305E625FF5: cfg_read_config (cfg.c:384) ==11004== by 0x305E642A6E: main_loop_init (mainloop.c:680) ==11004== by 0x401774: main (main.c:246) _cfg_lexer_lex (cfg-lex.l:199) is: yylval->cptr = strdup(yyextra->string_buffer->str); ==11004== 40,960 bytes in 40 blocks are definitely lost in loss record 596 of 596 ==11004== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==11004== by 0x305DA478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==11004== by 0x305E641778: log_writer_flush (logwriter.c:950) ==11004== by 0x305E64185C: log_writer_work_perform (logwriter.c:130) ==11004== by 0x305E6418C7: log_writer_io_flush_output (logwriter.c:210) ==11004== by 0x305E6605E6: iv_fd_poll_and_run (iv_fd.c:167) ==11004== by 0x305E660D93: iv_main (iv_main_posix.c:117) ==11004== by 0x305E641D86: main_loop_run (mainloop.c:737) ==11004== by 0x4017A0: main (main.c:267) log_writer_flush (logwriter.c:950) is: self->line_buffer->str = g_malloc(self->line_buffer->allocated_len); These were found in syslog-ng-3.4.2, and I don't think syslog-ng upstream has already fixed these problem, Does anybody know how to fix it? Thanks in advance! Xufeng

1 1

Millisecond Resolution Timestamps
by David Hauck 22 Apr '14

22 Apr '14

Hello, I'm using the following global options in order to format messages with millisecond resolution: ts_format(iso); frac_digits(3); Although the initial (syslog-ng starting) message appears to include sub-second resolution subsequent messages do not: 20140415 10:19:23.590 notice syslog(syslog-ng):syslog-ng starting up; version='3.5.4.1' ... 20140415 10:19:33.000 notice authpriv(su):FAILED su for test by root ... 20140415 10:20:11.000 notice user(root):test This includes messages originating from any number of sources (including all processes that log via syslog()), *except* messages originating from the kernel (these always seem to have sub-second resolution). Does anyone have any ideas what might be going on here? Thanks, -David PS: the timestamp formatting above is done via a simple template. Regardless of this (re-)formatting nominal iso messages also exhibit this limitation. For e.g.,: 2014-04-14T15:23:48.000-07:00 host99738728 nasysconfd: exit code 0 for /netacquire/bin/sysconf/osinfo read

3 4

CSV-Parser Conditions
by Justin B 20 Apr '14

20 Apr '14

Need information on this.I am using the CSV-PARSER and I want to put a condition that if a column is empty do not apply the CSV-PARSER but write the line. Primarily i get the application logs and i dont want to apply the CSV-Parser on those log lines where the whole error stack is there. Can someone help on this?? -- Kaladhar

2 5

pdbtool 'patternize'
by David Hauck 18 Apr '14

18 Apr '14

Hello, Does anyone have an explanation for why a "pdbtool patternize" generated pattern db indicates it is version '3'? I'm running the latest version of syslog-ng (3.5.4.1) so I was expecting that this would produce a version '4' pattern db. Easy enough to change in the generated XML, just wondering why the latest generator wouldn't create the latest version. Also, what is the nominal format for the log messages that the 'patternize' command is able to process (i.e., would this be logs that contain the nominally formatted syslog-ng output - e.g., via the default template: template("$ISODATE $HOST $MSGHDR$MSG\n");). I've seen some output that appears to suggest there's some nominal decoding of the input log messages. Thanks, -David

5 13

syslog-ng uses 100% cpu
by Yakup Kaya 17 Apr '14

17 Apr '14

Hello everyone, My problem is syslog-ng is using 100% CPU. When I trace the process for about 1 minute I get output as follows. As you can see epoll_wait and clock_gettime system calls are causing the problem. My syslog-ng version is 3.3.4. In the older versions I did not have such problems (e.g. 2.0.9). I would like to ask what exactly these system calls are used for and is there a configuration option or parameter to disable these calls or tune them? *top command output;* top - 15:28:46 up 4 days, 16:44, 7 users, load average: 1.33, 1.35, 1.33 Tasks: 245 total, 3 running, 240 sleeping, 2 stopped, 0 zombie Cpu(s): 9.2%us, 16.5%sy, 0.0%ni, 72.4%id, 1.9%wa, 0.1%hi, 0.0%si, 0.0%st Mem: 4029172k total, 3958124k used, 71048k free, 322556k buffers Swap: 4192928k total, 479076k used, 3713852k free, 1613548k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10192 root 20 0 8304 3908 2452 R 99.4 0.1 850:33.93 syslog-ng *strace command output:* strace -f -c -p `pgrep -f '/sbin/syslog-ng'` Process 10192 attached - interrupt to quit % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 54.37 0.073938 0 5740561 epoll_wait 45.54 0.061930 0 5740562 clock_gettime 0.03 0.000035 0 1884 fcntl64 0.01 0.000020 0 1488 370 read 0.01 0.000018 0 408 writev 0.01 0.000015 0 2166 epoll_ctl 0.01 0.000008 0 172 78 accept 0.01 0.000007 0 797 write 0.01 0.000007 0 848 182 setsockopt 0.00 0.000006 0 489 gettimeofday 0.00 0.000006 0 408 _llseek 0.00 0.000000 0 92 close 0.00 0.000000 0 190 alarm 0.00 0.000000 0 15 stat64 ------ ----------- ----------- --------- --------- ---------------- 100.00 0.135990 11490080 630 total -- Yakup KAYA (B.Sc., CCNA) K?demli Sistem Destek Uzman?/Senior System Support Specialist Labris Teknoloji A.S,. Silikon Blok 1 NK 24 ODTÜ-Teknokent / Ankara, TURKEY Tel: +90 312 210 11 13 Fax: +90 312 210 14 92 yakup.kaya(a)labrisnetworks.com yakup.kaya(a)labris.eu www.labrisnetworks.com

2 2

[Bug 261] New: DOCBUG: --rekey
by bugzilla＠bugzilla.balabit.com 15 Apr '14

15 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=261 Summary: DOCBUG: --rekey Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 In Example 2.3. (Using the rekey() option) from §2.10.1. (Specifying data types in value-pairs) the documentation claims: "The rekey() option can be used with the format-json template-function as well, using the following syntax: $(format-json --key .cee.* --rekey --shift 4 --add-prefix events.)" However, when using this syntax, the config parser fails with the following error: "Error parsing config, Error compiling template (Error parsing value-pairs: expected an equal sign in key=value pair)" The correct syntax (which works, as suggested by algeron on IRC) would be the following: $(format-json --rekey .cee.* --add-prefix events. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Fix several memory leaks caused by configuration reloading
by Xufeng Zhang 14 Apr '14

14 Apr '14

Hello all, Patch 1 is try to resolve below valgrind memory leak: ==25354== 26,112 bytes in 32 blocks are definitely lost in loss record 619 of 619 ==25354== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25354== by 0x3A05A478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25354== by 0x3A036416D2: log_writer_flush (logwriter.c:1020) ==25354== by 0x3A03641812: log_writer_deinit (logwriter.c:1138) ==25354== by 0x523AF4A: affile_dw_deinit (logpipe.h:268) ==25354== by 0x523B7EB: affile_dd_deinit (logpipe.h:268) ==25354== by 0x3A0362961E: cfg_tree_stop (logpipe.h:268) ==25354== by 0x3A036420DF: main_loop_reload_config_apply (mainloop.c:498) ==25354== by 0x3A036613FA: iv_signal_event (iv_signal.c:170) ==25354== by 0x3A0365FE48: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==25354== by 0x3A03660511: iv_fd_poll_and_run (iv_fd.c:163) ==25354== by 0x3A03660C93: iv_main (iv_main_posix.c:117) Patch 3 is try to resolve below valgrind memory leak: ==25354== 1,107 (176 direct, 931 indirect) bytes in 1 blocks are definitely lost in loss record 594 of 619 ==25354== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25354== by 0x3A05A478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25354== by 0x3A0362FFEF: g_process_set_argv_space (gprocess.c:502) ==25354== by 0x40164C: main (main.c:196) The memory leak problem which resolved by patch 2 is very clear. The only concern from me is patch 1, would it cause any side effect? Thanks!

1 3

Syslog-ng build issue (Autoconf?)
by Bendler, Ehren 11 Apr '14

11 Apr '14

Our build environment: Autoconf 2.68 Automake 1.11.3 m4 1.4.16 libtool 2.4.2 gcc 4.6.3 (cross compiling to PPC) We get this error when building syslog-ng 3.5.4.1: ./configure: line 12794: syntax error near unexpected token `-Wno-pointer-sign,' ./configure: line 12794: `AX_CFLAGS_GCC_OPTION(-Wno-pointer-sign, CFLAGS_NOWARN_POINTER_SIGN)' When I comment out that line in configure.ac (since we don't use that flag), I get an error from libmongo complaining about not having automake 1.14+ and that it only tries to build it because I changed the AC file. I tried to get around that by disabling mongodb support in the configure flags, but it still tries to build that library. I'd prefer to just get rid of the initial error. At the moment I am thinking it is related to our GNU Autotools versions, but I need confirmation of that before I can go ask the powers that be for updates. Or if that isn't the problem, alternate suggestions are appreciated. -Ehren Bendler

1 1

insider 2014-04: incubator packaged; tailing logs; PCI-DSS; eCSI;
by Czanik Péter 10 Apr '14

10 Apr '14

Dear syslog-ng users, This is the 33th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng incubator packaged and documented ------------------------------------------- Last month we had a nice list of new syslog-ng incubator features. By that time it was not much documented and was packaged only for a few platforms. Luckily both of these changed, as new features are now well documented in blogs and additional platforms are now supported. https://czanik.blogs.balabit.com/2014/03/syslog-ng-incubator-packaged/ Tailing SSB logs remotely ------------------------- Perhaps the most important new feature introduced in syslog-ng Store Box 3F2 is the REST-based API to access stored logs and SSB’s indexing engine. It can be used for “tail -f logfile.txt | grep foo” functionality by creating a simple shell script. https://gyp.blogs.balabit.com/2014/03/stuff-to-do-with-the-rest-api-tailing… Google Summer of Code (GsoC) ---------------------------- Student application deadline is over and we are happy to announce, that there are many promising applications. Right now we don't know yet how many of these we can accept, the number of slots will only be known by the end of the month. Stay tuned! PCI-DSS compliance and log management ------------------------------------- After the new version of PCI DSS was published in November, we decided to revisit our very popular Regulatory Compliance and System Logging white paper. Since PCI DSS is such an important standard, we decided to dedicate a white paper exclusively to this topic. In addition to highlighting the changes in version 3.0, we have changed the format of the paper to include information on how log management can help you meet each requirement – not just number 10 – as well as detailed information on how syslog-ng Premium Edition and the syslog-ng Store Box can help you fulfill PCI-DSS requirements. https://pages.balabit.com/pci-dss-compliance-and-log-management.html Become an eCSI Officer - free of charge --------------------------------------- Unknown hackers, industrial espionage, or malicious codes spread by zombie networks are not the greatest risks today: your own users do most of the real damage. To help you overcome these challenges, we have created an information-packed education series. The eCSI video series consists of separate modules that you can watch one-by-one, covering various aspects of logging, log management, and privileged activity management from the ground up. https://www.balabit.com/lp/ecsi-training-camp SHORT NEWS * LOADays 2014 experiences: https://czanik.blogs.balabit.com/2014/04/loadays-2014/ * Why somebody chose syslog-ng over rival logging solutions: http://sharknet.us/2014/04/04/a-review-of-syslog-ng/ SYSLOG-NG AT CONFERENCES The next confirmed events are: * Infosecurity Europe, 29 April - 1 May, London, United Kingdom: http://www.infosec.co.uk/, Giving a talk titled "Finding method in the madness: the challenges of the automatic classification of log messages". The talk will be given by Balazs Scheidler (Bazsi) who you will also be able to find at the BalaBit booth at the event. * LinuxTag, 8-10 May, Berlin, Germany: http://linuxtag.org, Giving a talk titled "Finding method in the madness: the challenges of the automatic classification of log messages". BalaBit will also be sponsoring the event so you'll be able to find us and get T-shirts and talk with our engineers. The talk will be a revised version of the talk given a week before at Infosecurity Europe. NEW RELEASES * Check git if you are impatient :) Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0