- syslog-ng - lists.balabit.hu

TCP packet collapse errors
by Xuri Nagarin 04 Jun '13

04 Jun '13

I have a pair of Syslog-NG servers running 3.2.5-3. The hardware specs are - Quad Xeon E5-2680 (32 cores), 32GB RAM, and two 1TB SAS 7200 RPM disks in RAID-1. OS is RHEL6.2 - Kernel 2.6.32-279.5.2. Filesystem is ext3. Global options are set as: options { flush_lines (1000); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); keep_timestamp(yes); dir_group("syslog"); perm(0640); dir_perm(0750); group("syslog"); }; I have already set TCP kernel buffers to 128MB max and set disk scheduler to "deadline". But even under light disk IO load, from ~8-25MB, I see "1320811067 packets collapsed in receive queue due to low socket buffer". I had some other processes on the host writing to disk. Stopping them reduced the packet errors but this number still keeps incrementing. To rule out other issues, I temporarily pointed my disk-based destinations to /dev/null and then packet losses/errors stopped. So either Syslog-NG isn't able to write to disk fast enough or there is an underlying OS/hardware issue. Both hosts have the same issue. Any pointers in troubleshooting it will be appreciated. TIA.

2 8

syslog-ng 3.3.10 has been released
by devel＠balabit.hu 03 Jun '13

03 Jun '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.10 SUMMARY : new stable release DATE : Jun 3, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.10) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.10 Mon, 3 Jun 2013 16:00:00 +0100 Highlights ========== This release is a bug-fix release, correcting a handful of issues discovered since the previous one. Bugfixes ======== * The persist state file (syslog-ng.persist) is now marked close-on-exec, so it does not leak through to forked subprocesses. * A rare race condition in the SQL and MongoDB destinations have been fixed. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila M. Magyar <athos(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 236] New: syslog-ng loses part of kernel log and replaces it with \0 characters
by bugzilla＠bugzilla.balabit.com 03 Jun '13

03 Jun '13

https://bugzilla.balabit.com/show_bug.cgi?id=236 Summary: syslog-ng loses part of kernel log and replaces it with \0 characters Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: kroz.nn(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 1. syslog-ng with default configuration inserts characters with code 0 into /var/log/messages (2nd line, beginning of kernel log / dmesg). Comparison with output of dmesg has shown, that no log records lost, just characters inserted. 2. syslog-ng with more complicated configuration inserts two batches of \0 characters (2nd line and 12th line). Second batch of characters replaces 118 lines of kernel log (comparing with log of default configuration and dmesg) OS: Gentoo Linux Attachments: - For each of scenario 1 and scenario 2: configuration file, relevant parts of /var/log/messages - Gentoo compilation environment (output of `emerge --info` ) - Beginning of syslog-ng compilation log (configure stage) Syslog-ng version: syslog-ng 3.4.1 Installer-Version: 3.4.1 Revision: ssh+git://algernon@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#master-3.4#dea0b2a11bffe1877ec10d29279fbb0f635d70e7 Compile-Date: Jun 2 2013 12:23:49 Available-Modules: syslogformat,dbparser,afsocket-notls,afuser,afamqp,basicfuncs,cryptofuncs,afsocket-tls,afprog,afsocket,system-source,confgen,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

problems compiling OSE 3.4.1 on redhat enterprise 6.4
by Russell Fulton 01 Jun '13

01 Jun '13

make[4]: Entering directory `/home/rful011/syslog-ng-3.4.1/modules/afsocket' /bin/sh ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c -o libafsocket_notls_la-afinet-dest.lo `test -f 'afinet-dest.c' || echo './'`afinet-dest.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c afinet-dest.c -fPIC -DPIC -o .libs/libafsocket_notls_la-afinet-dest.o In file included from /usr/include/net/ethernet.h:27, from /usr/include/libnet.h:98, from afinet-dest.c:43: when I first encountered this error I went and did some googling and followed some vague leads to install a bunch of -devel packages (most of which were already installed ) but it did not make any difference. Googling the message brings up a bunch of hits for both redhat and debian families and a range of packages. There did not seem to be any common cause. Russell

4 7

PatternDB git moved and updated
by Peter Czanik 31 May '13

31 May '13

Hello, The BalaBit patterndb git moved recently to github. It’s available now at https://github.com/balabit/syslog-ng-patterndb instead of our server. The main advantage of this is the access speed and that patterns are now available next to the syslog-ng sources. Patterns were not only moved, but also updated, reorganized and extended. You can read more about what is changed at https://czanik.blogs.balabit.com/2013/05/patterndb-git-moved-and-updated/ Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[PATCH (3.5)] lib/filter/tests: preload the syslogformat module
by Gergely Nagy 30 May '13

30 May '13

When linking the filter test program, preload the syslogformat module, so that make check will work without installing syslog-ng. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/filter/tests/Makefile.am | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/filter/tests/Makefile.am b/lib/filter/tests/Makefile.am index 44326ec..025dd5d 100644 --- a/lib/filter/tests/Makefile.am +++ b/lib/filter/tests/Makefile.am @@ -5,6 +5,7 @@ check_PROGRAMS += ${lib_filter_tests_TESTS} lib_filter_tests_test_filters_CFLAGS = $(TEST_CFLAGS) \ -I${top_srcdir}/lib/filter/tests -lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) +lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) \ + -dlpreopen ${top_builddir}/modules/syslogformat/libsyslogformat.la lib_filter_tests_test_filters_SOURCES = \ lib/filter/tests/test_filters.c -- 1.7.10.4

2 1

Outbound Facility Rewrite?
by Randy Baca 30 May '13

30 May '13

Hey folks. I have looked through everywhere I can find but cannot figure out how to rewrite the outbound syslog message to a remote host so that all messages come across on the same facility. The reason I need this is to automate sorting and parsing by type of device (all Linux on one facility, all IPS on another facility, Cisco firewalls on another, etc.). Is there a way to do this with syslog-ng? Regards, Randy B

3 3

syslog-ng with mongodb user survey
by Peter Czanik 30 May '13

30 May '13

Hello, Mongodb support is available in syslog-ng for more than a year now. We are aware, that there are many people using it, but now we would like to ask for some feedback about the use cases and environments. Here are just a few sample questions, but any feedback regarding mongodb support is very welcome: - average and peak message rates sent to the mongodb destination - what is your related syslog-ng.conf - how did you tune your mongodb server (if you did) - how and how often do you query your logs in mongodb - etc. Thanks for your help! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 3

[Bug 235] New: String arguments to block parameters get mistreated
by bugzilla＠bugzilla.balabit.com 27 May '13

27 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=235 Summary: String arguments to block parameters get mistreated Product: syslog-ng Version: 3.4.x Platform: All OS/Version: All Status: ASSIGNED Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: regression Estimated Hours: 0.0 I'm not quite sure who and where reported this bug first, but I'm recording it within Bugzilla, so we can keep track of it. With the config below, syslog-ng fails to start: ========================================================== @version: 3.5 block source s_logfile (filename(messages)) { file("/var/log/`filename`" ); }; source s_example { s_logfile(filename("logfile.log")); s_logfile(filename(logfile.log)); s_logfile(filename(foo bar baz)); }; ========================================================== Error parsing affile, syntax error, unexpected LL_IDENTIFIER in source block s_logfile at line 2, column 19: included from etc/block-syntax.conf line 8, column 36 file("/var/log/"logfile.log ^^^^^^^^^^^ The first line of s_example makes it fail badly. This is because the lexer recognises "logfile.log" as a block_string, and stores the quotation marks too, therefore the backtick expansion creates an invalid string. We should not store the "s in this case. I managed to bisect this down to cbd8b195e80f54f9b398456d434e39d9d23c2bdd, but have not managed to figure out how to fix the problem yet, without breaking anything else. syslog-ng 3.3 is not affected, but the last s_logfile() statement doesn't work there. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 223] New: syslog-ng doesn' t start on windows running in a cygwin environment
by bugzilla＠bugzilla.balabit.com 24 May '13

24 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=223 Summary: syslog-ng doesn't start on windows running in a cygwin environment Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Windows Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: tps(a)vr-web.de Type of the Report: bug Estimated Hours: 0.0 After having installed syslog-ng and given dependencies, I've configured with /usr/bin/syslog-ng-config No errors that far. Then started it with "net start syslog-ng": $ net start syslog-ng CYGWIN syslog-ng wird gestartet. CYGWIN syslog-ng konnte nicht gestartet werden. Der Dienst hat keinen Fehler gemeldet. Sie erhalten weitere Hilfe, wenn Sie NET HELPMSG 3534 eingeben. syslog-ng did not start. Trying to start with "cygrunsvc -S syslog-ng": $ cygrunsrv.exe -S syslog-ng cygrunsrv: Error starting a service: QueryServiceStatus: Win32 error 1062: Der Dienst wurde nicht gestartet. Same as above. Again no usefull error message. Next I tried to start syslog-ng directly with the paramaters passed by cygrunsvc: $ /usr/sbin/syslog-ng.exe -F Error opening plugin module; module='afsocket', error='No such process' Error opening plugin module; module='afsocket', error='No such process' Error parsing source, source plugin unix-dgram not found in source confgen system at line 1, column 1: included from /etc/syslog-ng/syslog-ng.conf line 10, column 9 unix-dgram ^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng The service does not start because of errors while loading modules. Given "-V" syslog-ng reports: $ /usr/sbin/syslog-ng.exe -V syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: Compile-Date: Apr 3 2012 14:52:24 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Files installed by cygwin are: $ cygcheck.exe -l syslog-ng /etc/postinstall/syslog-ng.sh /etc/defaults/etc/syslog-ng/syslog-ng.conf /etc/defaults/etc/syslog-ng/scl.conf /etc/defaults/etc/syslog-ng/modules.conf /usr/bin/update-patterndb /usr/bin/loggen.exe /usr/bin/syslog-ng-config /usr/bin/cygsyslog-ng-0.dll /usr/bin/pdbtool.exe /usr/lib/syslog-ng/cygdbparser.dll /usr/lib/syslog-ng/cygcsvparser.dll /usr/lib/syslog-ng/cygafsocket-tls.dll /usr/lib/syslog-ng/cygbasicfuncs.dll /usr/lib/syslog-ng/cygafprog.dll /usr/lib/syslog-ng/cygconvertfuncs.dll /usr/lib/syslog-ng/cygafuser.dll /usr/lib/syslog-ng/cygconfgen.dll /usr/lib/syslog-ng/cygsyslogformat.dll /usr/lib/syslog-ng/cygafsocket-notls.dll /usr/lib/syslog-ng/cygdummy.dll /usr/lib/syslog-ng/cygafsocket.dll /usr/lib/syslog-ng/cygaffile.dll /usr/sbin/syslog-ng.exe /usr/sbin/syslog-ng-ctl.exe /usr/share/syslog-ng/include/scl/syslogconf/plugin.conf /usr/share/syslog-ng/include/scl/syslogconf/convert-syslogconf.awk /usr/share/syslog-ng/include/scl/syslogconf/README /usr/share/syslog-ng/include/scl/system/plugin.conf /usr/share/syslog-ng/include/scl/system/generate-system-source.sh /usr/share/syslog-ng/include/scl/pacct/plugin.conf /usr/share/syslog-ng/xsd/patterndb-1.xsd /usr/share/syslog-ng/xsd/patterndb-2.xsd /usr/share/syslog-ng/xsd/patterndb-3.xsd /usr/share/man/man5/syslog-ng.conf.5.gz /usr/share/man/man8/syslog-ng.8.gz /usr/share/man/man1/pdbtool.1.gz /usr/share/man/man1/syslog-ng-ctl.1.gz /usr/share/man/man1/loggen.1.gz /usr/share/doc/syslog-ng/AUTHORS /usr/share/doc/syslog-ng/README /usr/share/doc/syslog-ng/COPYING /usr/share/doc/syslog-ng/ChangeLog /usr/share/doc/syslog-ng/NEWS /usr/share/doc/Cygwin/syslog-ng.README The generated config-file "/etc/syslog-ng/syslog-ng.conf" holds: $ cat /etc/syslog-ng/syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); }; Looks like a bug analyzing the installed files correctly and creating a not working setup. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3