May 2013 - syslog-ng - lists.balabit.hu

Upcoming changes regarding my Debian/Ubuntu packages
by Gergely Nagy 04 Jul '13

04 Jul '13

Hi! For a good while now, I've been providing Debian and Ubuntu packages[1] for syslog-ng, and rest assured, I will continue to do so. However, the way they are prepared and how the repository is structured, will change. At the moment, there are two different components one can choose from: syslog-ng and syslog-ng-devel, which have packages for 3.3 and 3.4, respectively. However, -devel is not appropriate for 3.4 anymore, but I don't want to move it to syslog-ng either, partly because up to this point, there were no Ubuntu Lucid packages of 3.4. So, I will be making the following changes: * I'll create syslog-ng-3.3, syslog-ng-3.4 and syslog-ng-3.5 components * syslog-ng will point to the latest stable branch (3.4) * syslog-ng will point to the current development branch (3.5) * I'll be preparing Ubuntu Lucid packages for 3.4 too, so all branches will be available for every debian or ubuntu system that are still supported by their respective vendors. To be able to provide Ubuntu Lucid (10.04) packages sanely, I have to do the packaging a little bit differently: the source package will not be shared between debian and ubuntu anymore, and systemd support will be explicitly disabled on Ubuntu. But since Ubuntu uses upstart by default, ubuntu systems with systemd are fairly unlikely anyway. I plan to make these changes once 3.4.2 comes out. Until then, I'll introduce the syslog-ng-3.3 and -3.4 components, but syslog-ng will point to -3.3 still, and -devel to -3.4. Once 3.4.2 is out, though, I'll change the symlinks. [1]: http://asylum.madhouse-project.org/projects/debian/ -- |8]

1 2

[libsyslog_ng_la-afinter.lo] Error 1, syslog-ng-3.4.1 on x86_64
by Flur Blubr 15 Jun '13

15 Jun '13

Dear all, looking for help, for 4 days trying to compile syslog-ng-3.4.1 on some kind of custom RED HAT derivate and still failing. RPMs and YUM are forbidden and system has libraries spread around all directories / using exports export EVTLOG_CFLAGS, export EVTLOG_LIBS, export GLIB_CFLAGS, export GLIB_LIBS to overwrite PKG CONFIG . uname -a Linux 2.6.32-279.1.1.el6.x86_64 #1 SMP Wed Jun 20 11:41:22 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.3 (Santiago) Configuration: ./configure --prefix=/my-prefix/syslog-ng --enable-sql --enable-ipv6 --enable-dynamic-linking --build=x86_64-pc-linux-gnu --host=i686-pc-linux-gnu --enable-shared --disable-static CFLAGS=-O2 Getting this: syslog-ng Open Source Edition 3.4.1 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -O2 -Wall -pthread -I/my-prefix/sources/glib-2.15.4/ -I/my-prefix/eventlog/include/eventlog/ -I$(top_srcdir)/lib/ivykis/src/include -I$(top_builddir)/lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : prefix : /my-prefix/syslog-ng linking mode : dynamic embedded crypto : no __thread keyword : yes Submodules: ivykis : internal libmongo-client : internal librabbitmq : internal Features: Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : no tcp-wrapper support : no Linux capability support : yes PCRE support : yes Env wrapper support : no systemd support : no (unit dir: none) Modules: Module search path : /my-prefix/syslog-ng/lib/syslog-ng Sun STREAMS support (module): no SSL support (module) : yes SQL support (module) : yes PACCT module (EXPERIMENTAL) : no MongoDB destination (module): yes JSON support (module) : no SMTP support (module) : no AMQP destination (module) : yes GEOIP support (module) : no But make is failing no matter what I do: afinter.c: At top level: afinter.c:385: error: expected ')' before 'mark_freq' afinter.c: In function 'afinter_message_posted': afinter.c:404: error: 'internal_msg_lock' undeclared (first use in this function) afinter.c:405: error: 'internal_msg_queue' undeclared (first use in this function) afinter.c:407: warning: implicit declaration of function 'g_queue_new' afinter.c:409: warning: implicit declaration of function 'g_queue_push_tail' afinter.c:411: error: 'AFInterSource' has no member named 'post' afinter.c: At top level: afinter.c:416: error: expected ')' before 'hook_type' afinter.c: In function 'afinter_global_init': afinter.c:424: warning: implicit declaration of function 'register_application_hook' afinter.c:424: error: 'afinter_register_posted_hook' undeclared (first use in this function) make[4]: *** [libsyslog_ng_la-afinter.lo] Error 1 make[4]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[2]: *** [all] Error 2 make[2]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/my-prefix/syslog-ng-3.4.1' make: *** [all] Error 2 Can somebody advice? Thx for your time. Regards Michal

4 9

[Bug 230] New: Passing args to blocks is broken in syslog-ng v.3.4
by bugzilla＠bugzilla.balabit.com 13 Jun '13

13 Jun '13

https://bugzilla.balabit.com/show_bug.cgi?id=230 Summary: Passing args to blocks is broken in syslog-ng v.3.4 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: koldaevav(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 After an upgrade from 3.3.7 to 3.4.1: ... block root mylogs(log()) { }; mylogs(log(123)) ... Gives: Unknown argument; context='root block mylogs', arg='log', value='123' Error parsing config, syntax error, unexpected LL_ERROR, expecting $end in /etc/syslog-ng/syslog-ng.conf at line 11, column 1: mylogs(log(123)) ^^^^^^ -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Reaping unused destination files
by Jordi Prats 06 Jun '13

06 Jun '13

Hi, After updating to syslog-ng 3.2, I'm no longer getting any logs. It reports some destinations as unused: May 23 17:09:16 borg syslog-ng[27957]: Reaping unused destination files; template='/var/log/firewalls/$YEAR/$MONTH/$DAY/$HOST.log' May 23 17:09:16 borg syslog-ng[27957]: Reaping unused destination files; template='/var/log/troncal/$YEAR/$MONTH/$DAY/$SOURCEIP.log' May 23 17:09:16 borg syslog-ng[27957]: Reaping unused destination files; template='/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/$PROGRAM/$FACILITY' But I have no clue why, as this configuration was working on another system (I'm pretty sure it was an older version, but I can't tell which exactly) My config is as follows: options { time_reopen (10); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); sync (5000); time_reap(5); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); }; source remote_tcp { tcp(port(514) keep-alive(no)); }; source remote_default { udp(ip(192.168.1.1) port(514)); }; source remote_servers { udp(ip(192.168.1.2) port(514)); }; source firewalls { udp(port(1515)); }; source troncal { udp(ip(192.168.1.3) port(514)); }; destination hosts { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/$PROGRAM/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; destination firewalls { file("/var/log/firewalls/$YEAR/$MONTH/$DAY/$HOST.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; destination troncal { file("/var/log/troncal/$YEAR/$MONTH/$DAY/$SOURCEIP.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; log { source(remote_default); source(remote_servers); destination(hosts); }; log { source(firewalls); destination(firewalls); }; log { source(troncal); destination(troncal); }; Anyone can help me out? Thank you! -- Jordi

2 1

TCP packet collapse errors
by Xuri Nagarin 04 Jun '13

04 Jun '13

I have a pair of Syslog-NG servers running 3.2.5-3. The hardware specs are - Quad Xeon E5-2680 (32 cores), 32GB RAM, and two 1TB SAS 7200 RPM disks in RAID-1. OS is RHEL6.2 - Kernel 2.6.32-279.5.2. Filesystem is ext3. Global options are set as: options { flush_lines (1000); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); keep_timestamp(yes); dir_group("syslog"); perm(0640); dir_perm(0750); group("syslog"); }; I have already set TCP kernel buffers to 128MB max and set disk scheduler to "deadline". But even under light disk IO load, from ~8-25MB, I see "1320811067 packets collapsed in receive queue due to low socket buffer". I had some other processes on the host writing to disk. Stopping them reduced the packet errors but this number still keeps incrementing. To rule out other issues, I temporarily pointed my disk-based destinations to /dev/null and then packet losses/errors stopped. So either Syslog-NG isn't able to write to disk fast enough or there is an underlying OS/hardware issue. Both hosts have the same issue. Any pointers in troubleshooting it will be appreciated. TIA.

2 8

problems compiling OSE 3.4.1 on redhat enterprise 6.4
by Russell Fulton 01 Jun '13

01 Jun '13

make[4]: Entering directory `/home/rful011/syslog-ng-3.4.1/modules/afsocket' /bin/sh ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c -o libafsocket_notls_la-afinet-dest.lo `test -f 'afinet-dest.c' || echo './'`afinet-dest.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c afinet-dest.c -fPIC -DPIC -o .libs/libafsocket_notls_la-afinet-dest.o In file included from /usr/include/net/ethernet.h:27, from /usr/include/libnet.h:98, from afinet-dest.c:43: when I first encountered this error I went and did some googling and followed some vague leads to install a bunch of -devel packages (most of which were already installed ) but it did not make any difference. Googling the message brings up a bunch of hits for both redhat and debian families and a range of packages. There did not seem to be any common cause. Russell

4 7

PatternDB git moved and updated
by Peter Czanik 31 May '13

31 May '13

Hello, The BalaBit patterndb git moved recently to github. It’s available now at https://github.com/balabit/syslog-ng-patterndb instead of our server. The main advantage of this is the access speed and that patterns are now available next to the syslog-ng sources. Patterns were not only moved, but also updated, reorganized and extended. You can read more about what is changed at https://czanik.blogs.balabit.com/2013/05/patterndb-git-moved-and-updated/ Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[PATCH (3.5)] lib/filter/tests: preload the syslogformat module
by Gergely Nagy 30 May '13

30 May '13

When linking the filter test program, preload the syslogformat module, so that make check will work without installing syslog-ng. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/filter/tests/Makefile.am | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/filter/tests/Makefile.am b/lib/filter/tests/Makefile.am index 44326ec..025dd5d 100644 --- a/lib/filter/tests/Makefile.am +++ b/lib/filter/tests/Makefile.am @@ -5,6 +5,7 @@ check_PROGRAMS += ${lib_filter_tests_TESTS} lib_filter_tests_test_filters_CFLAGS = $(TEST_CFLAGS) \ -I${top_srcdir}/lib/filter/tests -lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) +lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) \ + -dlpreopen ${top_builddir}/modules/syslogformat/libsyslogformat.la lib_filter_tests_test_filters_SOURCES = \ lib/filter/tests/test_filters.c -- 1.7.10.4

2 1

Outbound Facility Rewrite?
by Randy Baca 30 May '13

30 May '13

Hey folks. I have looked through everywhere I can find but cannot figure out how to rewrite the outbound syslog message to a remote host so that all messages come across on the same facility. The reason I need this is to automate sorting and parsing by type of device (all Linux on one facility, all IPS on another facility, Cisco firewalls on another, etc.). Is there a way to do this with syslog-ng? Regards, Randy B

3 3

syslog-ng with mongodb user survey
by Peter Czanik 30 May '13

30 May '13

Hello, Mongodb support is available in syslog-ng for more than a year now. We are aware, that there are many people using it, but now we would like to ask for some feedback about the use cases and environments. Here are just a few sample questions, but any feedback regarding mongodb support is very welcome: - average and peak message rates sent to the mongodb destination - what is your related syslog-ng.conf - how did you tune your mongodb server (if you did) - how and how often do you query your logs in mongodb - etc. Thanks for your help! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 3