- syslog-ng - lists.balabit.hu

PatternDB git moved and updated
by Peter Czanik 31 May '13

31 May '13

Hello, The BalaBit patterndb git moved recently to github. It’s available now at https://github.com/balabit/syslog-ng-patterndb instead of our server. The main advantage of this is the access speed and that patterns are now available next to the syslog-ng sources. Patterns were not only moved, but also updated, reorganized and extended. You can read more about what is changed at https://czanik.blogs.balabit.com/2013/05/patterndb-git-moved-and-updated/ Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[PATCH (3.5)] lib/filter/tests: preload the syslogformat module
by Gergely Nagy 30 May '13

30 May '13

When linking the filter test program, preload the syslogformat module, so that make check will work without installing syslog-ng. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/filter/tests/Makefile.am | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/filter/tests/Makefile.am b/lib/filter/tests/Makefile.am index 44326ec..025dd5d 100644 --- a/lib/filter/tests/Makefile.am +++ b/lib/filter/tests/Makefile.am @@ -5,6 +5,7 @@ check_PROGRAMS += ${lib_filter_tests_TESTS} lib_filter_tests_test_filters_CFLAGS = $(TEST_CFLAGS) \ -I${top_srcdir}/lib/filter/tests -lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) +lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) \ + -dlpreopen ${top_builddir}/modules/syslogformat/libsyslogformat.la lib_filter_tests_test_filters_SOURCES = \ lib/filter/tests/test_filters.c -- 1.7.10.4

2 1

Outbound Facility Rewrite?
by Randy Baca 30 May '13

30 May '13

Hey folks. I have looked through everywhere I can find but cannot figure out how to rewrite the outbound syslog message to a remote host so that all messages come across on the same facility. The reason I need this is to automate sorting and parsing by type of device (all Linux on one facility, all IPS on another facility, Cisco firewalls on another, etc.). Is there a way to do this with syslog-ng? Regards, Randy B

3 3

syslog-ng with mongodb user survey
by Peter Czanik 30 May '13

30 May '13

Hello, Mongodb support is available in syslog-ng for more than a year now. We are aware, that there are many people using it, but now we would like to ask for some feedback about the use cases and environments. Here are just a few sample questions, but any feedback regarding mongodb support is very welcome: - average and peak message rates sent to the mongodb destination - what is your related syslog-ng.conf - how did you tune your mongodb server (if you did) - how and how often do you query your logs in mongodb - etc. Thanks for your help! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 3

[Bug 235] New: String arguments to block parameters get mistreated
by bugzilla＠bugzilla.balabit.com 27 May '13

27 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=235 Summary: String arguments to block parameters get mistreated Product: syslog-ng Version: 3.4.x Platform: All OS/Version: All Status: ASSIGNED Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: regression Estimated Hours: 0.0 I'm not quite sure who and where reported this bug first, but I'm recording it within Bugzilla, so we can keep track of it. With the config below, syslog-ng fails to start: ========================================================== @version: 3.5 block source s_logfile (filename(messages)) { file("/var/log/`filename`" ); }; source s_example { s_logfile(filename("logfile.log")); s_logfile(filename(logfile.log)); s_logfile(filename(foo bar baz)); }; ========================================================== Error parsing affile, syntax error, unexpected LL_IDENTIFIER in source block s_logfile at line 2, column 19: included from etc/block-syntax.conf line 8, column 36 file("/var/log/"logfile.log ^^^^^^^^^^^ The first line of s_example makes it fail badly. This is because the lexer recognises "logfile.log" as a block_string, and stores the quotation marks too, therefore the backtick expansion creates an invalid string. We should not store the "s in this case. I managed to bisect this down to cbd8b195e80f54f9b398456d434e39d9d23c2bdd, but have not managed to figure out how to fix the problem yet, without breaking anything else. syslog-ng 3.3 is not affected, but the last s_logfile() statement doesn't work there. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 223] New: syslog-ng doesn' t start on windows running in a cygwin environment
by bugzilla＠bugzilla.balabit.com 24 May '13

24 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=223 Summary: syslog-ng doesn't start on windows running in a cygwin environment Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Windows Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: tps(a)vr-web.de Type of the Report: bug Estimated Hours: 0.0 After having installed syslog-ng and given dependencies, I've configured with /usr/bin/syslog-ng-config No errors that far. Then started it with "net start syslog-ng": $ net start syslog-ng CYGWIN syslog-ng wird gestartet. CYGWIN syslog-ng konnte nicht gestartet werden. Der Dienst hat keinen Fehler gemeldet. Sie erhalten weitere Hilfe, wenn Sie NET HELPMSG 3534 eingeben. syslog-ng did not start. Trying to start with "cygrunsvc -S syslog-ng": $ cygrunsrv.exe -S syslog-ng cygrunsrv: Error starting a service: QueryServiceStatus: Win32 error 1062: Der Dienst wurde nicht gestartet. Same as above. Again no usefull error message. Next I tried to start syslog-ng directly with the paramaters passed by cygrunsvc: $ /usr/sbin/syslog-ng.exe -F Error opening plugin module; module='afsocket', error='No such process' Error opening plugin module; module='afsocket', error='No such process' Error parsing source, source plugin unix-dgram not found in source confgen system at line 1, column 1: included from /etc/syslog-ng/syslog-ng.conf line 10, column 9 unix-dgram ^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng The service does not start because of errors while loading modules. Given "-V" syslog-ng reports: $ /usr/sbin/syslog-ng.exe -V syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: Compile-Date: Apr 3 2012 14:52:24 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Files installed by cygwin are: $ cygcheck.exe -l syslog-ng /etc/postinstall/syslog-ng.sh /etc/defaults/etc/syslog-ng/syslog-ng.conf /etc/defaults/etc/syslog-ng/scl.conf /etc/defaults/etc/syslog-ng/modules.conf /usr/bin/update-patterndb /usr/bin/loggen.exe /usr/bin/syslog-ng-config /usr/bin/cygsyslog-ng-0.dll /usr/bin/pdbtool.exe /usr/lib/syslog-ng/cygdbparser.dll /usr/lib/syslog-ng/cygcsvparser.dll /usr/lib/syslog-ng/cygafsocket-tls.dll /usr/lib/syslog-ng/cygbasicfuncs.dll /usr/lib/syslog-ng/cygafprog.dll /usr/lib/syslog-ng/cygconvertfuncs.dll /usr/lib/syslog-ng/cygafuser.dll /usr/lib/syslog-ng/cygconfgen.dll /usr/lib/syslog-ng/cygsyslogformat.dll /usr/lib/syslog-ng/cygafsocket-notls.dll /usr/lib/syslog-ng/cygdummy.dll /usr/lib/syslog-ng/cygafsocket.dll /usr/lib/syslog-ng/cygaffile.dll /usr/sbin/syslog-ng.exe /usr/sbin/syslog-ng-ctl.exe /usr/share/syslog-ng/include/scl/syslogconf/plugin.conf /usr/share/syslog-ng/include/scl/syslogconf/convert-syslogconf.awk /usr/share/syslog-ng/include/scl/syslogconf/README /usr/share/syslog-ng/include/scl/system/plugin.conf /usr/share/syslog-ng/include/scl/system/generate-system-source.sh /usr/share/syslog-ng/include/scl/pacct/plugin.conf /usr/share/syslog-ng/xsd/patterndb-1.xsd /usr/share/syslog-ng/xsd/patterndb-2.xsd /usr/share/syslog-ng/xsd/patterndb-3.xsd /usr/share/man/man5/syslog-ng.conf.5.gz /usr/share/man/man8/syslog-ng.8.gz /usr/share/man/man1/pdbtool.1.gz /usr/share/man/man1/syslog-ng-ctl.1.gz /usr/share/man/man1/loggen.1.gz /usr/share/doc/syslog-ng/AUTHORS /usr/share/doc/syslog-ng/README /usr/share/doc/syslog-ng/COPYING /usr/share/doc/syslog-ng/ChangeLog /usr/share/doc/syslog-ng/NEWS /usr/share/doc/Cygwin/syslog-ng.README The generated config-file "/etc/syslog-ng/syslog-ng.conf" holds: $ cat /etc/syslog-ng/syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); }; Looks like a bug analyzing the installed files correctly and creating a not working setup. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Filtering binary data fields and catch all
by Xuri Nagarin 22 May '13

22 May '13

Hi, My log source sends data over syslog in CEF (Arcsight format). I have a simple parser written in Syslog-NG that takes advantage of the CEF format (that is all messages are in eight fields separated by a pipe char). -----------xxxxxxxxxxxxxxxxx------------------- parser p_cef { csv-parser(columns("cef.ff","cef.vendor","cef.product","cef.c4","cef.c5","cef.c6","cef.c7","cef.c8") delimiters("|") flags(drop-invalid) ); }; destination d_file { file("/var/log/net/${cef.vendor}/${cef.product}/logfile"); }; log { source(s_tcp); parser(p_cef); destination(d_file); }; -----------xxxxxxxxxxxxxxxxx------------------- Some events coming in contain binary data that get translated into control characters and create thousands of directories with garbage in the names. How do I handle binary data in the message? Should I use the sanitize function in the destination/file definition or is there a better way to do it? Also, how do I create a catch-all destination for all the messages that do not match my parser? Thanks!

2 2

Extremely slow receive on TCP
by Xuri Nagarin 18 May '13

18 May '13

Hi, I have a syslog-ng server that does not seem to receive well over TCP. Instead of an expected ~150 Mbits/s, I get a trickle of few events/sec. The setup is: Server L is an ArcSight Logger that streams out events over TCP or UDP. Server X is a test Syslog-ng VM that is running 3.2.5 (from EPEL). Server P is running Syslog-NG 3.2.5 built from EPEL srpms with TLS enabled. L -> X over UDP Port 514, I get ~100 Mbit/sec L -> X over TCP Port 514, I get ~150 Mbit/sec L-> P over UDP Port 514, I get ~100 Mbit/sec L-> P over TCP Port 514, I get almost nothing. Just a few tens of events make it through over a half hour period. From L, I relay the same data over and over so there isn't an issue with what's being streamed. Config on X and P is identical and very basic: ----------------------------------------------------- options { time_reopen (10); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); keep_timestamp(yes); log_fifo_size (1000); stats_freq(60); flush_timeout(10000); log_fetch_limit(100); flush_lines(100); log_iw_size(100); }; source s_tcp { tcp( port(514) max-connections(100) ); }; source s_udp { udp( port(514)); }; destination d_tcp { file("/var/log/tcp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); }; destination d_udp { file("/var/log/udp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); }; ----------------------------------------------------- L -> X over TCP, L opens one TCP connections and everything comes over it. L -> P over TCP, L keeps opening up connections with P but no data comes over those connections or very few events come over those connections. I am going to investigate if there are any network QoS rules in place between L and P but wondering if there might be something in Syslog-NG that might be broken. I ran syslog-ng in debug mode but saw no errors. I looked at the pcap captures between L and P and saw no errors. Any other clues/ideas for troubleshooting are welcome. TIA!

2 3

Re: [syslog-ng] weird filter problem
by Evan Rempel 17 May '13

17 May '13

Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x My advice my be specific to version 3.4 :-( Evan Rempel 250.271.7691 University Systems, University of Victoria Evan Rempel <erempel(a)uvic.ca> wrote: This definitely works. I'm using it right now. If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: On 11/05/2013, at 2:26 PM, Evan Rempel <erempel(a)uvic.ca> wrote: > Try this filter > > > filter f_unknown { > tags(".classifier.unknown"); > }; > This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start: Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory So far as I can tell all the lib files are present and correct and in the same place as the previous version? I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use. Russell ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 5

[PATCH (3.5)] system-source: Make /proc/kmsg optional
by Gergely Nagy 16 May '13

16 May '13

In certain environments (vserver and OpenVZ come to mind), /proc/kmsg is not readable, not even for root. On these systems, emit a warning, and skip it, instead of aborting the startup. Reported-by: Tamas Pal <folti(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/system-source/system-source.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 3c72859..955ade7 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -167,8 +167,17 @@ system_sysblock_add_linux_kmsg(GString *sysblock) close (fd); } - system_sysblock_add_file(sysblock, kmsg, -1, - "kernel", "kernel", format); + if (access(kmsg, R_OK) == -1) + { + msg_warning("system(): The kernel message buffer is not readable, " + "please check permissions if this is unintentional.", + evt_tag_str("device", kmsg), + evt_tag_errno("error", errno), + NULL); + } + else + system_sysblock_add_file(sysblock, kmsg, -1, + "kernel", "kernel", format); } gboolean -- 1.7.10.4

2 1