- syslog-ng - lists.balabit.hu

PatternDB git moved and updated
by Peter Czanik 31 May '13

31 May '13

Hello, The BalaBit patterndb git moved recently to github. It’s available now at https://github.com/balabit/syslog-ng-patterndb instead of our server. The main advantage of this is the access speed and that patterns are now available next to the syslog-ng sources. Patterns were not only moved, but also updated, reorganized and extended. You can read more about what is changed at https://czanik.blogs.balabit.com/2013/05/patterndb-git-moved-and-updated/ Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[PATCH (3.5)] lib/filter/tests: preload the syslogformat module
by Gergely Nagy 30 May '13

30 May '13

When linking the filter test program, preload the syslogformat module, so that make check will work without installing syslog-ng. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/filter/tests/Makefile.am | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/filter/tests/Makefile.am b/lib/filter/tests/Makefile.am index 44326ec..025dd5d 100644 --- a/lib/filter/tests/Makefile.am +++ b/lib/filter/tests/Makefile.am @@ -5,6 +5,7 @@ check_PROGRAMS += ${lib_filter_tests_TESTS} lib_filter_tests_test_filters_CFLAGS = $(TEST_CFLAGS) \ -I${top_srcdir}/lib/filter/tests -lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) +lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) \ + -dlpreopen ${top_builddir}/modules/syslogformat/libsyslogformat.la lib_filter_tests_test_filters_SOURCES = \ lib/filter/tests/test_filters.c -- 1.7.10.4

2 1

Outbound Facility Rewrite?
by Randy Baca 30 May '13

30 May '13

Hey folks. I have looked through everywhere I can find but cannot figure out how to rewrite the outbound syslog message to a remote host so that all messages come across on the same facility. The reason I need this is to automate sorting and parsing by type of device (all Linux on one facility, all IPS on another facility, Cisco firewalls on another, etc.). Is there a way to do this with syslog-ng? Regards, Randy B

3 3

syslog-ng with mongodb user survey
by Peter Czanik 30 May '13

30 May '13

Hello, Mongodb support is available in syslog-ng for more than a year now. We are aware, that there are many people using it, but now we would like to ask for some feedback about the use cases and environments. Here are just a few sample questions, but any feedback regarding mongodb support is very welcome: - average and peak message rates sent to the mongodb destination - what is your related syslog-ng.conf - how did you tune your mongodb server (if you did) - how and how often do you query your logs in mongodb - etc. Thanks for your help! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 3

[Bug 235] New: String arguments to block parameters get mistreated
by bugzilla＠bugzilla.balabit.com 27 May '13

27 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=235 Summary: String arguments to block parameters get mistreated Product: syslog-ng Version: 3.4.x Platform: All OS/Version: All Status: ASSIGNED Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: regression Estimated Hours: 0.0 I'm not quite sure who and where reported this bug first, but I'm recording it within Bugzilla, so we can keep track of it. With the config below, syslog-ng fails to start: ========================================================== @version: 3.5 block source s_logfile (filename(messages)) { file("/var/log/`filename`" ); }; source s_example { s_logfile(filename("logfile.log")); s_logfile(filename(logfile.log)); s_logfile(filename(foo bar baz)); }; ========================================================== Error parsing affile, syntax error, unexpected LL_IDENTIFIER in source block s_logfile at line 2, column 19: included from etc/block-syntax.conf line 8, column 36 file("/var/log/"logfile.log ^^^^^^^^^^^ The first line of s_example makes it fail badly. This is because the lexer recognises "logfile.log" as a block_string, and stores the quotation marks too, therefore the backtick expansion creates an invalid string. We should not store the "s in this case. I managed to bisect this down to cbd8b195e80f54f9b398456d434e39d9d23c2bdd, but have not managed to figure out how to fix the problem yet, without breaking anything else. syslog-ng 3.3 is not affected, but the last s_logfile() statement doesn't work there. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 223] New: syslog-ng doesn' t start on windows running in a cygwin environment
by bugzilla＠bugzilla.balabit.com 24 May '13

24 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=223 Summary: syslog-ng doesn't start on windows running in a cygwin environment Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Windows Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: tps(a)vr-web.de Type of the Report: bug Estimated Hours: 0.0 After having installed syslog-ng and given dependencies, I've configured with /usr/bin/syslog-ng-config No errors that far. Then started it with "net start syslog-ng": $ net start syslog-ng CYGWIN syslog-ng wird gestartet. CYGWIN syslog-ng konnte nicht gestartet werden. Der Dienst hat keinen Fehler gemeldet. Sie erhalten weitere Hilfe, wenn Sie NET HELPMSG 3534 eingeben. syslog-ng did not start. Trying to start with "cygrunsvc -S syslog-ng": $ cygrunsrv.exe -S syslog-ng cygrunsrv: Error starting a service: QueryServiceStatus: Win32 error 1062: Der Dienst wurde nicht gestartet. Same as above. Again no usefull error message. Next I tried to start syslog-ng directly with the paramaters passed by cygrunsvc: $ /usr/sbin/syslog-ng.exe -F Error opening plugin module; module='afsocket', error='No such process' Error opening plugin module; module='afsocket', error='No such process' Error parsing source, source plugin unix-dgram not found in source confgen system at line 1, column 1: included from /etc/syslog-ng/syslog-ng.conf line 10, column 9 unix-dgram ^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng The service does not start because of errors while loading modules. Given "-V" syslog-ng reports: $ /usr/sbin/syslog-ng.exe -V syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: Compile-Date: Apr 3 2012 14:52:24 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Files installed by cygwin are: $ cygcheck.exe -l syslog-ng /etc/postinstall/syslog-ng.sh /etc/defaults/etc/syslog-ng/syslog-ng.conf /etc/defaults/etc/syslog-ng/scl.conf /etc/defaults/etc/syslog-ng/modules.conf /usr/bin/update-patterndb /usr/bin/loggen.exe /usr/bin/syslog-ng-config /usr/bin/cygsyslog-ng-0.dll /usr/bin/pdbtool.exe /usr/lib/syslog-ng/cygdbparser.dll /usr/lib/syslog-ng/cygcsvparser.dll /usr/lib/syslog-ng/cygafsocket-tls.dll /usr/lib/syslog-ng/cygbasicfuncs.dll /usr/lib/syslog-ng/cygafprog.dll /usr/lib/syslog-ng/cygconvertfuncs.dll /usr/lib/syslog-ng/cygafuser.dll /usr/lib/syslog-ng/cygconfgen.dll /usr/lib/syslog-ng/cygsyslogformat.dll /usr/lib/syslog-ng/cygafsocket-notls.dll /usr/lib/syslog-ng/cygdummy.dll /usr/lib/syslog-ng/cygafsocket.dll /usr/lib/syslog-ng/cygaffile.dll /usr/sbin/syslog-ng.exe /usr/sbin/syslog-ng-ctl.exe /usr/share/syslog-ng/include/scl/syslogconf/plugin.conf /usr/share/syslog-ng/include/scl/syslogconf/convert-syslogconf.awk /usr/share/syslog-ng/include/scl/syslogconf/README /usr/share/syslog-ng/include/scl/system/plugin.conf /usr/share/syslog-ng/include/scl/system/generate-system-source.sh /usr/share/syslog-ng/include/scl/pacct/plugin.conf /usr/share/syslog-ng/xsd/patterndb-1.xsd /usr/share/syslog-ng/xsd/patterndb-2.xsd /usr/share/syslog-ng/xsd/patterndb-3.xsd /usr/share/man/man5/syslog-ng.conf.5.gz /usr/share/man/man8/syslog-ng.8.gz /usr/share/man/man1/pdbtool.1.gz /usr/share/man/man1/syslog-ng-ctl.1.gz /usr/share/man/man1/loggen.1.gz /usr/share/doc/syslog-ng/AUTHORS /usr/share/doc/syslog-ng/README /usr/share/doc/syslog-ng/COPYING /usr/share/doc/syslog-ng/ChangeLog /usr/share/doc/syslog-ng/NEWS /usr/share/doc/Cygwin/syslog-ng.README The generated config-file "/etc/syslog-ng/syslog-ng.conf" holds: $ cat /etc/syslog-ng/syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); }; Looks like a bug analyzing the installed files correctly and creating a not working setup. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Filtering binary data fields and catch all
by Xuri Nagarin 22 May '13

22 May '13

Hi, My log source sends data over syslog in CEF (Arcsight format). I have a simple parser written in Syslog-NG that takes advantage of the CEF format (that is all messages are in eight fields separated by a pipe char). -----------xxxxxxxxxxxxxxxxx------------------- parser p_cef { csv-parser(columns("cef.ff","cef.vendor","cef.product","cef.c4","cef.c5","cef.c6","cef.c7","cef.c8") delimiters("|") flags(drop-invalid) ); }; destination d_file { file("/var/log/net/${cef.vendor}/${cef.product}/logfile"); }; log { source(s_tcp); parser(p_cef); destination(d_file); }; -----------xxxxxxxxxxxxxxxxx------------------- Some events coming in contain binary data that get translated into control characters and create thousands of directories with garbage in the names. How do I handle binary data in the message? Should I use the sanitize function in the destination/file definition or is there a better way to do it? Also, how do I create a catch-all destination for all the messages that do not match my parser? Thanks!

2 2

Extremely slow receive on TCP
by Xuri Nagarin 18 May '13

18 May '13

Hi, I have a syslog-ng server that does not seem to receive well over TCP. Instead of an expected ~150 Mbits/s, I get a trickle of few events/sec. The setup is: Server L is an ArcSight Logger that streams out events over TCP or UDP. Server X is a test Syslog-ng VM that is running 3.2.5 (from EPEL). Server P is running Syslog-NG 3.2.5 built from EPEL srpms with TLS enabled. L -> X over UDP Port 514, I get ~100 Mbit/sec L -> X over TCP Port 514, I get ~150 Mbit/sec L-> P over UDP Port 514, I get ~100 Mbit/sec L-> P over TCP Port 514, I get almost nothing. Just a few tens of events make it through over a half hour period. From L, I relay the same data over and over so there isn't an issue with what's being streamed. Config on X and P is identical and very basic: ----------------------------------------------------- options { time_reopen (10); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); keep_timestamp(yes); log_fifo_size (1000); stats_freq(60); flush_timeout(10000); log_fetch_limit(100); flush_lines(100); log_iw_size(100); }; source s_tcp { tcp( port(514) max-connections(100) ); }; source s_udp { udp( port(514)); }; destination d_tcp { file("/var/log/tcp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); }; destination d_udp { file("/var/log/udp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); }; ----------------------------------------------------- L -> X over TCP, L opens one TCP connections and everything comes over it. L -> P over TCP, L keeps opening up connections with P but no data comes over those connections or very few events come over those connections. I am going to investigate if there are any network QoS rules in place between L and P but wondering if there might be something in Syslog-NG that might be broken. I ran syslog-ng in debug mode but saw no errors. I looked at the pcap captures between L and P and saw no errors. Any other clues/ideas for troubleshooting are welcome. TIA!

2 3

Re: [syslog-ng] weird filter problem
by Evan Rempel 17 May '13

17 May '13

Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x My advice my be specific to version 3.4 :-( Evan Rempel 250.271.7691 University Systems, University of Victoria Evan Rempel <erempel(a)uvic.ca> wrote: This definitely works. I'm using it right now. If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: On 11/05/2013, at 2:26 PM, Evan Rempel <erempel(a)uvic.ca> wrote: > Try this filter > > > filter f_unknown { > tags(".classifier.unknown"); > }; > This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start: Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory So far as I can tell all the lib files are present and correct and in the same place as the previous version? I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use. Russell ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 5

[PATCH (3.5)] system-source: Make /proc/kmsg optional
by Gergely Nagy 16 May '13

16 May '13

In certain environments (vserver and OpenVZ come to mind), /proc/kmsg is not readable, not even for root. On these systems, emit a warning, and skip it, instead of aborting the startup. Reported-by: Tamas Pal <folti(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/system-source/system-source.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 3c72859..955ade7 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -167,8 +167,17 @@ system_sysblock_add_linux_kmsg(GString *sysblock) close (fd); } - system_sysblock_add_file(sysblock, kmsg, -1, - "kernel", "kernel", format); + if (access(kmsg, R_OK) == -1) + { + msg_warning("system(): The kernel message buffer is not readable, " + "please check permissions if this is unintentional.", + evt_tag_str("device", kmsg), + evt_tag_errno("error", errno), + NULL); + } + else + system_sysblock_add_file(sysblock, kmsg, -1, + "kernel", "kernel", format); } gboolean -- 1.7.10.4

2 1

insider 2013-05: syslog-ng configurator on Android; Using syslog-ng with Splunk; EU data protection and logging
by Peter Czanik 16 May '13

16 May '13

Dear syslog-ng users, This is the 24th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng configurator app for Android -------------------------------------- There is now a new configurator app available for Android, developed as a hobby project by one of the syslog-ng team members. The focus of the application is to create a syslog-ng.conf wich provides optimum performance based on a number of questions. For more details and download locations read the authors blog at http://pzolee.blogs.balabit.com/2013/04/little-syslog-ng-configurator-appli… syslog-ng and Splunk -------------------- We often receive questions, how to use syslog-ng and Splunk together in a logging infrastructure. We collected the most popular usage scenarios into a white paper, together with example configurations to make testing and integration even more easy. The WP is available at http://www.balabit.com/support/documentation/pdf/syslog-ng_splunk_deploymen… syslog-ng is participating GSoC again ------------------------------------- Just as last year, syslog-ng is participating Google Summer of Code under the umbrella of the openSUSE project. We have candidates for developing a native mysql destination, a redis destination, a XMPP (jabber) destination and log signing, which is a big improvement from last year, where we only had a single candidate for a similar number of development projects. The application process is closed now and there are still a couple of weeks to go, before the final list of approved students is announced. http://news.opensuse.org/2013/04/25/opensuse-hedgewars-and-owncloud-are-mov… Big changes ahead for EU data protection regulation --------------------------------------------------- This summer will most likely bring big changes in the regulation of Data Protection in the European Union. We collected these proposed changes and also how syslog-ng and proper central log management can help to comply with these regulations: http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu- <http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-data-pro…>data-protection-regulation/ <http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-data-pro…> Compiling syslog-ng with MS SQL support on RHEL / CentOS & Co. -------------------------------------------------------------- During the past few months many people asked, how to log from syslog-ng to MS SQL on RHEL or CentOS. If you cannot buy syslog-ng PE (http://www.balabit.com/network-security/syslog-ng/central-syslog-server) follow these steps to compile all the necessary components yourselfand configure the MS SQL part: https://czanik.blogs.balabit.com/2013/04/compiling-syslog-ng-with-ms-sql-su… SHORT NEWS: * RFC5424 syslog between syslog-ng and rsyslog: https://czanik.blogs.balabit.com/2013/04/using-the-rfc5424-syslog-protocol-… * Linuxwochen: syslog-ng, AMQP, and 3D printers: http://czanik.blogs.balabit.com/2013/05/czp-linuxwochen-syslog-ng-amqp-and-… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Problems with failed connections and time_reopen()?
by Matt Wise 13 May '13

13 May '13

We're running Syslog-NG 3.3.4 in our mixed Ubuntu 10/12 environment. We use SSL for all of our syslog-to-syslog connections, and have logging going to two different data pipelines. Data Dest #1: SyslogNG Client ----(SSL)----> SyslogNG Server ------> Logstash File-read-in-service Data Dest #2: SyslogNG Client ----(SSL)----> Stunnel Service ------> Flume Syslog Service The data streams work fine most of the time, but if we restart either the remote syslog-ng server, or the stunnel service, it seems that the syslog ng clients don't try to reconnect for a LONG time (or ever) to the endpoints again. I end up seeing the connection on the client go into a CLOSE_WAIT state, and syslog-ng keeps thinking that its sending log events through the connection, so it seems to never try to reconnect. I've tried setting time_reopen() to 0, 1 and 5... no luck or change in behavior. Any thoughts? --Matt

3 9

Question on custom log writer message output
by Dylan Kulesza 12 May '13

12 May '13

I'm working on a custom module to integrate with a third parties native log format. My intent is to have messages come into syslog-ng and processed as usual and then sent out a custom destination driver. Right now I've hacked together different code to make it work (tcp socket connection per log source) and now I'm at the point of actually sending a custom message. I've tried to stay as "true" to the syslog-ng as possible and have leveraged the log_forward_msg method to send my LogMessage. I was hoping I could just prepend data to the LogMessage but realized after doing all the other leg work that it wasn't a simple string :) So, my question is - what would the easiest way to leverage the existing queue->log_forward_msg (doesn't require the socket to be open vs examples such as spoof_source in afsocket) to write a custom message? I see that LogTemplate may have what I need, but after submerging myself in syslog-ng for the past week I'm not seeing clearly... Can anyone lend a hint/helping hand? What I'm trying to do: Open Socket Send Magic/StartPacket Prepend all log messages with a byte message - for example: 040404040400010MESSAGE (Of course Message would be in bytes/hex). It seems I would create an NVENTRY for my prepend message and then override log_writer_format_log? to do this? Not 100% clear how I would accomplish this. I also don't want to change any of the core/lib syslog-ng to accomplish this. Should be implemented purely as a module. Thanks!

2 2

weird filter problem
by Russell Fulton 11 May '13

11 May '13

Currently on 3.2.4. I am using patterndb to classify my incoming logs and want to send those that match pattern rules do a particular destination. filter f_unknown { "${.classifier.class}" ne 'unknown' ; }; but this appears to always fail (if I change ne to eq then I get everything). I have also tried using match with the same result. Russell

2 3

rewrite part of the message...
by Russell Fulton 11 May '13

11 May '13

Apologies to those of you who have already seen this on the ELSA list. I want to get rid of the explanatory essay that accompanies some eventlog messages from windows. I tried this: rewrite r_snarex { subst("\s+This event is generated when[^|]+\|", "|", value("MSGONLY") type("pcre")); }; and added it to log section, but it did not work. I have tried various variations on the theme too. Russell

2 2

syslog-ng and logrotate
by Rakesh Rajasekharan 10 May '13

10 May '13

I am using the syslog-ng 3.3.7 version. My question is , can we manage log rotation within the syslog-ng conf file ? Please advice

4 5

[Bug 234] New: eventlog: can' t build shared libraries under Windows with mingw
by bugzilla＠bugzilla.balabit.com 08 May '13

08 May '13

https://bugzilla.balabit.com/show_bug.cgi?id=234 Summary: eventlog: can't build shared libraries under Windows with mingw Product: syslog-ng Version: unspecified Platform: PC OS/Version: Windows Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: ramon.garcia.f+syslog(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 When trying to building, libtool gives an error message saying that it cannot find shared library versions of Windows base libraries, like KERNEL32.DLL The fix is simple: upgrade libtool. After libtoolizing with libtool 2.4.2 it builds without trouble. Best regards. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH (3.5)] afsocket: Fix compilation with systemd support enabled
by Gergely Nagy 08 May '13

08 May '13

self->super.socket_options->sock_type is replaced with self->super.transport_mapper->sock_type, follow up on this in the systemd-enabled codepath too. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afunix-source.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/afsocket/afunix-source.c b/modules/afsocket/afunix-source.c index dd8b8bb..c3dcbe7 100644 --- a/modules/afsocket/afunix-source.c +++ b/modules/afsocket/afunix-source.c @@ -74,7 +74,7 @@ afunix_sd_acquire_named_socket(AFSocketSourceDriver *s, gint *result_fd, if (sd_is_socket_unix(fd, 0, -1, filename, 0)) { /* check if it matches our idea of the socket type */ - if (sd_is_socket_unix(fd, self->super.socket_options->sock_type, -1, filename, 0)) + if (sd_is_socket_unix(fd, self->super.transport_mapper->sock_type, -1, filename, 0)) { *result_fd = fd; break; @@ -84,7 +84,7 @@ afunix_sd_acquire_named_socket(AFSocketSourceDriver *s, gint *result_fd, msg_error("The systemd supplied UNIX domain socket is of a different type, check the configured driver and the matching systemd unit file", evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", fd), - evt_tag_str("expecting", self->super.socket_options->sock_type == SOCK_STREAM ? "unix-stream()" : "unix-dgram()"), + evt_tag_str("expecting", self->super.transport_mapper->sock_type == SOCK_STREAM ? "unix-stream()" : "unix-dgram()"), NULL); return FALSE; } -- 1.7.10.4

1 0

Re: [syslog-ng] supervisor not restarting a failed daemon process
by Balazs Scheidler 07 May '13

07 May '13

Hi, No kernel source handy but my guess is that the kernel is logging the tid value, which is the same as pid as long as the process is single threaded. I've checked the supervisor code and the only way this could happen is a fork/pipe error which is not logged. It should be logged but who knows that message can be lost. The supervisor attempts to restart 3 times then gives up. Hmmm the supervisor messages may be redirected to syslog after the first startup which might explain why they don't get logged. But why does fork/pipe fail? Hope this helps. Evan Rempel <erempel(a)uvic.ca> wrote: >The logs below show a "standard" syslog-ng processID=15017 that reads /dev/log /proc/kmsg >The second instance of syslog-ng is what we call our "server" which just listens on the network ports >and does all of the complex patterndb, filtering and routing to destination processes > >kern.info kernel: syslog-ng[1561]: segfault at 7f65c0000078 ip 00007f65c0000078 sp 00007f65e1385a48 error 15 >--- this is the server instance segfaulting (I assume, see WAIT below) > >syslog.notice syslog-ng[15017]: Syslog connection closed; fd='20', client='AF_INET(142.104.47.145:49803)', local='AF_INET(127.0.0.1:1514)' >syslog.notice syslog-ng[15017]: Syslog connection broken; fd='14', server='AF_INET(142.104.47.146:514)', time_reopen='5' >--- this was the standard syslog loosing connection to the server, and detecting the drop of the server instance destination to it. > >daemon.info syslog-ng-stats: server stopping on socket "/var/local/syslog-ng.server.ctl" >daemon.info msgid_profiler[832]: committing residual data >local0.info flare-timer[834]: stopping >local0.info action-handler[833]: stopping >--- these are all program destinations of the server instance shutting down gracefully after the close of their stdin. > >daemon.crit supervise/syslog-ng[27221]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' >syslog.err syslog-ng[15017]: Syslog connection failed; fd='14', server='AF_INET(142.104.47.146:514)', error='Connection refused (111)', time_reopen='5' > >This "connection failed" message repeats every 5 seconds until I restart the server instance. > >syslog.notice syslog-ng[1911]: syslog-ng starting up; version='3.4.1' > >So it does not look like there is anything in the logs about attempted restarts. > > >WAIT... > >This is really odd. The line > >kern.info kernel: syslog-ng[1561]: segfault at 7f65c0000078 ip 00007f65c0000078 sp 00007f65e1385a48 error 15 > >implies that there was a process ID 1561 that segfaulted, but that line is the ONLY logged line with that process ID. >We take ps snapshots every 15 minutes, and those snapshots don't show anything for that process ID. >Also, the supervisor processID is shown > > USER PID PPID NI PRI CPU VSZ ELAPSED TIME COMMAND >root 27221 1 0 19 - 26556 9-04:00:59 00:00:00 supervising syslog-ng >root 27222 27221 0 19 - 977064 9-04:00:59 1-02:52:35 /usr/local/sbin/syslog-ng --cfgfile= ... > >which matches the line >daemon.crit supervise/syslog-ng[27221]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' > >so its child which dies should have been processID 27222 so why is the log line >kern.info kernel: syslog-ng[1561]: segfault at 7f65c0000078 ip 00007f65c0000078 sp 00007f65e1385a48 error 15 > > >I conclude that the 1561 is not the process ID. > >Can you shed any light on this? > >Evan. > >On 04/26/2013 11:00 PM, Balazs Scheidler wrote: > > >> Strange, indeed. The supervisor gives up if the restarted daemon exits for some reason. Eg. If there's an initialization error it gives up. Any indication in the logs? >> >> Evan Rempel <erempel(a)uvic.ca> wrote: >> >>> We are sing the log line >>> >>> supervise/syslog-ng[27221]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' >>> >>> >>> and it looks like it should restart, but instead of restarting, >>> the supervisor terminates and then no syslog-ng process is running. >>> >>> Is this a bug in the supervisor? >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > >-- >Evan Rempel erempel(a)uvic.ca >Senior Systems Administrator 250.721.7691 >Data Centre Services, University Systems, University of Victoria >______________________________________________________________________________ >Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

2 3

Re: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
by 不坏阿峰 06 May '13

06 May '13

thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there. how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work. 2013/4/28 <syslog-ng-request(a)lists.balabit.hu> > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Can not get DBParse match macro result (syslog-ng 3.13 > debian squeeze) (????) > 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 > debian squeeze) (Evan Rempel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 27 Apr 2013 22:34:50 +0800 > From: ???? <onlydebian(a)gmail.com> > Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng > 3.13 debian squeeze) > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= > V7wJ1EJdvE6Zx9zg(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > when use pdbtool do match test, it is success. but from syslog-ng can not > return result of macro > i can not get macro result. for example, ${.esxi.month} no value, same > as ${.esxi.host_ip} ${.esxi.time} > > test log output ,just like this. > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > === system,error,critical, HOST IP , > > > do the pdbtool test, it's ok. wish someone can give me some solution and > help. i have search some mail list but i can not get the right solution. > thanks a lot. > > root@debian:~# pdbtool match -D -c -p > /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 > 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd > 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path > "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. > Act:NONE" > Pattern matching part: > @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ > @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ > @ESTRING:.esxi.program= > vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a > (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. > Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: > nmp_ThrottleLogForDevice:2319: Cmd > 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ > Matching part: > Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: > nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev > mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid > sense data: 0x5 0x20 0x0. Act:NONE > Values: > MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: > nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev > mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid > sense data: 0x5 0x20 0x0. Act:NONE > PROGRAM=ESXI > .classifier.class=esxi > .classifier.rule_id=182437592347598 > .esxi.month=Apr > .esxi.date=26 > .esxi.time=15:17:31 > .esxi.host_ip=192.168.88.71 > .esxi.program= vmkernel > .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a > (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE > root@debian:~# > > > my configuration like as below > > ######## esxi_pattern.xml ############ > <?xml version="1.0" encoding="utf-8"?> > <patterndb version='3' pub_date='2009-04-17'> > <ruleset name='esxi' id='123456678'> > <pattern>ESXI</pattern> > <rules> > <rule provider='Fone Bro' id='182437592347598' class='esxi'> > <patterns> > <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ > @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ > @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > ######## syslog-ng.conf ######## > > #####Parser##### > parser pattern_db { > db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); > }; > > #Check pattern matching > destination udp_esxi_output { > file("/var/log/pattern_output" > template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST > IP ${.esxi.host_ip},${.esxi.message}\n") > template_escape(no)); > }; > > #####Log##### > log { > source(s_network); > parser(pattern_db); > destination(udp_esxi_output); > }; >

3 7

Re: [syslog-ng] syslog-ng Digest, Vol 97, Issue 4
by Yarick Tsagoyko 05 May '13

05 May '13

Dear R, cool issue. I have not seen this exact error on my compiles, here are some pointers: 1. Post your config.log 2. View the location of the referenced file(s) In file included from /usr/include/net/ethernet.h:27, from /usr/include/libnet.h:98, from afinet-dest.c:43 3. Rerun the configure, make clean, make. Sometimes rerunning the configure and running make cleac helps. Please post the findings. On May 5, 2013 6:00 AM, <syslog-ng-request(a)lists.balabit.hu> wrote: > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. problems compiling OSE 3.4.1 on redhat enterprise 6.4 > (Russell Fulton) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 5 May 2013 13:28:04 +1200 > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > Subject: [syslog-ng] problems compiling OSE 3.4.1 on redhat enterprise > 6.4 > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <D1B88BF5-CE66-485B-93FF-ECE892F2F950(a)auckland.ac.nz> > Content-Type: text/plain; charset=us-ascii > > make[4]: Entering directory > `/home/rful011/syslog-ng-3.4.1/modules/afsocket' > /bin/sh ../../libtool --tag=CC --mode=compile gcc -std=gnu99 > -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread > -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include > -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE > -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include > -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT > libafsocket_notls_la-afinet-dest.lo -MD -MP -MF > .deps/libafsocket_notls_la-afinet-dest.Tpo -c -o > libafsocket_notls_la-afinet-dest.lo `test -f 'afinet-dest.c' || echo > './'`afinet-dest.c > libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib > -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include > -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD > -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include > -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT > libafsocket_notls_la-afinet-dest.lo -MD -MP -MF > .deps/libafsocket_notls_la-afinet-dest.Tpo -c afinet-dest.c -fPIC -DPIC -o > .libs/libafsocket_notls_la-afinet-dest.o > In file included from /usr/include/net/ethernet.h:27, > from /usr/include/libnet.h:98, > from afinet-dest.c:43: > > > when I first encountered this error I went and did some googling and > followed some vague leads to install a bunch of -devel packages (most of > which were already installed ) but it did not make any difference. > > Googling the message brings up a bunch of hits for both redhat and debian > families and a range of packages. There did not seem to be any common > cause. > > Russell > > > > ------------------------------ > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > End of syslog-ng Digest, Vol 97, Issue 4 > **************************************** >

1 0

Prevent message loss while using tcp output
by Abhijeet Rastogi 03 May '13

03 May '13

Hi all, I've source defined as: source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); }; And one of the destination as: destination d_logstash { tcp( "127.0.0.1" port(515) template(t_both_timestamps) flush_lines(10)); }; Problem is when my consumer is down, I can see messages being dropped in "syslog-ng-ctl stats". How can I avoid this at the expense of memory (like a queueing mechanism or something)? -- Regards, Abhijeet Rastogi (shadyabhi) http://blog.abhijeetr.com

2 1

syslog-ng 3.3.6 Hangs on Solaris 8 sparc
by Rakesh Rajasekharan 02 May '13

02 May '13

Hi All, I have compiled the syslog-ng version 3.3.6 on a Solaris Sparc machine . The compilation went through fine but when I try to run syslog-ng .. though the daemon gets running somehow the unix prompt is not returned.. and it hangs... For ex:- # ps -ef|grep -i syslog root 14073 11758 0 07:11:37 pts/4 0:00 vi syslog-ng.d root 14097 13526 0 07:15:48 pts/1 0:00 grep -i syslog Now if I start the daemon using the below options... # /opt/soe/local/syslog-ng-3.3.6/sbin/syslog-ng -f /opt/soe/local/syslog_ng/etc/syslog-ng/syslog-ng.conf -p /var/run/syslog-ng.pid I never get back the prompt.. but if on another window I check.. the daemon has started.. ps -ef|grep -i syslog root 14099 11758 0 07:16:17 pts/4 0:00 grep -i syslog root 14098 13526 23 07:16:04 pts/1 0:13 /opt/soe/local/syslog-ng-3.3.6/sbin/syslog-ng -f /opt/soe/local/syslog_ng/etc/s though I guess... i should expect one more child to this... can someone please help me address this issue please... Thanks, Rkaesh.

3 10

[Bug 233] New: openssl CAfile configuration option
by bugzilla＠bugzilla.balabit.com 30 Apr '13

30 Apr '13

https://bugzilla.balabit.com/show_bug.cgi?id=233 Summary: openssl CAfile configuration option Product: syslog-ng Version: unspecified Platform: All OS/Version: All Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: james(a)chatid.com Type of the Report: enhancement Estimated Hours: 0.0 Today I was trying to set up syslog-ng on some servers and encountered trouble configuring the server to send intermediary certificates. >From what I can see[1], the ca_dir option is the only way to get intermediary certs attached. I can only assume that this maps down to a call to the `SSL_CTX_load_verify_locations` function in the openssl library[2] with a `NULL` CAfile argument and `ca_path` as the third argument Can a configuration option that maps to CAfile be added to syslog-ng? [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid… [2] https://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 229] New: json parser does not maintain all json object data
by bugzilla＠bugzilla.balabit.com 30 Apr '13

30 Apr '13

https://bugzilla.balabit.com/show_bug.cgi?id=229 Summary: json parser does not maintain all json object data Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Then the json parser is used parser p_json { json-parser(); }; log { source(mysource); parser(p_json); ... } tag-value pairs are not maintained if they are ones that are normally populated by syslog-ng. Even with a source that has flags(no-parse) man macros are populated by syslog-ng C_* S_* R_* YEAR_DAY YEAR WEEKDAY_NAME WEEK_DAY_ABBREV WEEK_DAY WEEKDAY WEEK USEC UNIXTIME TZOFFSET TZ TAGS SYSUPTIME STAMP SEC MSEC MONTH_WEEK MONTH_NAME MONTH_ABBREV MONTH MIN LOGHOST ISODATE HOUR12 HOUR FULLDATE DAY DATE AMPM the json parser should populate the macros after the initial population allowing the json data to overwrite/control all of the macros. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4