May 2016 - syslog-ng - lists.balabit.hu

Building syslog-ng 3.7.3 in Ubuntu Xenial
by Vithulan MV 15 May '16

15 May '16

Hi all, I got the tarball from, https://github.com/balabit/syslog-ng/releases (version 3.7.3). When I tried to build it, - configure passed. - make passed - make check failed with the following message, /usr/bin/ld: modules/systemd-journal/tests/modules_systemd_journal_tests_test_systemd_journal-test_systemd_journal.o: undefined reference to symbol 'iv_task_registered@@IVYKIS_0.29' //usr/local/lib/libivykis.so.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status Makefile:6297: recipe for target 'modules/systemd-journal/tests/test_systemd_journal' failed make[3]: *** [modules/systemd-journal/tests/test_systemd_journal] Error 1 Makefile:11738: recipe for target 'check-am' failed make[2]: *** [check-am] Error 2 Makefile:11313: recipe for target 'check-recursive' failed make[1]: *** [check-recursive] Error 1 Makefile:11741: recipe for target 'check' failed make: *** [check] Error 2 Also, getting this warning when compiling, Makefile:12423: warning: overriding recipe for target 'eclipse' Makefile:12398: warning: ignoring old recipe for target 'eclipse' What could be the cause for this problem? Thanks. Best regards, Vithulan. -- Vithulan MV. Undergraduate, Department of Computer Science & Engineering, University of Moratuwa, Sri lanka.

2 3

Syslog-ng meetup in London
by Balazs Scheidler 14 May '16

14 May '16

Hi, We are experimenting with a new syslog-ng related event format in London, 26th of May. I am travelling there and will be speaking about syslog-ng, past, present and future. Peter Czanik will also be there. The general approach is more about questions and discussions instead of presentations. Anyone in the area is welcome, beer is on @Balabit. :) RSVP at http://www.meetup.com/syslog-ng-london/events/231038819 Cheers, Bazsi

1 0

Syslog-ng with MongoDB authentication
by Ivan Adji - Krstev 13 May '16

13 May '16

Hi all, I have configure MongoDB with Username and Password, but can't find how to configure destination on Syslog-ng using MongoDB with Username and Password. For now i have just this: destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; But i should add some username and passwords somewhere and don't know how. Any hints on this ? Kind regards Ivan

2 1

Re: [syslog-ng] Installing Syslog-NG 3.7 on CentOS 7
by jrhendri 12 May '16

12 May '16

This has to be something very basic.Have you tried checking if another syslog server is running? ps -aef |grep syslog Assuming this shows nothing, try a very simple syslog-ng config file and a manual start on the command line. Make sure you check all the things in your configuration that your copy should open beforehand. This should narrow down the problem I hope :-) Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Ivan Adji - Krstev <akivanradix(a)gmail.com> Date: 5/12/16 5:26 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Installing Syslog-NG 3.7 on CentOS 7 So i have install EPEL and i have install syslog-ng and mongodb and when i start the syslog-ng service with syslog-ng -Fevd command i have the following error AGAIN :). Im not sure what is it and how to prevent it and what to do. But i really need this to work :(. [2016-05-12T05:21:10.739940] Error binding socket; addr='AF_INET(0.0.0.0:6514)', error='Address already in use (98)' [2016-05-12T05:21:10.739973] Error initializing message pipeline; [root@syslogserver loganalyzer]# netstat -tupl Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:27017 0.0.0.0:* LISTEN 1352/mongod tcp 0 0 0.0.0.0:syslog-tls 0.0.0.0:* LISTEN 11377/syslog-ng tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN 8562/sshd tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN 1778/master tcp6 0 0 [::]:http [::]:* LISTEN 11264/httpd tcp6 0 0 [::]:ssh [::]:* LISTEN 8562/sshd tcp6 0 0 localhost:smtp [::]:* LISTEN 1778/master udp 0 0 0.0.0.0:bootpc 0.0.0.0:* 638/dhclient udp 0 0 0.0.0.0:60094 0.0.0.0:* 638/dhclient udp6 0 0 [::]:3126 [::]:* 638/dhclient [root@syslogserver loganalyzer]# lsof | grep LISTEN mongod 1352 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 1393 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2028 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2033 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2034 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2138 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2139 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2141 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2148 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2404 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2446 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2447 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2448 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2449 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2450 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2451 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 11380 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) master 1778 root 13u IPv4 15893 0t0 TCP localhost:smtp (LISTEN) master 1778 root 14u IPv6 15894 0t0 TCP localhost:smtp (LISTEN) sshd 8562 root 3u IPv4 23963 0t0 TCP *:ssh (LISTEN) sshd 8562 root 4u IPv6 23965 0t0 TCP *:ssh (LISTEN) httpd 11264 root 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11265 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11267 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11268 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11269 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11270 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11275 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11276 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11277 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11278 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) syslog-ng 11377 root 14u IPv4 34906 0t0 TCP *:syslog-tls (LISTEN) syslog-ng 11377 11378 root 14u IPv4 34906 0t0 TCP *:syslog-tls (LISTEN) syslog-ng 11377 11541 root 14u IPv4 34906 0t0 TCP *:syslog-tls (LISTEN) httpd 11384 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) and the source config is as follow: source s_sys { system(); unix-stream("/dev/log"); internal(); network( port(6514) # tcp(port(5140)); # file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; destination d_mongodb { mongodb( # servers("localhost:27017") # database("syslog") # uri('mongodb://localhost/syslog-ng') collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan On 05/10/2016 01:35 PM, Czanik, Péter wrote: Do you also have EPEL? The RHEL7/CentOS7 repo is built against EPEL, as some of the dependencies are missing from the base distribution: https://fedoraproject.org/wiki/EPEL Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Tue, May 10, 2016 at 1:29 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: Hi i note this error of mine but i try the other one: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czan… And i have the similar errors when ever i try to install on new CentOS The procedure im doing is: Fresh installation of CentOS yum update yum install httpd php vim wget then install mongodb ( add repo ) then install syslog-ng ( add repo ) I'm using: CentOS Linux release 7.2.1511 (Core) And im having the following repos: [root@syslogserver ~]# yum repolist Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.switch.ch * extras: mirror.switch.ch * updates: mirror.switch.ch repo id repo name status base/7/x86_64 CentOS-7 - Base 9,007 czanik-syslog-ng37/x86_64 Copr repo for syslog-ng37 owned by czanik 59 extras/7/x86_64 CentOS-7 - Extras 266 mongodb-org-3.2/7 MongoDB Repository 35 updates/7/x86_64 CentOS-7 - Updates 1,437 repolist: 10,804 [root@syslogserver ~]# yum install syslog-ng Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.switch.ch * extras: mirror.switch.ch * updates: mirror.switch.ch Resolving Dependencies --> Running transaction check ---> Package syslog-ng.x86_64 0:3.7.3-3.el7.centos will be installed --> Processing Dependency: ivykis >= 0.36.1 for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.29)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.30)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libevtlog.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libnet.so.1()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Running transaction check ---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed ---> Package syslog-ng.x86_64 0:3.7.3-3.el7.centos will be installed --> Processing Dependency: ivykis >= 0.36.1 for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.29)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.30)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libevtlog.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Finished Dependency Resolution Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libivykis.so.0(IVYKIS_0.30)(64bit) Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libivykis.so.0()(64bit) Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: ivykis >= 0.36.1 Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libevtlog.so.0()(64bit) Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libivykis.so.0(IVYKIS_0.29)(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Any idea ? On 05/09/2016 04:09 PM, Czanik, Péter wrote: Hi, You should add the repository using the file: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czan… to yum and not just download individual packages. You can use then "yum install syslog-ng" which will also download all necessary dependencies. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Mon, May 9, 2016 at 3:07 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: I have the following errors when i try to install Syslog-NG 3.7 on CentOS 7 I have problem when i try to install Syslog-NG 3.7 on CentOS 7. The following errors i get: --> Finished Dependency Resolution Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libevtlog.so.0()(64bit) Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libpcre.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Any hints on this ? Kind regards Ivan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 11

Installing Syslog-NG 3.7 on CentOS 7
by Ivan Adji - Krstev 12 May '16

12 May '16

I have the following errors when i try to install Syslog-NG 3.7 on CentOS 7 I have problem when i try to install Syslog-NG 3.7 on CentOS 7. The following errors i get: --> Finished Dependency Resolution Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libevtlog.so.0()(64bit) Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libpcre.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Any hints on this ? Kind regards Ivan

3 6

Syslog-ng 3.8
by Ivan Adji - Krstev 12 May '16

12 May '16

Hi all, I have open a issue about the error i got when i update the Syslog-NG to 3.8 for "syslog-ng Address already in use (98)". I'm still working on the issue, but i would like to know if this newest version 3.8 is stabled as im building new environment and im asking myself do i use this 3.8 version or 3.7 ? Thanks Ivan

5 13

[GSOC]WebSocket for syslog-ng - Configuration file design to get feedbacks.
by Yilin Li 05 May '16

05 May '16

Hi, Very excited and thanks to the community that my proposal is accepted by syslog-ng in GSOC 2016. Here is my proposal: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-WebSocket-for… I have some spare time recently. So I plan to begin my work early This document describes what the configuration file will look like for the project that I will accomplish. WebSocket for syslog-ng (Yilin Li) - Configure ... <https://docs.google.com/document/d/1FVc4EySiTu9RHdlsPd-NGQ9JzvV0WInQrM2VBA4…> . I post it to the community to get feed backs. I will reiterate the design on the community feedback. I hope it will be what our users exactly want at last. Any comments will be greatly apprecitated :) And there is a question for my mentors Laszlo Meszaros and Viktor Juhasz, will it be OK that I go faster than my schedule ? -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

1 0

Insider 2016-05: BMW; GSoC; Hadoop; grouping-by; SIEM; Java; Rust;
by Czanik, Péter 05 May '16

05 May '16

Dear syslog-ng users, This is the 49th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng in the BMW i3 ----------------------- The BMW i3 electric car uses many open source software under the hood. BMW respects the GPL and made the source code of software used available upon request. You can find syslog-ng among them: https://github.com/edent/BMW-OpenSource Google Summer of Code (GSoC) ---------------------------- The syslog-ng project will participate in GSoC this year again, this time with five students. We are looking for Feature Owners who could represent syslog-ng users in the process. Topics range from websockets through CLI to Java-based Kafka source. Read more about the topics and learn how to contact the team at https://lists.balabit.hu/pipermail/syslog-ng/2016-April/022862.html syslog-ng is Hortonworks Data Platform certified ------------------------------------------------ Both syslog-ng OSE and PE provide a Hadoop destination and are now Hortonworks Data Platform certified. You can read more about why use syslog-ng in a Hadoop environment at https://czanik.blogs.balabit.com/2016/02/filling-your-data-lake-with-log-me… or on the Hortonworks website at http://hortonworks.com/partner/balabit/ The grouping-by() parser ------------------------ Until recently, you could only correlate and aggregate information from multiple messages with the PatternDB parser. The new grouping-by() parser in the upcoming syslog-ng version 3.8 can correlate and aggregate information independently from PatternDB, using data from any of the available parsers (CSV, key=value, PatternDB, JSON): https://czanik.blogs.balabit.com/2016/04/the-grouping_by-parser-in-syslog-n… Optimize your SIEM ------------------ Join us on Wednesday, May 18, 2016, at 2:00 PM EST to hear about the benefits of front-ending your SIEM with syslog-ng. Benefits include: increased performance, higher quality data, and cost-effective scalability. Register for the live webinar at https://pages2.balabit.com/optimize-siem/ Troubleshooting Java support ---------------------------- Java based destination drivers were introduced to syslog-ng last year. The syslog-ng application uses libjvm.so to embed a Java Virtual Machine inside syslog-ng. Learn which Java to use and how to make sure, that it is found by syslog-ng: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sy… Cybersecurity Excellence award for SSB -------------------------------------- Balabit has been named the winner of the 2016 Cybersecurity Excellence Award for Forensics for syslog-ng Store Box (SSB). Based on syslog-ng – one of the most widely adopted log management software packages – SSB is a highly reliable and high-performance log management appliance which collects, classifies, organizes, and securely stores log messages for any enterprise who operates a log management infrastructure. http://www.marketwired.com/press-release/balabits-syslog-ng-store-box-earns… Packaging Rust parsers ---------------------- Up until now, if you wanted to try the new Rust-based parsing in syslog-ng, you also had to build syslog-ng yourself from source code. Not anymore, if you use one of the RPM-based Linux distributions: https://czanik.blogs.balabit.com/2016/04/syslog-ng-3-8-preview-parsers-in-r… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com . To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

race condition causes copytruncate log rotation problems
by Jim Segrave 04 May '16

04 May '16

On a set of very busy Usenet servers, we are seeing problems with using syslog-ng together with copytruncate in logrotation. I am not sure why, but syslog-ng opens log files without having O_APPEND set, and appends log mesages with separate syscalls - an lseek to SEEK_END, followed by a write. On a busy server- perhaps 300 logs/sec each about 120 bytes long, we sometimes find that logrotate, which is configured to do a copy-truncate, actually produces a sparse file which begins with blocks of zeroes up to the pre-rotate log file size, followed by logs written after the copytruncate completes. Looking at the Linux /proc/PID/fdinfo information, I see that the logfiles are not opened in append mode, although they are opened O_WR. I haven't begun looking through the source to see if syslog-ng ever tries to overwrite data it's already logged by seeking backwards in the file - I must confess I can't think of any reason to do so. The only other reason for not having O_APPEND set would be to avoid problems for people using NFS to collect logs centrally by having different servers all writing to one NFS file. As I see it, the sequence leading to this problem is: syslog-ng logrotate copytruncate function ------------- ------------------------------------------- ,,, begin copytruncate, reach EOF on the copy lseek(fd, SEEK_END) <------------------------------------------------------------- the end of the file before logrotate ftruncates() it ftruncate(fd, 0) write(fd, log_msg, log_msg_size) <----------------------------------------- the write takes place at the old EOF, so the kernel considers the file to have empty blocks from 0 to that point If there is a reason that some users need the logfile opened without O_APPEND mode, then making that choice a configuration option would be helpful. For those using O_APPEND, the lseek() calls are harmless and won't add any more resource usage than is already present, but file truncation will prevent a zero filled pad being prefixed to the file.

3 2

Re: [syslog-ng] syslog-ng 3.7.3 with Java enable Can't find class when startup
by Ho, Ming 04 May '16

04 May '16

Hi, I use the following conf file to start syslog-ng 3.7.3 complied with Java supported version and got the following error: # syslog-ng -f /opt/syslog-ng/syslog-ng.conf [2016-05-03T23:13:54.191256] Can't find class; class_name='org/syslog_ng/SyslogNgClassLoader' ** ERROR:modules/java/native/java_machine.c:126:java_machine_get_class_loader: assertion failed: (self->loader) syslog-ng.conf: @version: 3.7 @include "scl.conf" @module mod-java source s_syslog { syslog(); }; destination d_elastic { java( class_path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class_name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("client-mode", "transport") option("server", "sl73ovnapd018") option("port", "9200") option("flush-limit", "10") ); }; log { source(s_syslog); destination(d_elastic); flags(flow-control); }; I found previous post from last year with the exact same error: https://github.com/balabit/syslog-ng/issues/620 According to the post the problem was make install fail to copy "syslog-ng-core.jar" and/or "syslog-ng-common.jar" to target directory. Including these two files in the directory referenced by the class_path should solve the problem. I have these two files included in my class_path already but still got the same error: -rw-r--r-- 1 root root 11411 May 2 06:50 /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar -rw-r--r-- 1 root root 12536 May 2 06:51 /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar Any help is highly appreciated. Ming Ho

1 0