September 2017 - syslog-ng - lists.balabit.hu

UDP devices stop logging after network services restart.
by Vincenti Francesco 05 Oct '17

05 Oct '17

Hello I have syslog-ng 3.7 installed on a CentOS7 Server with 4G RAM and 4 CPU. The server is receiving logs from almost 300 devices, either network devices (UDP) and servers (TCP). The problem I have is the following: to reduce incoming traffic on initial two NICs I added two more NICs to the server, where I sent some UDP devices and it worked immediately, without any issue. But when I restarted network services (systemctl restart network) the devices stopped to log, they restarted only after I resent them to the initial NIC, without any other action on syslog-ng server. When I started syslog-ng in debug mode to verify this behavior I saw that these devices ARE NOT considered by syslog-ng at all, they are not present in logs generated by debug, but they REACH the server (tcpdump shows them entering). I'm getting mad about this issue, any suggestion will be welcome. Following my configuration files: - /etc/syslog-ng/syslog-ng.conf # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (persist_only); dns-cache-hosts(/etc/hosts); use_fqdn (no); create_dirs (yes); keep_hostname (yes); owner ("1007"); group ("1007"); dir_owner ("1007"); dir_group ("1007"); dir_perm (0750); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_auth_loc { network ("127.0.0.1" port (601) ); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); destination(d_auth_loc); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: - /etc/syslog-ng/conf.d/network_dev.conf source s_network_appa { network ( ip(0.0.0.0) port (514) transport ("udp") so-rcvbuf (4096000) ); }; # Destinazioni destination d_network_udp { file("/var/log/syslog-ng/APPARATI/${YEAR}${MONTH}${DAY}/${HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}" create_dirs(yes) dir_perm(0755) perm(0755) ) ; }; filter f_appa_rete { not host ("SymantecServer") and not host ("part8") and not host ("parti") and not host ("partd") and not host ("part1") and not host ("parte") and not host ("part10") and not host ("part11") and not host ("part12") and not host ("part3") and not host ("part6") and not host ("part8") and not host ("part9") and not host ("parta") and not host ("partc") and not host ("partd") and not host ("partf") and not host ("partg") and not host ("parth") and not host ("partn") and not host ("parto") and not host ("SLITES8736.sercom.aspasiel.it<http://SLITES8736.sercom.aspasiel.it>") and not host ("172.23.112.47") and not host ("SLITES4557.sercom.aspasiel.it<http://SLITES4557.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES4558.sercom.aspasiel.it<http://SLITES4558.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES4559.sercom.aspasiel.it<http://SLITES4559.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES4560.sercom.aspasiel.it<http://SLITES4560.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8731.sercom.aspasiel.it<http://SLITES8731.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8732.sercom.aspasiel.it<http://SLITES8732.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8733.sercom.aspasiel.it<http://SLITES8733.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8734.sercom.aspasiel.it<http://SLITES8734.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8735.sercom.aspasiel.it<http://SLITES8735.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("slitdd2727"); }; log { source(s_network_appa); filter(f_appa_rete); destination(d_network_udp); }; - /etc/syslog-ng/conf.d/server_TCP.conf source s_server_sop { network ( ip(0.0.0.0) port(601) transport("tcp") max-connections(200) log_fetch_limit(100) log_iw_size(20000) ); }; destination d_server_sop { file("/var/log/syslog-ng/SERVER/${YEAR}${MONTH}${DAY}/${HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}" create_dirs(yes) dir_perm(0755) perm(0755) flush_lines(100) ); }; # Indirizzo le destinazioni. log { source(s_server_sop); destination(d_server_sop); flags(flow-control); }; I changed the following kernel parameters as follow: net.core.rmem_max = 4096000 net.core.wmem_max = 4096000 Thanks in advance Francesco Vincenti RHCSA Area Data Center Open Source, Quality and Security Aspasiel Divisione della Società Acciai Speciali Terni S.p.A. con Unico Socio Strada di Pentima, 3 - 05100 Terni francesco.vincenti(a)acciaiterni.it<mailto:francesco.vincenti@acciaiterni.it> www.aspasiel.it<http://www.aspasiel.it/> Ufficio: +39 0744 203224 Fax: +39 0744 203444 This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the error at the following email address: helpdesk(a)aspasiel.it<mailto:helpdesk@aspasiel.it> or at Aspasiel Helpdesk Team by phone (phone number +390744203555), and then delete this message from your system. P Please consider our environment and think before you print. Thank you! q

3 6

Cross-compiling syslog-ng 3.11.1 for an embedded application.
by Robert King 04 Oct '17

04 Oct '17

Greetings! We are currently running syslog-ng v3.8.1 in an embedded Linux system. However, we want to upgrade to 3.11.1 in order to eliminate some memory leaks. Unfortunately we've run into a bit of a snag getting 3.11.1 to run on our target system. Under 3.8.1, syslogND is an ELF binary installed in the sbin directory. Under 3.11.1, it is a *script* installed in sbin. This script is trying to do some bad things including linking an executable (lt-syslog-ng, I believe.) We do not have a linker that will run on the target system. We cannot link anything. What is the reason behind this and how can it be circumvented? -- Robert King, Tellabs

5 11

syslog-ng 3.11 to elasticsearch 2.4.6
by Blake Pomeroy 29 Sep '17

29 Sep '17

Hello, I am trying to set up syslog-ng to elasticsearch, not using as the middleware. According to https://www.balabit.com/blog/elasticsearch-and-syslog-ng-fast-and-simple/ it is doable. However, I am not finding an example that works for my config. I have syslog-ng version 3.11 Elasticsearch 2.4.6 All running on one machine with Ubuntu 16.04. My syslog-ng is standard with the @include for the conf.d directory. In which I had hoped to create a working elsasticsearch.conf to define the how to send the logs. Likewise I have syslog-ng-mod-elasticsearch installed. Any help or pointing me to some examples would be great. Thanks Blake Pomeroy Security Engineer Cloud Engineering/InfoSec EBSCO Desk 978-356-6500 x3693 www.ebsco.com<http://www.ebsco.com> [cid:image001.png@01D3393C.98252AF0]

2 1

Bugfix releases?
by Fabien Wernli 29 Sep '17

29 Sep '17

Hi, While I'm very happy with the current rapid development cycle of syslog-ng, I can't stop wondering: what happened to bugfix releases? The quality of the releases is very high indeed, but regressions still appear now and then, and some users (not us) prefer the safe approach of sticking to one major release, and applying only bug fix releases. I think these people may have a hard time nowadays. What's the mailing-list reader's opinion on that? Cheers

2 1

syslog-ng web interfaces
by Czanik, Péter 28 Sep '17

28 Sep '17

Hi, Two years are almost like an eternity in IT, so I'm planning to update my blog about syslog-ng web interfaces: https://www.balabit.com/blog/web-interfaces-for-your-syslog-server-an-overv… End of life software will be removed, I will update information on others and if there are any new apps, I'm more than happy to add those. I need your help with the latest category. Are you aware of any software which you use as web GUI for syslog-ng but not listed in my blog? If yes, please let me know either here on the list or in a direct e-mail. Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

unofficial syslog-ng 3.12 RPM packages
by Czanik, Péter 25 Sep '17

25 Sep '17

Hi, Syslog-ng 3.12 was released this morning. I have now my unofficial RPM packages available for openSUSE / SLES 12 / EPEL 7 (RHEL, CentOS, etc.). Fedora is missing from the list due to https://github.com/balabit/syslog-ng/issues/1669 You can read my updated installation instructions at https://www.balabit.com/blog/installing-latest-syslog-ng-on-rhel-and-other-… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

OSE 3.12
by Laszlo Budai 25 Sep '17

25 Sep '17

3.12.1 Features * HDFS: support macro in filename (#1638<https://github.com/balabit/syslog-ng/pull/1638>) * HDFS: add append support (#1675<https://github.com/balabit/syslog-ng/pull/1675>) * Java: allow to use sequence numbers in templates (#1628<https://github.com/balabit/syslog-ng/pull/1628>) * TLS improvements (#1603<https://github.com/balabit/syslog-ng/pull/1603>, #1636<https://github.com/balabit/syslog-ng/pull/1636>) * Add PKCS 12 support with the new pkcs12-file() TLS option * startup time ssl-options() and peer-verify() check * startup time key_file, cert_file, ca_dir, crl_dir and cipher_suite check * ECDH cipher support (OpenSSL 1.0.1, 1.0.2, 1.1.0) with the ecdh-curve-list() option (only available >= 1.0.2) * for < 1.0.2, a hard-coded curve is used * for >= 1.0.2, automatic curve selection is used (the ecdh-curve-list() option can restrict this list) * DH cipher support with the dhparam-file() option * if the option is not specified, fallback RFC 3526 parameters are used * minor fixes * stdin() source driver (#1605<https://github.com/balabit/syslog-ng/pull/1605>) * Implement read_old_records option for systemd-journal source (#1642<https://github.com/balabit/syslog-ng/pull/1642>) * Add tags-parser: a new module to parse $TAGS values (#1658<https://github.com/balabit/syslog-ng/pull/1658>) * Add a Windows eventlog parser scl module (#1572<https://github.com/balabit/syslog-ng/pull/1572>) * Add XML parser module (#1659<https://github.com/balabit/syslog-ng/pull/1659>, #1684<https://github.com/balabit/syslog-ng/pull/1684>) Bugfixes * Fix cannot parse ipv6 into hostname (#1617<https://github.com/balabit/syslog-ng/pull/1617>) * Speedup add-contextual-data by making ordering optional (#1645<https://github.com/balabit/syslog-ng/pull/1645>) * Fix monitor-method() option not working for wildcard-file() source (#1651<https://github.com/balabit/syslog-ng/pull/1651>) * Sanitize SDATA keys in syslog-protocol messages to avoid generating non-valid messages (#1650<https://github.com/balabit/syslog-ng/pull/1650>, #1654<https://github.com/balabit/syslog-ng/pull/1654>) * Fix memory leaks reported using Valgrind (#1649<https://github.com/balabit/syslog-ng/pull/1649>) * Fix memory leak related to cloning pipes and reload (#1647<https://github.com/balabit/syslog-ng/pull/1647>) * Fix getent protocol number returns incorrect value (#1665<https://github.com/balabit/syslog-ng/pull/1665>) * Fix elasticsearch2 destination flush mechanism (#1668<https://github.com/balabit/syslog-ng/pull/1668>) * Fix file destination related memory leak (#1685<https://github.com/balabit/syslog-ng/pull/1685>) * Fix a possible memory leak around affile destination (#1685<https://github.com/balabit/syslog-ng/pull/1685>) Other changes * Improve syslog-ng debun functionality (#1633<https://github.com/balabit/syslog-ng/pull/1633>, #1641<https://github.com/balabit/syslog-ng/pull/1641>, #1663<https://github.com/balabit/syslog-ng/pull/1663>) * Java: allow to set JVM options form global syslog-ng options (#1639<https://github.com/balabit/syslog-ng/pull/1639>) * Do steps towards Python 3 support: * Fix string compatibility for Python 3 (#1632<https://github.com/balabit/syslog-ng/pull/1632>) * Improve Python version auto detection (#1660<https://github.com/balabit/syslog-ng/pull/1660>) * HTTP destination: display verbose logs on debug level (#1526<https://github.com/balabit/syslog-ng/pull/1526>) * Improvements for Solaris packing (#1664<https://github.com/balabit/syslog-ng/pull/1664>) Notes to the Developers * Update internal RabbitMQ (#1662<https://github.com/balabit/syslog-ng/pull/1662>) * Update internal ivykis to v0.42 (#1566<https://github.com/balabit/syslog-ng/pull/1566>) * Fix Travis and test related issues (#1566<https://github.com/balabit/syslog-ng/pull/1566>, #1644<https://github.com/balabit/syslog-ng/pull/1644>, #1674<https://github.com/balabit/syslog-ng/pull/1674>) * Update docker images (#1637<https://github.com/balabit/syslog-ng/pull/1637>) * Fix some clang compile errors (#1662<https://github.com/balabit/syslog-ng/pull/1662>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy, Gergely Orosz, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Mate Farkas, Marton Suranyi, Peter Kokai, Szilard Pfeiffer, Tamas Nagy, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.12.1 [https://avatars1.githubusercontent.com/u/986981?v=4&s=400]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.12.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.12.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. github.com

1 0

upgrading syslog-ng on Debian/Ubuntu packages from unofficial OBS repository
by Laszlo Budai 22 Sep '17

22 Sep '17

Hi, if you got the following error: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY xxxxxx just update the APT key: curl -4 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… | sudo apt-key add - (if you are using other version of Debian/Ubuntu, set the correct version) and then apt-get update && apt-get upgrade regards, Laszlo Budai

1 0

unofficial OSE 3.12 packages for Debian/Ubuntu
by Laszlo Budai 22 Sep '17

22 Sep '17

Hi, I'm starting to work on upgrading my packages in OBS. Please test the packages before upgrading your servers. If you find any issues, just drop me a mail and I'll try to fix them. regards, Laszlo Budai

1 0

Insider 2017-09: 3.12; survey; geoip2; docker; AWS;
by Czanik, Péter 21 Sep '17

21 Sep '17

Dear syslog-ng users, This is the 61th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.12 is coming ------------------------ The latest version of syslog-ng, 3.12 is expected to arrive soon. The most important new feature is the XML parser, which creates name-value pairs from XML-based log messages. You will also be able to configure JVM options for Java destinations, use PKCS-12, and utilize macros in HDFS file names. There are many more smaller features and bug fixes. For a complete list, check the git commits: https://github.com/balabit/syslog-ng/commits/master Test RPM packages from git head: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ New syslog-ng website --------------------- We will introduce a brand new website for all that is syslog-ng, and with that will come an official syslog-ng OSE repository with binaries for some of the most popular Linux distributions. We expect to launch syslog-ng.com in November, moving content for both OSE and its commercial versions to this new site so that it can serve as a single hub for all syslog-ng users. Learn more about it and help our work by answering a brief survey: https://www.balabit.com/blog/new-website-and-official-repo-coming-up-tell-u… Collecting logs from containers ------------------------------- There are many software products that log to files or pipes instead of their stdout, the place where Docker expects them. Fortunately by using Docker volumes, you can share data among containers, and syslog-ng can collect these logs as well. Learn how: https://www.balabit.com/blog/collecting-logs-containers-using-docker-volume… Heat maps using the new geoip2 parser ------------------------------------- The new geoip2 parser of syslog-ng 3.11 is not only faster than its predecessor, but can also provide a lot more detailed geographical information about IP addresses. Next to the usual country name and longitude/latitude information, it also provides the continent, time zone, postal code and even county name. Some of these are available in multiple languages. Learn how you can utilize all this information by parsing logs from iptables using syslog-ng, storing them to Elasticsearch, and displaying the results in Kibana! https://www.balabit.com/blog/creating-heat-maps-using-new-syslog-ng-geoip2-… Learn how you can impress your C-level leader(s) even more by animating these heat maps: https://www.balabit.com/blog/creating-time-lapse-videos-log-messages-using-… Installing syslog-ng on AWS Linux AMI ------------------------------------- You do not have to live without your favorite syslog implementation even in Amazon Web Services (AWS) Linux AMI. This Linux distribution is based on Red Hat Enterprise Linux version 6 and it takes minimal extra work to install syslog-ng on it. https://www.balabit.com/blog/installing-syslog-ng-aws-linux-ami/ EVENTS Virtualization day in Budapest, November 7th: https://vday.hu/2017/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0