December 2015 - syslog-ng - lists.balabit.hu

syslog-ng 3.7.2
by Budai, László 22 Mar '16

22 Mar '16

3.7.2 This is the first maintenance release for the 3.7.x series. Changes compared to 3.7.1: Improvements - Added mbox() source. This source can be used to fetch emails from local mbox files: source { mbox("/var/spool/mail/root"); }; This will fetch root emails and parse them into a multiline $MSG. Original implementation by Fabien Wernli, I only converted it into an SCL. - It is possible to append dynamically options into SCL blocks from now. - concurrent_request option added to ElasticSearch with default value 1. - In elasticsearch destinaton, message_template() argument renamed to template(). - SCL added to every Java module (ElasticSearch, Kafka, HDFS). - Linux Audit Parser added for parsing key-value pairs produced by the Linux Audit subsystem. - HTTP destination is now able to receive HTTP method as an option. All the supported methods are available (POST, PUT, HEAD, OPTIONS, DELETE, TRACE, GET). Fixes - In some circumstances syslog-ng mod-journal re-read every already processed messages. - When syslog-ng got a reload and the reload process done within 1 second then mafter the reload, syslog-ng stop generating mark-messages. - When initialization of a network destination in syslog-ng failed (eg. due to DNS resolution failure) we didn't create a queue which caused message loss. - syslog-ng segfaulted on TLS errors when wrong certs was provided (eg.: CA cert with the cert-file directive instead of the server cert). - Fixed a continuous spinning case in the file driver, when the destination file is a device (e.g. /dev/stdout). - A memory leak in around template functions in grammar fixed. - Fixed Python3 support. - Fixed Python GIL issue in python destination. - From now, instead of skipping doc/ alltogether when ENABLE_MANPAGES is not set, only skip the actual man pages, but handle the rest properly. - Allow overriding the python setup.py options. When installing the python modules, allow overriding the options. This is useful for distributions that want to pass extra options. For example, on Debian, we want --install-layout="deb" instead of the --prefix and --root options. With this change, the previous behaviour remains the default, but one can supply PYSETUP_OPTIONS on the make command-line to override it. - The systemd service file read /etc/default/syslog-ng and /etc/sysconfig/syslog-ng, but didn't do anything with their contents. $SYSLOGNG_OPTS added to ExecStart, so that the EnvironmentFiles have an effect (at least on Debian). - Java support checking fixed (not only jdk is required but also gradle). - Memory leak around ping() in Redis fixed. - A crash in pdbtool fixed around r_parser_email(). - Removed cygwin fdlimit statement. Make the default for RLIMIT_NOFILE equal to the current system limits. --fd-limit can still override this, but the default will be configured based on existing system limits. - Fixed BSD year inference. Fixed logic and made clearer the inference of year from bsd-style rfc3164 syslog-messages, which do not include a year. - Handle correctly the epoch 0 timestamp. (Previously, syslog-ng cached the zero timestamp and treated 1970 as it was 1900.) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Arsenault, Adam Istvan Mozes, Andras Mitzki, Avleen Vig, Balazs Scheidler, Fabien Wernli, Gergely Czuczy, Gergely Nagy, Gergo Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Saurabh Shukla, Tamas Nagy, Tibor Benke, Viktor Juhasz, Vincent Bernat, Wang Long, Zdenek Styblik, Zoltan Pallagi. — View it on GitHub <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2>.

4 3

Reg; Syslog-ng does not recognize the audit facility
by Justin Kala 22 Feb '16

22 Feb '16

Hi Syslog-ng is unable to recognize the facility audit. When I put filter as audit and restart syslog-ng it errors out. When I put the facility code as 13 ,it does not error on restarting the service but does not capture the syslog message received through this filter code 13 as well. Please advise. -- Kaladhar

3 10

Proper formatting of incoming Log4j, Jetty events
by Clayton Dukes 16 Dec '15

16 Dec '15

Hi, I'm having trouble with getting "good" fields from Jetty. I found this: http://blog.trifork.com/2010/01/14/logging-to-the-syslog-from-a-java-applic… And had my user configure their log4j with the given recommendation, but the events still come in quite poorly. Our syslog-ng template uses the following template: @${R_UNIXTIME:--}.${R_USEC}\t${HOST:--}\t${PRI:--}\t${PROGRAM:--}\t${MSGONLY}\n"); Some sample events: @1450275007.912996 host001 173 2015-12-16 16:10:07,836 DEBUG [HeartbeatGeneratorTimer] <-- sessionKey=179: out>{"method":"Heartbeat","jsonrpc":"2.0"} As you can see - this event is sending the date as the program name. Another: @1450275007.924140 host002 13 521 <174>1 2015-12-16T16:10:07+02:00 host002 /openam 1000430466 AUTHENTICATION-303 [logRecord@36733 LoginID="id=2565a24-0829-11e2-b614-001e371e7e40,ou=user,dc=opensso,dc=java,dc=net" ContextID="b1de62b91530d5303" IPAddr="10.3.109.13" LogLevel="INFO" Domain="dc=opensso,dc=java,dc=net" ModuleName="ProfileMapper|Mandate|LDAP" HostName="10.3.109.13" LoggedBy="cn=dsameuser,ou=DSAME Users,dc=opensso,dc=java,dc=net" MessageID="AUTHENTICATION-303" NameID="" TIME="2015-12-16 16:10:07"] Logout|service|mailService This one is sending the program name as an integer (521) There are many others, the program names consist of the following (out of about 100k events): - " .0.9"},"parentID" ...80) ame" at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) del" ...e.common.AuthenticationFilter.doFilter(AuthenticationFilter.java ...ee.elion.smarthome.common.AuthenticationFilter.doFilter(AuthenticationFilter.java ...eJspFile(JspServlet.java EST in ,"is_recording" iWeather"},{"id" leRecordID" me" model" ng" ngth" ...rg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java se,"is_recording" ....servlet.JspServlet.serviceJspFile(JspServlet.java tin Does anyone have a good way to fix this? ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 1

rabbitmq errors
by Al Itchon 15 Dec '15

15 Dec '15

I'm seeing an error in my log indicating that messages aren't being sent to a queue in RabbitMQ. This happens a few times a week. Here's the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren't sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that's limiting the messages?

4 8

Madhouse apt key is missing
by Clayton Dukes 09 Dec '15

09 Dec '15

Anyone know why https://packages.madhouse-project.org/debian/archive-key.txt is no longer available? ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 5

insider 2015-12: Elasticsearch & Kibana; SCL; Kafka; syslog-ng on Windows; conferences
by Czanik, Péter 03 Dec '15

03 Dec '15

Dear syslog-ng users, This is the 45th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS ESK: Elasticsearch + syslog-ng + Kibana --------------------------------------- An official Elasticsearch destination was a long awaited feature of syslog-ng. The Elasticsearch syslog-ng Kibana (ESK) stack provides high performance log collection, message parsing and filtering on the syslog-ng side and storage and analysis on the Elastic side. Read more about the benefits at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-s… and https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-a… The power of SCL ---------------- SCL, the syslog-ng configuration library is a lesser known feature of syslog-ng. It can provide reusable configuration blocks, which help to format messages to a specific format, hide complex configurations, This blog shows how to create a simple SCL through the example of a Loggly (logging as a service provider) destination: https://bazsi.blogs.balabit.com/2015/11/the-power-of-scl-integrating-with-l… Kafka and syslog-ng ------------------- Kafka is a high-throughput distributed messaging system. It was originally developed by LinkedIn as the backbone of a website activity tracking infrastructure and is now used by Netflix, Spotify, Twitter and many other companies for message queuing, log aggregation and stream processing. Read more about how syslog-ng can collect messages from multiple sources and process as well as filter them before forwarding them to Kafka: https://czanik.blogs.balabit.com/2015/11/kafka-and-syslog-ng/ syslog-ng on Windows -------------------- Most Linux admins only consider Windows as a client machine and use syslog-ng Agent for Windows if they need to collect logs from Windows. The following use cases demonstrate why you would want to use syslog-ng Premium Edition server on Windows instead of the syslog-ng Agent for Windows: https://czanik.blogs.balabit.com/2015/11/syslog-ng-on-windows/ syslog-ng @ conferences ----------------------- This autumn, syslog-ng was present at many conferences. I gave presentations about syslog-ng at the annual FSF Hungary and Virtualization & DevOps day conferences (in Hungarian), and participated SUSECon, LISA and Crunch Big Data Conference as an exhibitor. Most questions I received were related to the Elasticsearch destination and also many people were interested in commercial support for syslog-ng. In January I'll present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo: https://www.socallinuxexpo.org/scale/14x/presentations/syslog-ng-raw-data-b… See you there! NEW RELEASES syslog-ng PE 5F5: https://jluby.blogs.balabit.com/2015/11/16/syslog-ng-premium-edition-5f5-re… syslog-ng OSE 3.7.2: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

(no subject)
by Harsha S Aryan 01 Dec '15

01 Dec '15

hi, can any one let me know how to configure syslog-ng -- Rehards, Harsha

2 1

need help to configure
by Harsha S Aryan 01 Dec '15

01 Dec '15

hi, i am looking to configure syslog-ng in my system can any one let me know how to configure and add client (some linux servers and router) -- Rehards, Harsha

2 4