June 2019 - syslog-ng - lists.balabit.hu

Missing messages
by Klimek, Bryan J. 16 Jul '19

16 Jul '19

First time poster, so be gentle. We run syslog-ng in our environment as a centralized syslog manager for the archiving of syslog data. We have over 2000+ Unix/Linux systems sending their syslog data with a daily ingest rate of about 10GB per day. It was recently pointed out to me that one particular message from one specific host is not getting persisted to the files on our syslog-ng server all the time. That is to say, it is intermittent. If one person can find one message that is not making into our syslog-ng archive, I can only assume that we are dropping other messages as well. How can I debug if and when messages are being lost and not making into the files on my centralized syslog server? Bryan Klimek Mayo Clinic

4 10

Periodic "Destination timed out, reaping" and disconnection on a write fifo (pipe)
by Charles Eidsness 27 Jun '19

27 Jun '19

I kind of feel like this is going to be a dumb question but I just can't figure this out. I'm trying to get syslog-ng working with sagan, but every 60 seconds syslog-ng closes the fifo it's writing to, which closes the sagan connection to the fifo. When running in verbose mode syslog-ng is throwing up a "Destination timed out, reaping..." message. I simplified my syslog-ng configuration to: source s_sys { system(); }; destination d_fifo { pipe( "/tmp/test.fifo" ); }; log { source(s_sys); destination(d_fifo); }; And then run use cat to read the fifo: cat /tmp/test.fifo It sends all of the log traffic every but every 60 seconds cat closes. Is this expected behavior? Am I missing something? I'm using syslog-ng version 3.19.1. Thanks! Charles == Charles Eidsness Chief Engineer *CCX Technologies* (613)791-9715

2 7

Sending Messages to Slack via a HTTP Proxy
by Frank Crawford 26 Jun '19

26 Jun '19

We have recently upgraded to syslog-ng v3.19 so we can use the slack hook, but have an issue that to access the URL we will need to go through a squid web proxy. How do we configure the details of the proxy in either syslog-ng or in the slack destination? Regards Frank -- [http://www.ac3.com.au/emailsigs/v2/ac3_logo.png]<http://www.ac3.com.au> the human touch in a technical world Frank Crawford Technology and Infrastructure Manager t 02 9199 0995 | m 0401 714 870 e frank.crawford(a)ac3.com.au<mailto:frank.crawford@ac3.com.au> Level 8 East, 8 Central Ave, Eveleigh NSW 2015 [http://www.ac3.com.au/emailsigs/v2/linkedin.png] <https://www.linkedin.com/company/ac3> [http://www.ac3.com.au/emailsigs/v2/twitter.png]<https://twitter.com/ac3ptyltd> [http://www.ac3.com.au/emailsigs/v2/iso.png]<http://www.ac3.com.au/about-us/news/double-iso-certification-ac3> This email is for intended recipients only. If you are not an intended recipient, you must not disclose, copy or distribute this email to others or act upon the information contained in this email in any way. This email may contain privileged or confidential information. If you have received this email in error, please notify the sender and then delete it from your system.

2 4

error on syslog-ng elasticsearch2 destination
by joao reis 26 Jun '19

26 Jun '19

Hi, I'm getting the following error when trying to configure the elasticseach2 backend: Error parsing destination statement, destination plugin java not found in block destination elasticsearch2() at /usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf:25:4:3-4:7: 1 2 #Start Block block destination elasticsearch2() at /usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf:25 3 4-----> java( 4-----> ^^^^ 5 class_path("//usr/lib64/syslog-ng/java-modules/*.jar://usr/lib64/syslog-ng/java-modules/elastic-jest-client/*.jar:/*.jar") 6 class_name("org.syslog_ng.elasticsearch_v2.ElasticSearchDestination") 7 option("index", "logserver1") 8 option("type", "syslog") 9 option("server", "localhost") This is the block that causes the error: destination d_es { elasticsearch2 ( cluster("logserver1") client-mode("http") index("logserver1") type("syslog") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ); }; root@elk-server1 conf.d]# rpm -qa|grep syslog-ng syslog-ng-3.21.1-3.el7.x86_64 syslog-ng-java-3.21.1-3.el7.x86_64 syslog-ng-http-3.21.1-3.el7.x86_64 Any ideia? Thanks in advance.

1 1

syslog-ng-3.22: unofficial Debian/Ubuntu packages
by Laszlo Budai 25 Jun '19

25 Jun '19

Hi, syslog-ng 3.22.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.22.1
by Laszlo Budai 25 Jun '19

25 Jun '19

3.22.1 Highlights * Sending SNMP traps: Using the new snmp() destination, incoming log messages can be converted to SNMP traps, as the fields of the SNMP messages can be customized with macros. (#2693<https://github.com/balabit/syslog-ng/pull/2693>) * $(template) dynamic binding: Extends the $(template) template function to allow dynamic binding. For example, the name of the template to be invoked can come from the message (name-value pairs). (#2716<https://github.com/balabit/syslog-ng/pull/2716>) * syslog(), network(): Add dynamic-window-size() option to enable dynamic flow control that distributes the specified amount of window between active connections at runtime. This can be used in low-memory environments, where only a small subset of the active clients sends messages at high rate. (#2772<https://github.com/balabit/syslog-ng/pull/2772>) Features * match(): Add support for the template() option (#2715<https://github.com/balabit/syslog-ng/pull/2715>) * add-contextual-data(): Allow using templates in name-value pairs (#2711<https://github.com/balabit/syslog-ng/pull/2711>) * Add support for floating point operations in template functions (#2742<https://github.com/balabit/syslog-ng/pull/2742>) * Add support for usec precision when parsing time (#2709<https://github.com/balabit/syslog-ng/pull/2709>) Bugfixes * Fix null pointer access when destinations are suspended (#2778<https://github.com/balabit/syslog-ng/pull/2778>) * Fix grouping-by() deadlock (#2758<https://github.com/balabit/syslog-ng/pull/2758>) * Fix a general source-related crash and enhance wildcard-file()'s bookmark handling (#2589<https://github.com/balabit/syslog-ng/pull/2589>) * Fix infinite loop (reload/reopen) (#2739<https://github.com/balabit/syslog-ng/pull/2739>) * Fix python() package/module name collision (#2438<https://github.com/balabit/syslog-ng/pull/2438>) * Fix escaped quote in block argument (#2781<https://github.com/balabit/syslog-ng/pull/2781>) * Reintroduce test on SYSLOG_NG_HAVE_TIMEZONE (#2774<https://github.com/balabit/syslog-ng/pull/2774>) * snmp(): Fix template leak (#2746<https://github.com/balabit/syslog-ng/pull/2746>) Other changes * Never drop flow-controlled messages: The meaning of log-fifo-size() has changed to avoid dropping flow-controlled messages when log-fifo-size() is misconfigured. From now on, log-fifo-size() only affects messages that are not flow-controlled. (#2753<https://github.com/balabit/syslog-ng/pull/2753>) * The -d/--debug syslog-ng command line flag no longer implies -e/--stderr. If you want to redirect internal() source to stderr, use the -e/--stderr option explicitly. (#2731<https://github.com/balabit/syslog-ng/pull/2731>) * dbld, RPM and DEB packaging improvements (#2724<https://github.com/balabit/syslog-ng/pull/2724>) * Checkpoint parser improvements (#2740<https://github.com/balabit/syslog-ng/pull/2740>) * Reset the timezone on config reload event (#2691<https://github.com/balabit/syslog-ng/pull/2691>) * geoip2(): Include IP into the error message (#2743<https://github.com/balabit/syslog-ng/pull/2743>) * Improve regexp error messages (#2796<https://github.com/balabit/syslog-ng/pull/2796>) * http(): Warn if less workers used than urls (#2757<https://github.com/balabit/syslog-ng/pull/2757>) * http(): Allow URLs to be specified by a space/comma separated string (#2699<https://github.com/balabit/syslog-ng/pull/2699>) * loggen: Change message rate at runtime using signals (#2756<https://github.com/balabit/syslog-ng/pull/2756>) * debun: add acquire_running_syslog_config function (#2752<https://github.com/balabit/syslog-ng/pull/2752>) * FreeBSD fixes for the test suite (#2783<https://github.com/balabit/syslog-ng/pull/2783>) Notes to the developers * ivykis: update to 0.42.4 (#2736<https://github.com/balabit/syslog-ng/pull/2736>) * Support generator plugins in global options (#2747<https://github.com/balabit/syslog-ng/pull/2747>) * logthrfetcher: new constants (#2766<https://github.com/balabit/syslog-ng/pull/2766>) * logthrsourcedrv: support position tracking (#2750<https://github.com/balabit/syslog-ng/pull/2750>) * Light: Support pre-commit and tox (#2725<https://github.com/balabit/syslog-ng/pull/2725>) * Enable Bison error flags: conflicts-sr/rr (#2762<https://github.com/balabit/syslog-ng/pull/2762>) * Dynamic stats constant registration (#2761<https://github.com/balabit/syslog-ng/pull/2761>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Christian Michallek, Fabien Wernli, Gabor Nagy, Kyeong Yoo, Laszlo Budai, Laszlo Szemere, László Várady, Mehul Prajapati, Norbert Takacs, Oleksii Hamov, Péter Kókai, Romain Tartière, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. - balabit/syslog-ng github.com

1 0

Re: [syslog-ng] grouping-by question
by Газин Максим Алексеевич 20 Jun '19

20 Jun '19

bypassed the problem in this way $(- $(context-length) 1) And one more question: is it possible to use the context-length instead of a timeout as the context closure value? -----Original Message----- From: Fabien Wernli [mailto:wernli@in2p3.fr] Sent: Thursday, June 20, 2019 12:25 PM To: Газин Максим Алексеевич <Maksim.Gazin(a)rt.ru> Cc: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: RE: [syslog-ng] grouping-by question Hi, On Thu, Jun 20, 2019 at 07:17:28AM +0000, Газин Максим Алексеевич wrote: > what does it mean? Is it possible to make the context-length equal to the value of the aggregated messages? In your original email, your config indicated that you were using `timeout(15)` in the grouping-by config. As far as I understand the grouping-by parser, this means that the context will only be closed (and thus context-len reinitialized) if no message matches the condition for 15 seconds.

3 2

Insider 2019-06: Python; Google Stackdriver; elasticsearch-http(); a year of syslog-ng; Red Hat Summit;
by Peter Czanik (pczanik) 20 Jun '19

20 Jun '19

Dear syslog-ng users, This is the 74th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Format your log messages in Python ---------------------------------- Sometimes getting log messages into the desired format can be a problem, but with syslog-ng you can use Python to get the exact format you need. The syslog-ng Python template function allows you to write custom templates for syslog-ng in Python. In this blog post, I will show you a simple use of the Python parser: resolving IP addresses to host names. I will also show you the logger method, a nice new feature that enables you to log to syslog-ng’s internal() log source instead of writing logs from Python to stdout. This way you can follow what your Python code is doing even if syslog-ng is running as a daemon in the background. https://www.syslog-ng.com/community/b/blog/posts/format-your-log-messages-i… Sending logs to Google Stackdriver using syslog-ng PE ----------------------------------------------------- Google Stackdriver collects and analyses logs, events and metrics of your infrastructure. Using syslog-ng PE 7.0.14 or later versions, you can send your logs to Google Stackdriver. While originally designed to quickly respond to events in the Google Cloud Platform (GCP), you can use Google Stackdriver with any other cloud providers (for example, Amazon Web Services) or on-premises data as well. This way you can view events of a hybrid infrastructure at a single location. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-google-sta… How to forward logs to Elasticsearch using the elasticsearch-http() destination in syslog-ng ------------------------------------------------------------- Starting with syslog-ng 3.21.1, you can use the elasticsearch-http() destination (based on the http() destination) to send logs to Elasticsearch. In most cases it is more efficient than the old Java-based Elasticsearch destination, and it is definitely easier to configure. https://www.syslog-ng.com/community/b/blog/posts/how-to-forward-logs-to-ela… A year of syslog-ng ------------------- With about two thirds of syslog-ng users running their logging application on Red Hat Enterprise Linux (or CentOS), it is one of our most important platforms both for the open source edition (OSE) and the premium edition (PE) of syslog-ng. Preparing for the annual Red Hat Summit, I made a list of major new features in syslog-ng since last year’s event. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-is-coming-to-red… Sudo + syslog-ng: two softwares at two conferences -------------------------------------------------- I visited two conferences recently: LOADays and Red Hat Summit. Both conferences focus on open source softwares, but there the similarity ends. LOADays in Antwerp is small, free and focuses on Linux administrators. The Red Hat Summit in Boston is huge, expensive and covers a wide variety of topics, including administration (among many others). Regardless of the differences, both events are among my favorites. https://www.syslog-ng.com/community/b/blog/posts/sudo-syslog-ng-two-softwar… CONFERENCES * Pass the SALT 2019 (https://2019.pass-the-salt.org/) syslog-ng workshop * All Things Open (https://allthingsopen.org/) WEBINARS * How to use the syslog-ng PE’s new Google Stackdriver destination https://www.syslog-ng.com/event/live-webinar-how-to-use-the-syslogng-pes-ne… Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

grouping-by question
by Газин Максим Алексеевич 20 Jun '19

20 Jun '19

Hello, I'm new to syslog-ng, and I'm having some trouble. I want to use syslog-ng to aggregate similar events from our firewalls. I use the grouping-by parser to create an aggregated event, but I have a question, how do I get only aggregated events on the output without source ones? My configuration: parser p_patterndb { db-parser(file("/etc/syslog-ng/conf.d/forti.xml")); }; log { source { file("/tmp/in.log" flags(no-parse)); }; parser { kv_parser (prefix("forti.")); grouping-by( key("${forti.devname}${forti.srcip}${forti.dstip}${forti.dstport}") scope("process") timeout(15) aggregate( value("MESSAGE" "aggegated devname=${forti.devname} srcip=${forti.srcip} dstip=${forti.dstip} dstport=${forti.dstport} count=$(context-length)") inherit-mode("context") ) ); }; destination { file("tmp/out.log"); }; }; My current output: Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="fw01" devid=" " vd="" date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="-fw01" devid="" vd=" " date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 What I want: Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 Any guidance would be very much appreciated. Maxim Gazin | Senior security engineer | Rostelecom

3 5

An External Application For Parsing Existing Syslog-ng Log File
by Sezer BAGLAN 18 Jun '19

18 Jun '19

Dear all, Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below: ${ISODATE} ${HOST} ${FACILITY} ${LEVEL} ${LEVEL_NUM} ${TAGS} ${MESSAGE} 1) What is your opinion about the best practices to handle this issue? Please note that I'm not talking about logging messages with filters and parsers. I'm talking about parsing an existing syslog-ng log file. 2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job? 3) Do you suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on? Thanks in advance. Kind regards, Sezer

2 1