May 2019 - syslog-ng - lists.balabit.hu

Generating configuration blocks for Option
by Sezer BAGLAN 23 May '19

23 May '19

Dear all, In Syslog-ng Administration Guide document there is a statement give as below: "The context option determines the type of the configuration snippet that the script generates, and must be one of the following: destination, filter , log, parser, rewrite, root, source. " >From this statement, we understand that we cannot use "option" type during auto generation of conf. blocks. Why do not we have "Option" type? How can we generate "option" blocks via scripts? Kind regards, Sezer

2 1

snmptrap handler parses SNMP traps correctly, but the file it reads from is not emptying
by Allen Pouratian 21 May '19

21 May '19

Hello - I set up snmptrapd to write to /var/log/snmptrapd.log and syslog-ng is configured to read from it like this ... source { snmptrap(filename("/var/log/snmptrapd.log")); }; ... but /var/log/snmptrapd.log is not getting emptied, so it needs to be truncated every so often to avoid filling up the filesystem. But the problem with truncating /var/log/snmptrapd.log is that we're going to lose traps, since we get a lot of them. When I was testing this syslog-ng snmptrap facility with a few traps a minute, I saw /var/log/snmptrapd.log empty, but with hundreds of traps incoming per second, it does not empty any more. I have what looks like a work-around to the syslog-ng snmptrap handler/parser where ... 1. 1) Snmptrapd 5.7.3 writes to syslog with -Lsd 2. 2) Syslog-ng 3.20 reads from syslog with system-journal() 3. 3) Rewrite the trap $MESSAGE with a series of "substitutions" (subst) into space separated key=value pairs 4. 4) point kv-parser() at $MESSAGE and specify " " as a separator 5. 5) delete the original $MESSAGE block ... but perhaps I didn't have to do that, since perhaps I'm mis-using the snmptrap facility built into syslog-ng, and thus causing /var/log/snmptrapd.log to not empty. Your comments and insights would be appreciated. - Allen -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Altice USA and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. --------------------------------------------------------

2 3

Forum upcoming on syslog-ng.com
by Hunor Voith (hvoith) 21 May '19

21 May '19

Dear syslog-ng users, The syslog-ng mailing list has been around for two decades to be a channel for newcomers, enthusiasts and developers to discuss their syslog-ng related topics. It gave place to many interesting and useful exchanges; however, it is also a semi-closed channel, lacking real public visibility. In this email, we would like to share some information about an upcoming forum on syslog-ng.com, and the future of this mailing list. We long wanted syslog-ng.com to be a sort of hub for everything syslog-ng related. A forum would be a great place to act as an ever-expanding knowledge base, complementing the extensive technical documentation of syslog-ng. Currently we have the forum up on the syslog-ng.com staging, and we would like to ask you to give it a spin and provide constructive feedback. To get access, please send an email to hunor.voith@<mailto:hunor.voith@oneidentity.com>oneidentity.com and I will share the access details with you. To pre-populate the forum, previous mailing list content has been (and will be, to an extent) migrated as threaded topics. We hope that the new platform will encourage more users to actively take part in syslog-ng related conversations. Of course, we will promote it once it is up and running to raise awareness among those who are currently not part of the mailing list community. We cannot share an exact go-live date with you yet, however, you can expect to see it on the live site in a couple weeks. We will keep you posted. What happens to the mailing list? The mailing list would be phased out after the forum goes live. The team is open to constructive feedback and there will no doubt be enhancements further down to road. -- Hunor Voith Product Marketing Manager @ syslog-ng

3 3

Fedora i386
by Czanik, Péter 21 May '19

21 May '19

Hi, Anyone still using syslog-ng on Fedora i386? I ran into some compile problems recently. If there is someone still using it, I can put some effort into a workaround (most likely disabling Java support on i386). If there is no positive response until the end of this week, I'll simply drop i386 platform support from my Copr repositories. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.21.1 RPM packages
by Czanik, Péter 21 May '19

21 May '19

Hi, The initial version of my syslog-ng 3.21.1 RPM packages for Fedora / RHEL 7 and openSUSE / SLES are now ready. Right now they are available in my git snapshot repositories. Learn how to install them from my blog at: https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-n… Unlike the development snapshots these should not eat your server for breakfast, but still they went through only minimal testing. This is especially true about the Riemann and Kafka destinations, where I packaged the latest versions of external dependencies to make sure that microsecond precision and Kafka are available. I don't have the test environment for them. Please test the packages and let me know not just if you ran into problems, but also your success stories. I'm still traveling. I plan to create a dedicated 3.21 repo once I'm back to Hungary and read some positive feedback from you! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Multiple log statements vs If/else
by Faine, Mark R. (MSFC-IS40)[NICS] 20 May '19

20 May '19

Is there any functional difference between something like: log { .... }; log { ... flags(final); }; and log { if { ... } else { ... }; I was thinking about trying to build a configuration specific for an app from a Jinja2 template in Ansible and it seems like to me that if they aren't different it would be easier to do multiple log statements if generated dynamically. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

4 10

Systemd service times out for syslog-ng 3.16.1
by Ronald Fenner 16 May '19

16 May '19

I'm trying to switch from an init.d script to a system service and having problems getting the service to run. The syslog-ng service gets timed out and never starts up. My service looks like this: [Unit] Description="Syslog-ng an enhanced syslog" Documentation=man:syslog-ng(8) After=network.target ConditionFileNotEmpty=/opt/syslog-ng/etc/syslog-ng.conf [Service] Type=notify ExecStart=/opt/syslog-ng/sbin/syslog-ng --no-caps -p /var/run/syslogd.pid -rdvt ExecReload=/bin/kill -HUP $MAINPID StandardOutput=journal StandardError=journal Restart=on-failure Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java/jre/lib/amd64/server:$LD_LIBRARY_PATH [Install] WantedBy=multi-user.target My config looks like this: ############################################################################# # Default syslog-ng.conf.partial file which collects all local logs into a # single file called /var/log/messages. # @version: 3.16 @module "mod-java" @include "scl.conf" @include "conf.d/gc-parsers.conf" source s_internal {internal();}; source s_rtl_stream { unix-stream("/var/gamecircus/log/rtl-stream.sock" flags(no-parse) log_msg_size(1000000)); }; # Backup for when syslog-ng crashes and doesn't restart queues the messages source s_rtl_backup { file("/var/gamecircus/log/rtl-backup.log" flags(no-parse) log_msg_size(1000000)); }; destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("syslog-ng-`DeployedEnvironment`") ); }; destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("${topic}.${environment}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; destination d_syslog_ng { file("/var/log/syslog-ng"); }; parser p_json { json-parser(); }; log { source(s_internal); destination(d_syslog_ng); }; log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); }; log { source(s_rtl_backup); parser(p_json); destination(d_kafka_structured); }; # Build script will add stuff below for the specific build. ############################################################################### The logging that I turned on in the service above isbn't helping. Here's the last little bit May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217091] Add path to classpath: /opt/kafka/libs/kafka-streams-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217248] Add path to classpath: /opt/kafka/libs/rocksdbjni-5.0.1.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217309] Add path to classpath: /opt/kafka/libs/kafka-streams-examples-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.361062] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365718] Running application hooks; hook='1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365930] Running application hooks; hook='3' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366137] syslog-ng starting up; version='3.16.1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366480] Worker thread started; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service start operation timed out. Terminating. May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273058] Running application hooks; hook='4' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273074] syslog-ng shutting down; version='3.16.1' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.376882] Worker thread finished; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377030] Closing log transport fd; fd='92' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377186] Java machine free; May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.378858] Running application hooks; hook='5' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Failed to start "Syslog-ng an enhanced syslog". May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Unit syslog-ng.service entered failed state. May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service failed. Any suggestions on what me causing it to timeout? I'm new to Ssytemd but if I understand the notify type it would suggest that syslog-ng isn't notifying systemd that it's started. I'm not sure what would be holding it up. Any help is much appreciated. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 5

Rest2Syslog
by C B 15 May '19

15 May '19

hello all, just wanted to share with you a new python syslog-ng plugin I was working on to support ingestion of any REST API into any syslog destination. this is off course open source and available here - https://github.com/chenbekor/Rest2Syslog I'll be happy to receive any feedback/suggestion. Also - if any of you guys know of a public page with all available plugins/extensions please let me know so I can add this nice project to the list. Regards, Chen.

2 1

Re: [syslog-ng] Troubleshooting Question
by Walter Tienken 14 May '19

14 May '19

Hi, Yes, the em1 interface is for our front-end connectivity and the em2 interface is for the F5 LB where traffic is sent through. By dual stack, I assume you mean IPV4/IPV6 ... in that case, no, only IPV4. It looks like syslog-ng is listening on all interfaces: Here is the output of netstat -ulp: https://gist.github.com/MrTink76/ade770cfe6d497aa50fe820a9c438f54 The part that's throwing me off is "distinct" ... shouldn't that be 9999? You see it is listening on all interfaces and the netstat -anp | grep 9999 here as well: https://gist.github.com/MrTink76/7741d90831fe4f331abe4630f3a8b29e Thanks much, Walter Tienken walter.tienken(a)asu.edu On 5/12/19, 8:56 AM, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: Does your box have multiple interfaces? Does it run a dual-stack? If yes, does the program (nc or syslogng) listen on the correct interface? (netstat -ulp)

2 1

How to create two independent patternDB instances in syslog-ng
by Nitish Saboo 13 May '19

13 May '19

I understand PatternDB is a syslog-ng pluginand its dynamic library can be found in as libdbparser.so. I went through the following tests in ' https://github.com/balabit/syslog-ng/blob/master/modules/dbparser/patterndb… ' I want to create two independent pattern_db instances having two different ruleset. My queries: 1) How can I create two independent instances of patternDB having two different ruleset ? 2)How can I get the handle of these two independent PatternDB instances ? Can someone please help me on this ? Thanks

1 0