- syslog-ng - lists.balabit.hu

Ring buffered Log into file or Db
by Yiğit YAŞAR 13 Jun '19

13 Jun '19

Hi All, I got a limited system so i need ring buffered logging for protect disk space. I did some research but i could not found the answer. Does syslog-ng support ring buffer for logging to file or db ? Best Regards, Yigit

2 1

syslog-ng at Splunk Live in London
by Peter Czanik (pczanik) 12 Jun '19

12 Jun '19

Hi, The next Splunk Live event is tomorrow in London: https://live.splunk.com/london While there you are there you should also visit the syslog-ng booth and learn, how syslog-ng can provide you with a scalable log management layer in front of Splunk. Register for demo with my colleagues at https://www.syslog-ng.com/register/120335/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Need Assistance:error while loading shared libraries: libsyslog-ng-3.6.so.0
by Nitish Saboo 11 Jun '19

11 Jun '19

Hi, I am using syslog-ng-3.6.2 ( https://src.fedoraproject.org/lookaside/pkgs/syslog-ng/syslog-ng_3.6.2.tar.… ). I want to use syslog-ng header files in my project. I am calling syslog-ng code from my Go code. I compiled my source code in my local system ( go build -v -x main.go) and created a binary called 'main'. My binary is running successfully on my local system. When I copy the binary ('main') in another Linux box and try to run it as './main' it is giving me the following error: ubuntu@ingest-0693b91d:~$ ./main ./main: error while loading shared libraries: libsyslog-ng-3.6.so.0: cannot open shared object file: No such file or directory I have syslog-ng-3.6.2 shared files in the following location '/home/ubuntu/test/syslog/syslog-ng-3.6.2/install/lib/syslog-ng' and I have set LD_LIBRARY_PATH to {prefix}/lib/syslog-ng Can someone please guide me, why is it giving such error ? Thanks, Nitish

2 1

Generating configuration blocks for Option
by Sezer BAGLAN 23 May '19

23 May '19

Dear all, In Syslog-ng Administration Guide document there is a statement give as below: "The context option determines the type of the configuration snippet that the script generates, and must be one of the following: destination, filter , log, parser, rewrite, root, source. " >From this statement, we understand that we cannot use "option" type during auto generation of conf. blocks. Why do not we have "Option" type? How can we generate "option" blocks via scripts? Kind regards, Sezer

2 1

snmptrap handler parses SNMP traps correctly, but the file it reads from is not emptying
by Allen Pouratian 21 May '19

21 May '19

Hello - I set up snmptrapd to write to /var/log/snmptrapd.log and syslog-ng is configured to read from it like this ... source { snmptrap(filename("/var/log/snmptrapd.log")); }; ... but /var/log/snmptrapd.log is not getting emptied, so it needs to be truncated every so often to avoid filling up the filesystem. But the problem with truncating /var/log/snmptrapd.log is that we're going to lose traps, since we get a lot of them. When I was testing this syslog-ng snmptrap facility with a few traps a minute, I saw /var/log/snmptrapd.log empty, but with hundreds of traps incoming per second, it does not empty any more. I have what looks like a work-around to the syslog-ng snmptrap handler/parser where ... 1. 1) Snmptrapd 5.7.3 writes to syslog with -Lsd 2. 2) Syslog-ng 3.20 reads from syslog with system-journal() 3. 3) Rewrite the trap $MESSAGE with a series of "substitutions" (subst) into space separated key=value pairs 4. 4) point kv-parser() at $MESSAGE and specify " " as a separator 5. 5) delete the original $MESSAGE block ... but perhaps I didn't have to do that, since perhaps I'm mis-using the snmptrap facility built into syslog-ng, and thus causing /var/log/snmptrapd.log to not empty. Your comments and insights would be appreciated. - Allen -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Altice USA and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. --------------------------------------------------------

2 3

Forum upcoming on syslog-ng.com
by Hunor Voith (hvoith) 21 May '19

21 May '19

Dear syslog-ng users, The syslog-ng mailing list has been around for two decades to be a channel for newcomers, enthusiasts and developers to discuss their syslog-ng related topics. It gave place to many interesting and useful exchanges; however, it is also a semi-closed channel, lacking real public visibility. In this email, we would like to share some information about an upcoming forum on syslog-ng.com, and the future of this mailing list. We long wanted syslog-ng.com to be a sort of hub for everything syslog-ng related. A forum would be a great place to act as an ever-expanding knowledge base, complementing the extensive technical documentation of syslog-ng. Currently we have the forum up on the syslog-ng.com staging, and we would like to ask you to give it a spin and provide constructive feedback. To get access, please send an email to hunor.voith@<mailto:hunor.voith@oneidentity.com>oneidentity.com and I will share the access details with you. To pre-populate the forum, previous mailing list content has been (and will be, to an extent) migrated as threaded topics. We hope that the new platform will encourage more users to actively take part in syslog-ng related conversations. Of course, we will promote it once it is up and running to raise awareness among those who are currently not part of the mailing list community. We cannot share an exact go-live date with you yet, however, you can expect to see it on the live site in a couple weeks. We will keep you posted. What happens to the mailing list? The mailing list would be phased out after the forum goes live. The team is open to constructive feedback and there will no doubt be enhancements further down to road. -- Hunor Voith Product Marketing Manager @ syslog-ng

3 3

Fedora i386
by Czanik, Péter 21 May '19

21 May '19

Hi, Anyone still using syslog-ng on Fedora i386? I ran into some compile problems recently. If there is someone still using it, I can put some effort into a workaround (most likely disabling Java support on i386). If there is no positive response until the end of this week, I'll simply drop i386 platform support from my Copr repositories. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.21.1 RPM packages
by Czanik, Péter 21 May '19

21 May '19

Hi, The initial version of my syslog-ng 3.21.1 RPM packages for Fedora / RHEL 7 and openSUSE / SLES are now ready. Right now they are available in my git snapshot repositories. Learn how to install them from my blog at: https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-n… Unlike the development snapshots these should not eat your server for breakfast, but still they went through only minimal testing. This is especially true about the Riemann and Kafka destinations, where I packaged the latest versions of external dependencies to make sure that microsecond precision and Kafka are available. I don't have the test environment for them. Please test the packages and let me know not just if you ran into problems, but also your success stories. I'm still traveling. I plan to create a dedicated 3.21 repo once I'm back to Hungary and read some positive feedback from you! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Multiple log statements vs If/else
by Faine, Mark R. (MSFC-IS40)[NICS] 20 May '19

20 May '19

Is there any functional difference between something like: log { .... }; log { ... flags(final); }; and log { if { ... } else { ... }; I was thinking about trying to build a configuration specific for an app from a Jinja2 template in Ansible and it seems like to me that if they aren't different it would be easier to do multiple log statements if generated dynamically. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

4 10

Systemd service times out for syslog-ng 3.16.1
by Ronald Fenner 16 May '19

16 May '19

I'm trying to switch from an init.d script to a system service and having problems getting the service to run. The syslog-ng service gets timed out and never starts up. My service looks like this: [Unit] Description="Syslog-ng an enhanced syslog" Documentation=man:syslog-ng(8) After=network.target ConditionFileNotEmpty=/opt/syslog-ng/etc/syslog-ng.conf [Service] Type=notify ExecStart=/opt/syslog-ng/sbin/syslog-ng --no-caps -p /var/run/syslogd.pid -rdvt ExecReload=/bin/kill -HUP $MAINPID StandardOutput=journal StandardError=journal Restart=on-failure Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java/jre/lib/amd64/server:$LD_LIBRARY_PATH [Install] WantedBy=multi-user.target My config looks like this: ############################################################################# # Default syslog-ng.conf.partial file which collects all local logs into a # single file called /var/log/messages. # @version: 3.16 @module "mod-java" @include "scl.conf" @include "conf.d/gc-parsers.conf" source s_internal {internal();}; source s_rtl_stream { unix-stream("/var/gamecircus/log/rtl-stream.sock" flags(no-parse) log_msg_size(1000000)); }; # Backup for when syslog-ng crashes and doesn't restart queues the messages source s_rtl_backup { file("/var/gamecircus/log/rtl-backup.log" flags(no-parse) log_msg_size(1000000)); }; destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("syslog-ng-`DeployedEnvironment`") ); }; destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("${topic}.${environment}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; destination d_syslog_ng { file("/var/log/syslog-ng"); }; parser p_json { json-parser(); }; log { source(s_internal); destination(d_syslog_ng); }; log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); }; log { source(s_rtl_backup); parser(p_json); destination(d_kafka_structured); }; # Build script will add stuff below for the specific build. ############################################################################### The logging that I turned on in the service above isbn't helping. Here's the last little bit May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217091] Add path to classpath: /opt/kafka/libs/kafka-streams-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217248] Add path to classpath: /opt/kafka/libs/rocksdbjni-5.0.1.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217309] Add path to classpath: /opt/kafka/libs/kafka-streams-examples-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.361062] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365718] Running application hooks; hook='1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365930] Running application hooks; hook='3' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366137] syslog-ng starting up; version='3.16.1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366480] Worker thread started; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service start operation timed out. Terminating. May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273058] Running application hooks; hook='4' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273074] syslog-ng shutting down; version='3.16.1' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.376882] Worker thread finished; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377030] Closing log transport fd; fd='92' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377186] Java machine free; May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.378858] Running application hooks; hook='5' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Failed to start "Syslog-ng an enhanced syslog". May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Unit syslog-ng.service entered failed state. May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service failed. Any suggestions on what me causing it to timeout? I'm new to Ssytemd but if I understand the notify type it would suggest that syslog-ng isn't notifying systemd that it's started. I'm not sure what would be holding it up. Any help is much appreciated. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 5