- syslog-ng - lists.balabit.hu

Re: [syslog-ng] grouping-by question
by Газин Максим Алексеевич 20 Jun '19

20 Jun '19

bypassed the problem in this way $(- $(context-length) 1) And one more question: is it possible to use the context-length instead of a timeout as the context closure value? -----Original Message----- From: Fabien Wernli [mailto:wernli@in2p3.fr] Sent: Thursday, June 20, 2019 12:25 PM To: Газин Максим Алексеевич <Maksim.Gazin(a)rt.ru> Cc: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: RE: [syslog-ng] grouping-by question Hi, On Thu, Jun 20, 2019 at 07:17:28AM +0000, Газин Максим Алексеевич wrote: > what does it mean? Is it possible to make the context-length equal to the value of the aggregated messages? In your original email, your config indicated that you were using `timeout(15)` in the grouping-by config. As far as I understand the grouping-by parser, this means that the context will only be closed (and thus context-len reinitialized) if no message matches the condition for 15 seconds.

3 2

Insider 2019-06: Python; Google Stackdriver; elasticsearch-http(); a year of syslog-ng; Red Hat Summit;
by Peter Czanik (pczanik) 20 Jun '19

20 Jun '19

Dear syslog-ng users, This is the 74th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Format your log messages in Python ---------------------------------- Sometimes getting log messages into the desired format can be a problem, but with syslog-ng you can use Python to get the exact format you need. The syslog-ng Python template function allows you to write custom templates for syslog-ng in Python. In this blog post, I will show you a simple use of the Python parser: resolving IP addresses to host names. I will also show you the logger method, a nice new feature that enables you to log to syslog-ng’s internal() log source instead of writing logs from Python to stdout. This way you can follow what your Python code is doing even if syslog-ng is running as a daemon in the background. https://www.syslog-ng.com/community/b/blog/posts/format-your-log-messages-i… Sending logs to Google Stackdriver using syslog-ng PE ----------------------------------------------------- Google Stackdriver collects and analyses logs, events and metrics of your infrastructure. Using syslog-ng PE 7.0.14 or later versions, you can send your logs to Google Stackdriver. While originally designed to quickly respond to events in the Google Cloud Platform (GCP), you can use Google Stackdriver with any other cloud providers (for example, Amazon Web Services) or on-premises data as well. This way you can view events of a hybrid infrastructure at a single location. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-google-sta… How to forward logs to Elasticsearch using the elasticsearch-http() destination in syslog-ng ------------------------------------------------------------- Starting with syslog-ng 3.21.1, you can use the elasticsearch-http() destination (based on the http() destination) to send logs to Elasticsearch. In most cases it is more efficient than the old Java-based Elasticsearch destination, and it is definitely easier to configure. https://www.syslog-ng.com/community/b/blog/posts/how-to-forward-logs-to-ela… A year of syslog-ng ------------------- With about two thirds of syslog-ng users running their logging application on Red Hat Enterprise Linux (or CentOS), it is one of our most important platforms both for the open source edition (OSE) and the premium edition (PE) of syslog-ng. Preparing for the annual Red Hat Summit, I made a list of major new features in syslog-ng since last year’s event. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-is-coming-to-red… Sudo + syslog-ng: two softwares at two conferences -------------------------------------------------- I visited two conferences recently: LOADays and Red Hat Summit. Both conferences focus on open source softwares, but there the similarity ends. LOADays in Antwerp is small, free and focuses on Linux administrators. The Red Hat Summit in Boston is huge, expensive and covers a wide variety of topics, including administration (among many others). Regardless of the differences, both events are among my favorites. https://www.syslog-ng.com/community/b/blog/posts/sudo-syslog-ng-two-softwar… CONFERENCES * Pass the SALT 2019 (https://2019.pass-the-salt.org/) syslog-ng workshop * All Things Open (https://allthingsopen.org/) WEBINARS * How to use the syslog-ng PE’s new Google Stackdriver destination https://www.syslog-ng.com/event/live-webinar-how-to-use-the-syslogng-pes-ne… Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

grouping-by question
by Газин Максим Алексеевич 20 Jun '19

20 Jun '19

Hello, I'm new to syslog-ng, and I'm having some trouble. I want to use syslog-ng to aggregate similar events from our firewalls. I use the grouping-by parser to create an aggregated event, but I have a question, how do I get only aggregated events on the output without source ones? My configuration: parser p_patterndb { db-parser(file("/etc/syslog-ng/conf.d/forti.xml")); }; log { source { file("/tmp/in.log" flags(no-parse)); }; parser { kv_parser (prefix("forti.")); grouping-by( key("${forti.devname}${forti.srcip}${forti.dstip}${forti.dstport}") scope("process") timeout(15) aggregate( value("MESSAGE" "aggegated devname=${forti.devname} srcip=${forti.srcip} dstip=${forti.dstip} dstport=${forti.dstport} count=$(context-length)") inherit-mode("context") ) ); }; destination { file("tmp/out.log"); }; }; My current output: Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="fw01" devid=" " vd="" date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="-fw01" devid="" vd=" " date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 What I want: Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 Any guidance would be very much appreciated. Maxim Gazin | Senior security engineer | Rostelecom

3 5

An External Application For Parsing Existing Syslog-ng Log File
by Sezer BAGLAN 18 Jun '19

18 Jun '19

Dear all, Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below: ${ISODATE} ${HOST} ${FACILITY} ${LEVEL} ${LEVEL_NUM} ${TAGS} ${MESSAGE} 1) What is your opinion about the best practices to handle this issue? Please note that I'm not talking about logging messages with filters and parsers. I'm talking about parsing an existing syslog-ng log file. 2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job? 3) Do you suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on? Thanks in advance. Kind regards, Sezer

2 1

Difficulty with Elasticsearch Destination
by Kelly Marchewa 16 Jun '19

16 Jun '19

Greetings all: I recently installed syslog-ng on my Debian 9 system. I have tested syslog-ng with a file destination and everything seemed fine. Now, I am attempting?? to set the destination to my local Elasticsearch instance. However, I am receiving the following error: /Error parsing destination, destination plugin elasticsearch-http not found in /etc/syslog-ng/syslog-ng.conf at line 44, column 1:/ /elasticsearch-http(/ /^^^^^^^^^^^^^^^^^^/ My configuration, based on the documentation: /elasticsearch-http(////?? index("syslog")////?? type("test")////?? url("http:/localhost:9200/_bulk"));////};/ Output of syslog-ng --version: /syslog-ng 3.8.1////Installer-Version: 3.8.1////Revision: 3.8.1-10////Module-Directory: /usr/lib/syslog-ng/3.8////Module-Path: /usr/lib/syslog-ng/3.8////Available-Modules: disk-buffer,redis,date,afstomp,afprog,afsocket,pseudofile,confgen,csvparser,afsql,kvformat,geoip-plugin,afuser,afsmtp,system-source,mod-python,afamqp,riemann,linux-kmsg-format,dbparser,basicfuncs,syslogformat,graphite,afmongodb,json-plugin,cryptofuncs,affile,sdjournal,cef,add-contextual-data////Enable-Debug: off////Enable-GProf: off////Enable-Memtrace: off////Enable-IPv6: on////Enable-Spoof-Source: on////Enable-TCP-Wrapper: on////Enable-Linux-Caps: off/ I saw no indication in the documentation that any special modules/plugins needed to be loaded to use Elasticsearch as the destination. Any pointers as to what the issue is would be appreciated. Thanks! -- Kelly Marchewa kelly_marchewa(a)yahoo.com

2 1

Ring buffered Log into file or Db
by Yiğit YAŞAR 13 Jun '19

13 Jun '19

Hi All, I got a limited system so i need ring buffered logging for protect disk space. I did some research but i could not found the answer. Does syslog-ng support ring buffer for logging to file or db ? Best Regards, Yigit

2 1

syslog-ng at Splunk Live in London
by Peter Czanik (pczanik) 12 Jun '19

12 Jun '19

Hi, The next Splunk Live event is tomorrow in London: https://live.splunk.com/london While there you are there you should also visit the syslog-ng booth and learn, how syslog-ng can provide you with a scalable log management layer in front of Splunk. Register for demo with my colleagues at https://www.syslog-ng.com/register/120335/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Need Assistance:error while loading shared libraries: libsyslog-ng-3.6.so.0
by Nitish Saboo 11 Jun '19

11 Jun '19

Hi, I am using syslog-ng-3.6.2 ( https://src.fedoraproject.org/lookaside/pkgs/syslog-ng/syslog-ng_3.6.2.tar.… ). I want to use syslog-ng header files in my project. I am calling syslog-ng code from my Go code. I compiled my source code in my local system ( go build -v -x main.go) and created a binary called 'main'. My binary is running successfully on my local system. When I copy the binary ('main') in another Linux box and try to run it as './main' it is giving me the following error: ubuntu@ingest-0693b91d:~$ ./main ./main: error while loading shared libraries: libsyslog-ng-3.6.so.0: cannot open shared object file: No such file or directory I have syslog-ng-3.6.2 shared files in the following location '/home/ubuntu/test/syslog/syslog-ng-3.6.2/install/lib/syslog-ng' and I have set LD_LIBRARY_PATH to {prefix}/lib/syslog-ng Can someone please guide me, why is it giving such error ? Thanks, Nitish

2 1

Generating configuration blocks for Option
by Sezer BAGLAN 23 May '19

23 May '19

Dear all, In Syslog-ng Administration Guide document there is a statement give as below: "The context option determines the type of the configuration snippet that the script generates, and must be one of the following: destination, filter , log, parser, rewrite, root, source. " >From this statement, we understand that we cannot use "option" type during auto generation of conf. blocks. Why do not we have "Option" type? How can we generate "option" blocks via scripts? Kind regards, Sezer

2 1

snmptrap handler parses SNMP traps correctly, but the file it reads from is not emptying
by Allen Pouratian 21 May '19

21 May '19

Hello - I set up snmptrapd to write to /var/log/snmptrapd.log and syslog-ng is configured to read from it like this ... source { snmptrap(filename("/var/log/snmptrapd.log")); }; ... but /var/log/snmptrapd.log is not getting emptied, so it needs to be truncated every so often to avoid filling up the filesystem. But the problem with truncating /var/log/snmptrapd.log is that we're going to lose traps, since we get a lot of them. When I was testing this syslog-ng snmptrap facility with a few traps a minute, I saw /var/log/snmptrapd.log empty, but with hundreds of traps incoming per second, it does not empty any more. I have what looks like a work-around to the syslog-ng snmptrap handler/parser where ... 1. 1) Snmptrapd 5.7.3 writes to syslog with -Lsd 2. 2) Syslog-ng 3.20 reads from syslog with system-journal() 3. 3) Rewrite the trap $MESSAGE with a series of "substitutions" (subst) into space separated key=value pairs 4. 4) point kv-parser() at $MESSAGE and specify " " as a separator 5. 5) delete the original $MESSAGE block ... but perhaps I didn't have to do that, since perhaps I'm mis-using the snmptrap facility built into syslog-ng, and thus causing /var/log/snmptrapd.log to not empty. Your comments and insights would be appreciated. - Allen -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Altice USA and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. --------------------------------------------------------

2 3

Forum upcoming on syslog-ng.com
by Hunor Voith (hvoith) 21 May '19

21 May '19

Dear syslog-ng users, The syslog-ng mailing list has been around for two decades to be a channel for newcomers, enthusiasts and developers to discuss their syslog-ng related topics. It gave place to many interesting and useful exchanges; however, it is also a semi-closed channel, lacking real public visibility. In this email, we would like to share some information about an upcoming forum on syslog-ng.com, and the future of this mailing list. We long wanted syslog-ng.com to be a sort of hub for everything syslog-ng related. A forum would be a great place to act as an ever-expanding knowledge base, complementing the extensive technical documentation of syslog-ng. Currently we have the forum up on the syslog-ng.com staging, and we would like to ask you to give it a spin and provide constructive feedback. To get access, please send an email to hunor.voith@<mailto:hunor.voith@oneidentity.com>oneidentity.com and I will share the access details with you. To pre-populate the forum, previous mailing list content has been (and will be, to an extent) migrated as threaded topics. We hope that the new platform will encourage more users to actively take part in syslog-ng related conversations. Of course, we will promote it once it is up and running to raise awareness among those who are currently not part of the mailing list community. We cannot share an exact go-live date with you yet, however, you can expect to see it on the live site in a couple weeks. We will keep you posted. What happens to the mailing list? The mailing list would be phased out after the forum goes live. The team is open to constructive feedback and there will no doubt be enhancements further down to road. -- Hunor Voith Product Marketing Manager @ syslog-ng

3 3

Fedora i386
by Czanik, Péter 21 May '19

21 May '19

Hi, Anyone still using syslog-ng on Fedora i386? I ran into some compile problems recently. If there is someone still using it, I can put some effort into a workaround (most likely disabling Java support on i386). If there is no positive response until the end of this week, I'll simply drop i386 platform support from my Copr repositories. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.21.1 RPM packages
by Czanik, Péter 21 May '19

21 May '19

Hi, The initial version of my syslog-ng 3.21.1 RPM packages for Fedora / RHEL 7 and openSUSE / SLES are now ready. Right now they are available in my git snapshot repositories. Learn how to install them from my blog at: https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-n… Unlike the development snapshots these should not eat your server for breakfast, but still they went through only minimal testing. This is especially true about the Riemann and Kafka destinations, where I packaged the latest versions of external dependencies to make sure that microsecond precision and Kafka are available. I don't have the test environment for them. Please test the packages and let me know not just if you ran into problems, but also your success stories. I'm still traveling. I plan to create a dedicated 3.21 repo once I'm back to Hungary and read some positive feedback from you! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Multiple log statements vs If/else
by Faine, Mark R. (MSFC-IS40)[NICS] 20 May '19

20 May '19

Is there any functional difference between something like: log { .... }; log { ... flags(final); }; and log { if { ... } else { ... }; I was thinking about trying to build a configuration specific for an app from a Jinja2 template in Ansible and it seems like to me that if they aren't different it would be easier to do multiple log statements if generated dynamically. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

4 10

Systemd service times out for syslog-ng 3.16.1
by Ronald Fenner 16 May '19

16 May '19

I'm trying to switch from an init.d script to a system service and having problems getting the service to run. The syslog-ng service gets timed out and never starts up. My service looks like this: [Unit] Description="Syslog-ng an enhanced syslog" Documentation=man:syslog-ng(8) After=network.target ConditionFileNotEmpty=/opt/syslog-ng/etc/syslog-ng.conf [Service] Type=notify ExecStart=/opt/syslog-ng/sbin/syslog-ng --no-caps -p /var/run/syslogd.pid -rdvt ExecReload=/bin/kill -HUP $MAINPID StandardOutput=journal StandardError=journal Restart=on-failure Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java/jre/lib/amd64/server:$LD_LIBRARY_PATH [Install] WantedBy=multi-user.target My config looks like this: ############################################################################# # Default syslog-ng.conf.partial file which collects all local logs into a # single file called /var/log/messages. # @version: 3.16 @module "mod-java" @include "scl.conf" @include "conf.d/gc-parsers.conf" source s_internal {internal();}; source s_rtl_stream { unix-stream("/var/gamecircus/log/rtl-stream.sock" flags(no-parse) log_msg_size(1000000)); }; # Backup for when syslog-ng crashes and doesn't restart queues the messages source s_rtl_backup { file("/var/gamecircus/log/rtl-backup.log" flags(no-parse) log_msg_size(1000000)); }; destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("syslog-ng-`DeployedEnvironment`") ); }; destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("${topic}.${environment}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; destination d_syslog_ng { file("/var/log/syslog-ng"); }; parser p_json { json-parser(); }; log { source(s_internal); destination(d_syslog_ng); }; log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); }; log { source(s_rtl_backup); parser(p_json); destination(d_kafka_structured); }; # Build script will add stuff below for the specific build. ############################################################################### The logging that I turned on in the service above isbn't helping. Here's the last little bit May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217091] Add path to classpath: /opt/kafka/libs/kafka-streams-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217248] Add path to classpath: /opt/kafka/libs/rocksdbjni-5.0.1.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217309] Add path to classpath: /opt/kafka/libs/kafka-streams-examples-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.361062] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365718] Running application hooks; hook='1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365930] Running application hooks; hook='3' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366137] syslog-ng starting up; version='3.16.1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366480] Worker thread started; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service start operation timed out. Terminating. May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273058] Running application hooks; hook='4' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273074] syslog-ng shutting down; version='3.16.1' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.376882] Worker thread finished; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377030] Closing log transport fd; fd='92' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377186] Java machine free; May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.378858] Running application hooks; hook='5' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Failed to start "Syslog-ng an enhanced syslog". May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Unit syslog-ng.service entered failed state. May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service failed. Any suggestions on what me causing it to timeout? I'm new to Ssytemd but if I understand the notify type it would suggest that syslog-ng isn't notifying systemd that it's started. I'm not sure what would be holding it up. Any help is much appreciated. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 5

Rest2Syslog
by C B 15 May '19

15 May '19

hello all, just wanted to share with you a new python syslog-ng plugin I was working on to support ingestion of any REST API into any syslog destination. this is off course open source and available here - https://github.com/chenbekor/Rest2Syslog I'll be happy to receive any feedback/suggestion. Also - if any of you guys know of a public page with all available plugins/extensions please let me know so I can add this nice project to the list. Regards, Chen.

2 1

Re: [syslog-ng] Troubleshooting Question
by Walter Tienken 14 May '19

14 May '19

Hi, Yes, the em1 interface is for our front-end connectivity and the em2 interface is for the F5 LB where traffic is sent through. By dual stack, I assume you mean IPV4/IPV6 ... in that case, no, only IPV4. It looks like syslog-ng is listening on all interfaces: Here is the output of netstat -ulp: https://gist.github.com/MrTink76/ade770cfe6d497aa50fe820a9c438f54 The part that's throwing me off is "distinct" ... shouldn't that be 9999? You see it is listening on all interfaces and the netstat -anp | grep 9999 here as well: https://gist.github.com/MrTink76/7741d90831fe4f331abe4630f3a8b29e Thanks much, Walter Tienken walter.tienken(a)asu.edu On 5/12/19, 8:56 AM, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: Does your box have multiple interfaces? Does it run a dual-stack? If yes, does the program (nc or syslogng) listen on the correct interface? (netstat -ulp)

2 1

How to create two independent patternDB instances in syslog-ng
by Nitish Saboo 13 May '19

13 May '19

I understand PatternDB is a syslog-ng pluginand its dynamic library can be found in as libdbparser.so. I went through the following tests in ' https://github.com/balabit/syslog-ng/blob/master/modules/dbparser/patterndb… ' I want to create two independent pattern_db instances having two different ruleset. My queries: 1) How can I create two independent instances of patternDB having two different ruleset ? 2)How can I get the handle of these two independent PatternDB instances ? Can someone please help me on this ? Thanks

1 0

Is it possible to create Two instances of syslog-ng engine ?
by Nitish Saboo 13 May '19

13 May '19

Hi, I have installed syslog-ng on my Linux box and I am planning you use syslog-ng parser and wanted to initialise it's engine for parsing the data. so initialise() method in my Go code calls C code that is wrapper around syslog-ng C header files. So ,I am making use of cgo. Following is my Go code: func (obj Syslogparser) initialise{ X := C.CString(obj.X) defer C.free(unsafe.Pointer(X)) Y := C.CString(obj.Y) defer C.free(unsafe.Pointer(Y)) C.initialize_engine(X, Y, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo))); } And following is my C method 'initialize_engine': int initialize_engine(const gchar* X, const gchar* Y, key_value_cb cb) // cb is callback function { module_path = _module_path; app_startup(); msg_init(TRUE); configuration = cfg_new(0x0302); plugin_load_module("basicfuncs", configuration, NULL); plugin_load_module("syslogformat", configuration, NULL); pattern_db_global_init(); if(patterndb != NULL){ pattern_db_free(patterndb); } patterndb = pattern_db_new(); pattern_db_reload_ruleset(patterndb, configuration, filename); pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb); return 0; } My queries: 1)What is pattern_db_global_init, 'patterndb = pattern_db_new()', 'pattern_db_reload_ruleset', 'pattern_db_set_emit_func' doing in the following case ? 2)Can I make two independent instances of syslog-ng engines? 3)How can I call two instances of syslog-ng engines independently from my Go code? Thanks

2 2

Troubleshooting Question
by Walter Tienken 12 May '19

12 May '19

Hello All, I'd like to see if we can get some troubleshooting help with Syslog-NG OSE. Here's some background: Our environment collects logs from various network locations (F5, checkpoint, plixer, etc) and sends them to our Syslog-NG cluster through two F5 load balancers. We have four netlog boxes in round-robin that are running version 5 of the PE version of Syslog-NG on RHEL6. We also have a dev server running rhel7 and the 3.19.1 OSE edition of Syslog-NG from the COPR repository. On each of these servers, we run the Splunk Universal Forwarder that then sends the logs over to our indexer cluster. The problem we are having is that the dev server will not listen to any traffic except via localhost. We can see the traffic just fine on a TCPDUMP as it comes out of the load balancer into the dev box, but watching Syslog-NG in the foreground with -Fevd the traffic never registers at all. We can send test messages with loggen or netcat, etc, from localhost and Syslog-NG will see it and log it to disk as expected. Anywhere else it just never sees the traffic nor logs anything to disk even though we have confirmed that Syslog-NG is listening on the 9999 port to UDP with netstat. Here is our syslog-ng.conf file from the working RHEL6 boxes: https://gist.github.com/MrTink76/9ee1e88f93a313f953e4033560af463a This is the syslog-ng.conf file on the OSE box: https://gist.github.com/MrTink76/e181f4c0bf052077440d7bdfaf418e02 This is an example of our testing CONF file located in conf.d on the OSE box: https://gist.github.com/MrTink76/a0433b9e908ba683e36cb6199f9cc43f We send our test traffic to the F5 load balancer vip using UDP 9999. Like I said above, when we send on 9999 localhost with loggen or netcat, Syslog-NG sees it just fine and logs it to disk, but anywhere else it never registers nor records the test message to disk. We currently have SELinux disabled and there is no firewall running on the dev box (we see the traffic fine via tcpdump). Any help/suggestions would be greatly appreciated. Please let me know if I need to provide further information. Thanks much, Walter Tienken walter.tienken(a)asu.edu<https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeB…> Cloud and Advanced Network Engineering Services

3 4

unofficial syslog-ng packages for syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

Hi, syslog-ng 3.21.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 (dependencies are still building) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

3.21.1 Highlights * Add an alternative, native, librdkafka based kafka-c() destination in parallel of the existing Java implementation, that provides the same configuration interface. Eventually, we expect this to replace the Java one (#2496<https://github.com/balabit/syslog-ng/pull/2496>) * Add a native, http() based destination based driver for elasticsearch called elasticsearch-http(), as an alternative of the Java one. Eventually, we expect this to replace the Java implementation. (#2509<https://github.com/balabit/syslog-ng/pull/2509>) * Add the ability to automatically determine the timezone value for an incoming log entry as long as the incoming stream is close to real time and the timezone information is missing from the timestamp. Enable this function by using flags(guess-timezone) for sources and the date-parser(). (#2517<https://github.com/balabit/syslog-ng/pull/2517>, #2673<https://github.com/balabit/syslog-ng/pull/2673>) Features * syslog(): Add the ability to work with messages larger than log-msg-size() in the source driver by using the trim-large-messages(yes) option. The characters over the limit will be truncated. Previously messages longer than the limit caused the connection to be closed abruptly. (#2644<https://github.com/balabit/syslog-ng/pull/2644>) * amqp(): add support for heartbeats and the "external" authentication mechanism. (#2676<https://github.com/balabit/syslog-ng/pull/2676>, #2626<https://github.com/balabit/syslog-ng/pull/2626>) * graylog2(): add support for TLS and UDP. (#2657<https://github.com/balabit/syslog-ng/pull/2657>) * udp(): Add spoof-source-max-msglen() option to allow setting the maximum spoofed datagram size, which was hard-wired to 1024 previously. (#2535<https://github.com/balabit/syslog-ng/pull/2535>) * db-parser(): add an option program-template() that customizes the value used for matching the PROGRAM field. (#2651<https://github.com/balabit/syslog-ng/pull/2651>) * pdbtool: Add sort option to pdbtool merge (#2664<https://github.com/balabit/syslog-ng/pull/2664>) * $(implode) and $(explode): add template functions to split and join strings based on a simple separator. The exploded array is represented as a syslog-ng list that can be manipulated with the $(list-*) template functions. (#2700<https://github.com/balabit/syslog-ng/pull/2700>) * Add an --omit-empty-values option for value-pairs based destinations & template functions. (#2519<https://github.com/balabit/syslog-ng/pull/2519>) * grouping-by() parser: add sort-key() option (#2701<https://github.com/balabit/syslog-ng/pull/2701>) Support for non-syslog or non-standard formats in SCL * apache-accesslog-parser(): support for vhost:port as the first field in common/combined log formats (#2688<https://github.com/balabit/syslog-ng/pull/2688>) * Add application adapter for Junos classification (#2684<https://github.com/balabit/syslog-ng/pull/2684>) * Add parser and adapter for CheckPoint LogExporter output (#2665<https://github.com/balabit/syslog-ng/pull/2665>) Bugfixes * Fix race condition of idle timer and scheduled I/O job (#2650<https://github.com/balabit/syslog-ng/pull/2650>) * Few leaks find via sanitizer (#2696<https://github.com/balabit/syslog-ng/pull/2696>) * syslogformat: set $MSG even if the incoming message is empty (#2672<https://github.com/balabit/syslog-ng/pull/2672>) * Fix double-free error in logproto unit tests (#2662<https://github.com/balabit/syslog-ng/pull/2662>) * groupingby: identical persist name (#2659<https://github.com/balabit/syslog-ng/pull/2659>) * stats: deindex pruned counters/clusters (#2648<https://github.com/balabit/syslog-ng/pull/2648>) * Type hinting should not accept empty values (#2639<https://github.com/balabit/syslog-ng/pull/2639>) * app-parser, pseudofile: fix crash with grammar error (#2640<https://github.com/balabit/syslog-ng/pull/2640>) * python: set_timestamp normalization (#2643<https://github.com/balabit/syslog-ng/pull/2643>) * db-parser: fix memory leak (#2652<https://github.com/balabit/syslog-ng/pull/2652>) * grouping-by: use after free, memory leak, missing init calls of filters (#2655<https://github.com/balabit/syslog-ng/pull/2655>) * amqp: fixing double connect (#2660<https://github.com/balabit/syslog-ng/pull/2660>) * old style definition warning fixes (#2680<https://github.com/balabit/syslog-ng/pull/2680>) * Fix "!=" filter (#2683<https://github.com/balabit/syslog-ng/pull/2683>) * dbparser: fix memleak (#2706<https://github.com/balabit/syslog-ng/pull/2706>) * nondumpable-allocator: fixing mmap error handling (#2666<https://github.com/balabit/syslog-ng/pull/2666>) * Fix timeutils warning (#2604<https://github.com/balabit/syslog-ng/pull/2604>) * Fix old style include statement compatibility (#2600<https://github.com/balabit/syslog-ng/pull/2600>) * Fix config revert (threaded destinations) (#2596<https://github.com/balabit/syslog-ng/pull/2596>) * Add warning on old style include statement (#2592<https://github.com/balabit/syslog-ng/pull/2592>) Other changes * cfg-parser: add aliases for yesno (#2671<https://github.com/balabit/syslog-ng/pull/2671>) * Include json-c in the dist tarball (#2590<https://github.com/balabit/syslog-ng/pull/2590>) * cmake: disable_all_modules support (#2647<https://github.com/balabit/syslog-ng/pull/2647>) * Cmake clang sanitizer (#2562<https://github.com/balabit/syslog-ng/pull/2562>) * timeutils refactor (#2483<https://github.com/balabit/syslog-ng/pull/2483>) * Expedite threaded flush at reload (#2656<https://github.com/balabit/syslog-ng/pull/2656>) * elasticsearch2: Added deprecation warning (#2628<https://github.com/balabit/syslog-ng/pull/2628>) * Astyle fixes (#2624<https://github.com/balabit/syslog-ng/pull/2624>) * Force C99 with GNU (#2623<https://github.com/balabit/syslog-ng/pull/2623>) * Make rewording and other small edits to README (#2608<https://github.com/balabit/syslog-ng/pull/2608>) * Port tests to Criterion (#2607<https://github.com/balabit/syslog-ng/pull/2607>, #2661<https://github.com/balabit/syslog-ng/pull/2661>, #2621<https://github.com/balabit/syslog-ng/pull/2621>, #2620<https://github.com/balabit/syslog-ng/pull/2620>, #2619<https://github.com/balabit/syslog-ng/pull/2619>, #2618<https://github.com/balabit/syslog-ng/pull/2618>, #2617<https://github.com/balabit/syslog-ng/pull/2617>, #2616<https://github.com/balabit/syslog-ng/pull/2616>, #2615<https://github.com/balabit/syslog-ng/pull/2615>, #2599<https://github.com/balabit/syslog-ng/pull/2599>, #2594<https://github.com/balabit/syslog-ng/pull/2594>, #2593<https://github.com/balabit/syslog-ng/pull/2593>, #2591<https://github.com/balabit/syslog-ng/pull/2591>, #2586<https://github.com/balabit/syslog-ng/pull/2586>, #2584<https://github.com/balabit/syslog-ng/pull/2584>, #2583<https://github.com/balabit/syslog-ng/pull/2583>) * test_reliable_backlog: fix random failure (#2668<https://github.com/balabit/syslog-ng/pull/2668>) * Fix unit test with function pointer dereference in case of ASLR, Criterion (#2669<https://github.com/balabit/syslog-ng/pull/2669>) * test-stats-query: fix unit test (#2603<https://github.com/balabit/syslog-ng/pull/2603>) Notes to the developers * Version from git describe (#2627<https://github.com/balabit/syslog-ng/pull/2627>) * light: example-msg-generator support (#2571<https://github.com/balabit/syslog-ng/pull/2571>) * light: test app parser applications (#2686<https://github.com/balabit/syslog-ng/pull/2686>) * light: Switch to native logger (#2546<https://github.com/balabit/syslog-ng/pull/2546>) * light: Remove SetupTestcase() dependency (#2587<https://github.com/balabit/syslog-ng/pull/2587>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Chris Spencer, David Liew, Fabien Wernli, Gabor Nagy, Laszlo Budai, Laszlo Szemere, Layne, László Várady, Mehul Prajapati, Nik Ambrosch, Parth Wazurkar, Péter Kókai, Terez Nemes, Victor Ma, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. - balabit/syslog-ng github.com

1 0

feature request idea
by Faine, Mark R. (MSFC-IS40)[NICS] 09 May '19

09 May '19

How hard would it be to make --syntax-only accept a path to a configuration file as an argument? This would be useful to me since I use an Ansible template module to install the file and it has a validate parameter but it requires an argument for the filename. It could of course still function normally If no path to a configuration file was provided. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

2 2

UDP alerts in syslog
by Swordfish Binary 09 May '19

09 May '19

Hi, I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes. we have syslog version 3.5.6. Questions - Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make? Environment: - RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk Errors: Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors - The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors. Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };

5 4

GSoC Improve Developer Guide Project
by Victor Ma 09 May '19

09 May '19

Hi, I will be working on a plugin programming guide for this year's Google Summer of Code. It will cover source drivers, parsers, filter functions, and template functions. Each section will come with a hopefully-useful-on-its-own example plugin to aid in the explanation. My proposal can be found here: https://github.com/balabit/syslog-ng/wiki/GSoC-2019-Proposal:-Improve-Devel… I am a rising junior at the University of Toronto. I plan to graduate with a major in computer science and minors in philosophy and creative writing. Up-and-coming logger, Victor Ma

1 0