- syslog-ng - lists.balabit.hu

IP WHITE LIST
by Onur GURSOY 21 Aug '19

21 Aug '19

Hello Guys, Nowadays, im finding whitelist method for IP. Is it possible ? How can i allow IP which is in whitelist ? So Many Thanks, My Best Regards. -- Onur GÜRSOY R&D Engineer in Embedded Systems Master Student at Gebze Institute Of Technology Department Of Electronic Engineering GSM : 0(545) 764 7653 e-mail: onurgursoygyte(a)gmail.com

2 1

Dynamic collection name in MongoDb
by PC LP 15 Aug '19

15 Aug '19

Hi, I am retrieving the value for *$M_ADDRESS* from the Syslog message using patterndb. $M_ADDRESS is a unique value. Based on that I have to create the collection in MongoDB dynamically. If I provide the value like below it is not recognizing collection("$M_ADDRESS") Is there any alternate way to create it? Please advise. Regards, PCLP.

2 1

if condition
by Alexey Vlasov 15 Aug '19

15 Aug '19

Hi, I would like to optimize my syslong-ng.conf file thus it could be used on some boxes without any changing. Now I have a few modifications of syslong-ng.conf on each server to copy logs to remote box. For example, group of hosts A* ============================== A1-machine (it is its hostname) has: destination clnt { udp("remote-box-a" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== ============================== A2-machine has: destination clnt { udp("remote-box-a" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== and group of hosts B* ============================== B1-machine has: destination clnt { udp("remote-box-b" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== ============================== B2-machine has: destination clnt { udp("remote-box-b" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== Is there possible to write something like that If (${HOSTNAME} ~ ^A) use remote-box-a If (${HOSTNAME} ~ ^B) use remote-box-b Thanks in advance!

2 1

3.22.1: understanding the location of the python's part
by Sergey 14 Aug '19

14 Aug '19

Hello I see that in 3.22 has changed the location of the python's part (relative 3.20): architecture independent files moved to architecture specific directory (lib64 on x86_64). For example, rhel spec 3.20 (syslog-ng-3.20.1/packaging/rhel/syslog-ng.spec) %files python /usr/lib/python2.7/site-packages/syslogng-1.0-py2.7.egg-info %dir /usr/lib/python2.7/site-packages/syslogng/ %dir /usr/lib/python2.7/site-packages/syslogng/debuggercli/ /usr/lib/python2.7/site-packages/syslogng/* /usr/lib/python2.7/site-packages/syslogng/debuggercli/* %{_libdir}/%{name}/libmod-python.so The same in 3.22 %files python %{_libdir}/%{name}/python/syslogng-1.0-py%{py_ver}.egg-info %{_libdir}/%{name}/python/syslogng/* %{_libdir}/%{name}/libmod-python.so Is the architecture specific file libmod-python.so only? Or am I missing something? -- Regards, Sergey

2 1

Parse message fields for use as columns in MySQL
by Allen Olivas 09 Aug '19

09 Aug '19

Hello, I'm really scratching my head trying to make this work and thought maybe the community has experienced this before. I'm collecting logs from Wazuh and Syslog-NG. Those logs are sent from my Wazuh server with Sylog-NG configured to send to my MySQL server. The Syslog-ng.conf file on the MySQL server is configured with a destination to mysql. In the declared the destination and list out columns and values. # MySQL define destination destination d_mysql { sql( type(mysql) username("syslog") password("xxxxxxx") database("syslog") host("127.0.0.1") table("logs") columns("host", "id", "location", "facility", "rule", "priority", "level", "tag", "datetime", "program", "msg") values("$HOST", "$ID", "$LOCATION","$FACILITY", "$RULE", "$PRIORITY", "$LEVEL", "$TAG","$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC","$PROGRAM", "$MSG") indexes("datetime", "host", "id", "location", "rule") ); }; So here's the problem. The Message data contains information like Rule and Location that really equate to the Wazuh Rule and Location = the Wazuh Agent that's reporting it. I had hoped "location" column would populate with the Location date from the message. Same with Rule. SO my question to the community is how on earth do I parse the data in the message field to populate columns (existing or new)? Any thoughts, guidance, recommendations are greatly appreciated. Thanks, Allen Olivas InfoDefense Office: (972) 848-7910 Email: allen.olivas(a)infodefense.com Toll Free: (877) INFODEFENSE www.infodefense.com<https://l.shatrk.com/r/e/DblvLSPvKY2IxMPE?r=https://app.salesforceiq.com/r?…>

2 5

Syslog-ng stops writing until restarted
by Faine, Mark R. (MSFC-IS40)[NICS] 07 Aug '19

07 Aug '19

I have a strange problem where logs are definitely coming in over tcp and udp however they are not written, well not until I restart the server. Whenever I restart the logs are written but shortly after starting they stop until the server is restarted again. Any ideas on where I can look? Thanks, -Mark

2 2

syslog-ng configuration from CLI
by SZIGETVÁRI János 06 Aug '19

06 Aug '19

Hello Team, A few years back I heard about a project that would have added functionality to syslog-ng that would have enabled configurability from the command line. This was the respective PR: https://github.com/balabit/syslog-ng/pull/1326 Since then the PR was closed and part of the functionality provided by the PR has been added to syslog-ng (in https://github.com/balabit/syslog-ng/pull/1605) but not the CLI configurability part. I personally think that being able to configure syslog-ng through its command line arguments would bear a great potential. Of course running it with a complex configuration passed in through CLI argument would be infeasible, but with simple, single-purpose logging setups it would prove to be a very powerful tool. For example when trying to empty a bunch of orphaned disk queue files, it would be very easy to spin up several syslog-ng instances from a script with only slightly differing configuration. What do you guys think? Do you agree as well that this feature would bear a lot of added value to syslog-ng? Thanks for sharing your opinions! Best Regards, János Szigetvári -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp

3 3

pattern documentation for patterndb
by Russell Fulton 06 Aug '19

06 Aug '19

I am sure I have asked this question before but I can’t find the answer now. The Admin Guide has a very sort section on patterns with a couple of examples. I can’t find the detailed docs for the actual pattern elements (parser types). I seem to remember that these we documented in a separate pdf but I can’t find it. I add new rules about once a year and struggle with it every time! Russell

2 1

Quoting in filters
by Faine, Mark R. (MSFC-IS40)[NICS] 05 Aug '19

05 Aug '19

What is the policy on quoting values in filters? In the docs it seems that sometimes they are quoted and other times they are not. For example, are these equivalent: netmask (127.0.0.1/32) and netmask("127.0.0.1/32") I think that maybe some filters, the ones that take strings essentially, are quoted, and not the ones that take numeric data. Is that correct? Thanks, -Mark

2 1

new install of OSE 3.22 fails on ubuntu 16,04
by Russell Fulton 05 Aug '19

05 Aug '19

Running on ubuntu 16.04 I have just compiled and installed syslog-ng version 3.22. When started it prints: iv_tls_user_register: called after iv_init and then exits. Google tells me there was a bug that caused this back in 2013 but there are not recent posts. Any idea what is going on? BTW I can’t use the binary packages as there are conflicts between the rsyslog and syslog-ng packages and I am stuck with having the rsyslog package by corporate policy. Russell

2 1

Fwd: Multithreaded inputs with same output file : is it safe ?
by Jean-Baptiste Fuzier 02 Aug '19

02 Aug '19

Hello, I have a few questions related to multithreading. >From the doc, my understanding is that inputs are multithreaded this way : 1/ Tcp connections are sent to different thread when multithreading is enabled, even if there is a single source enabled in config with a single tcp port listening 2/ UDP datagrams are NOT sent to different thread if there is a single source even if there are multiple listening ports in the source config 3/ UDP datagrams are sent to different threads if there are different sources in config (1 source -> 1 thread), so basically to benefit from multithreading with UDP one needs to configure several sources. Is my understanding correct ? If we have several UDP source with some round robin balancing of the input to different source on the same daemon process, that means that several host with the same name can send their syslog data to different source. For the output, I am using files with the name of the sending host in the name. Is there an issue if several source (= from different threads) ends up being outputed in the same file. Another way to turn the question is: are all the data from the source thread then send to some common thread so that output to a file is always handled by the same thread even if log is received from different source threads ? I tried to ask the same question with a schematic : Case1 : +-------------------------+ +-------------------+ +-----------------------------+ +-----------+ +----------+ | | | | | | | host foo +------->+ |X----->+ thread1 some foo log +----->+ +--->+ thread handling foo.log +------>foo.log +-----------+ | iptables |X XX| some bar log | | black box magic | | | | round | XXXX +-------------------------+ | | +-----------------------------+ | robin | XXXXX | | | |XX XX+-------------------------+ | | +-----------------------------+ +-----------+ | X------>+ | | | | | | host bar +------->+ | | thread2 some foo log +----->+ +--->+ thread handling bar.log +------>bar.log +-----------+ +----------+ | some bar log | | | | | +-------------------------+ +-------------------+ +-----------------------------+ Case2 : +-------------------------+ +-----------+ +----------+ | | | host foo +------->+ |X----->+ thread1 some foo log +---------------------> +--------+ +-----------+ | iptables |X XX| some bar log +----------| |foo.log | | round | XXXX +-------------------------+ +------------> +--------+ 2 different thread writting | robin | XXXXX | | to same file | |XX XX+-------------------------+ | +---------->----------+ likely bad ! +-----------+ | X------>+ +--------+ |bar.log | | host bar +------->+ | | thread2 some foo log +--------------------->----------+ +-----------+ +----------+ | some bar log | +-------------------------+ One example config : # Notice that host name is used in output file name # There is no afinity between 1 host <> 1 udp port => logs from host foo are send to every source/udp port in a round robin fashion source s_syslog__udp5141{ network(ip(192.168.1.1) transport("udp") port(5141) flags("threaded")); }; source s_syslog__udp5142{ network(ip(192.168.1.1) transport("udp") port(5142) flags("threaded")); }; source s_syslog__udp5143{ network(ip(192.168.1.1) transport("udp") port(5143) flags("threaded")); }; source s_syslog__udp5144{ network(ip(192.168.1.1) transport("udp") port(5144) flags("threaded")); }; destination d_network_fw_misc { file("`syslog_log_root_path`/fw/fw/$FULLHOST/`naming_scheme`" group("logs_fw") dir-group("logs_fw") log-fifo-size(500000)); }; destination d_network_wifi { file("`syslog_log_root_path`/wifi/$FULLHOST/`naming_scheme`" group("logs_wifi") dir-group("logs_wifi") log-fifo-size(500000)); }; destination d_network_debug { file("`syslog_log_root_path`/debug/`naming_scheme`" group("logs_network_debug") dir-group("logs_network_debug") log-fifo-size(600000)); }; log { source(s_syslog__udp5141); source(s_syslog__udp5142); source(s_syslog__udp5143); source(s_syslog__udp5144); # Every channel of the junction received a copy of the message, unless there is a final flag before # Final flags are applied only within the junction and not globally junction { channel {filter(f_debug); destination(d_network_debug);}; channel {filter(f_network_wifi); destination(d_network_wifi); flags(final);}; channel {destination(d_network_fw_misc); flags(final);}; }; }; #Round robin with iptables : iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 4 --packet 0 -j REDIRECT --to-port 5141 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 3 --packet 0 -j REDIRECT --to-port 5142 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 2 --packet 0 -j REDIRECT --to-port 5143 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 1 --packet 0 -j REDIRECT --to-port 5144 Thanks for your help -- Jean-Baptiste Fuzier

2 1

cisco templetes
by ciprian niculescu 31 Jul '19

31 Jul '19

Hello, i'm building a syslog relay to collect and duplicate the flows to multiple destinations. but the relayed messages are strange looking. my source are cisco network devices (catalyst, nexus, asa) and i want to relay to a Solarwinds, Splunk and a linux-syslog for archiving. i search the net for a templete but found none. What i got so far is that the catalyst is sending in syslog bsd format, but with the relay configured to source bsd and destination bsd, the end message is different (the date is doubled, the relay add his IP) any help is appreciated. Regards, Ciprian

2 1

Tips for handling large message load
by Faine, Mark R. (MSFC-IS40)[NICS] 30 Jul '19

30 Jul '19

I have several Splunk log aggregators that gets thousands of messages per second but we are seeing issues with dropping messages from UDP sources. I've read the section in the docs about handling large message load and we've made many of those changes. Do you have any other suggestions to improve performance? We are using flow control. We have made the following sysctl changes: - net.core.rmem_max = 268435456 - net.core.netdev_max_backlog = 2000 We have increased flush-lines to 100 We are also looking to increase the initial window size below and max connections as well as the so_rcvbuf. network(port(514) transport("tcp") max-connections(100) log_iw_size(10000) flags(syslog-protocol)); network(port(514) transport("udp") max-connections(100) log_iw_size(10000) flags(syslog-protocol)); network(ip(0.0.0.0) transport("tls") port(10514) max-connections(100) log_iw_size(10000) flags(syslog-protocol) I'm currently showing about 50 TCP connections and almost 1000 UDP connections, though it's early on Monday morning so it is likely to increase as the day goes on. Most of the log messages are coming from the VPNs (hence so many UDP connections). The servers are quite beefy with 64GB of RAM and 24 Xeon cores @ 2.4GHz (Dell PowerEdge R530). We are using two separate network interfaces one for TCP and the other for UDP. I'd appreciate any suggestions on how to further increase performance, also, any general rules or calculations I can use to determine optimal values for these parameters from available system metrics would be very helpful. Thanks, -Mark

3 2

SYSLOG to SNMP TRAP relay server, how can I spoof source address
by Armando Martires 23 Jul '19

23 Jul '19

Hello, I set up a Syslog-ng relay server to relay syslog messages as SNMP traps. The relay is working sending a SNMP TRAP whenever a SYSLOG message is received, but the source seems to be the syslog-ng server and not the client host that originated the syslog message. This is my config: options { time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); chain_hostnames (no); }; source s_labs_itsm { syslog(ip("192.168.50.20") transport("udp")); }; destination d_local { file("/var/log/messages_${HOST}"); }; destination d_ss { snmp( host("192.168.50.10") version("v2c") community("public") trap-obj('.1.3.6.1.6.3.1.1.4.1.0', 'Objectid', '.1.3.6.1.4.1.18372.3.1.1.1.2.1') snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.1.0', 'Octetstring', '${MESSAGE}') snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.2.0', 'Octetstring', 'admin') snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.3.0', 'Ipaddress', '${HOST}') ); }; log { source(s_labs_itsm); destination(d_local); destination(d_ss); }; Can anyone help me out understanding what I'm doing wrong? Thanks! -- Armando Mártires

2 1

Syslog-ng header & rfc5424
by Florian Goulais 19 Jul '19

19 Jul '19

Hi. I'am trying to understand why syslog-ng and an header (timestamp host) to log as the rfc specify another header. Someone can explain me the reason plz ? Regards Florian

3 6

Get local log files written immediately even if remote log server is unresponsive
by Jim Segrave 16 Jul '19

16 Jul '19

We are running syslog-ng 3.16.1 on Centos 7.4.1708 on a central logging host. We have a large (nearly 1,000) servers also running the same version of syslog-ng on the same Centos release. The servers are configured to log locally and also forward logs to the central logging host. This morning we encountered a problem - syslog-ng was running on the logging host, but was not processing incoming logs or locally generated ones. The servers forwarding to the central host did not write anything to their local log files, a small but significant portion of them had syslog crash, after which is was restarted by systemd, but still no logs were written until syslog-ng was forcibly stopped on the central server and then restarted. Connections to the central server weren't failing in the sense of TCP close or reset, but logs were accumulating on all the servers, including the central one, in the cache file for buffering logs. For our purposes, we need to have up-to-the moment logs available on the individual servers, so an admin going in to troubleshoot on a server who only has console access still has recent logs to consult if needed. Is there a way to tell syslog-ng to write local logs immediately even if it's currently buffering logs for sending to a non-responsive remote server?

3 3

Missing messages
by Klimek, Bryan J. 16 Jul '19

16 Jul '19

First time poster, so be gentle. We run syslog-ng in our environment as a centralized syslog manager for the archiving of syslog data. We have over 2000+ Unix/Linux systems sending their syslog data with a daily ingest rate of about 10GB per day. It was recently pointed out to me that one particular message from one specific host is not getting persisted to the files on our syslog-ng server all the time. That is to say, it is intermittent. If one person can find one message that is not making into our syslog-ng archive, I can only assume that we are dropping other messages as well. How can I debug if and when messages are being lost and not making into the files on my centralized syslog server? Bryan Klimek Mayo Clinic

4 10

Cannot send Syslog-ng to Elasticsearch
by Allen Olivas 12 Jul '19

12 Jul '19

Hello, Recently I've tried following along with the Syslog-NG to Elasticsearch and Kibana blog posts and Admin Documentation for integrating Syslog-NG into Elasticsearch but I'm unable to integrate the two. I see in the .conf files the destination calls for creating and Index Pattern for Syslog-NG but when I curl the existing indices I do not see syslog-ng. Also, I'm now receiving two errors. The first I'm fairly certain we need to resolve but I've not been able to find adequate documentation on how to identify the issue let along resolve it, and the second I'm not sure if we actually need to fix. The two issues: Issue#1: I/O error occurred syslog-ng[26432]: Syslog connection established; fd='12', server='AF_INET(127.0.0.1:9200)', local='AF_INET(0.0.0.0:0)' syslog-ng[26432]: I/O error occurred while writing; fd='12', error='Broken pipe (32)' syslog-ng[26432]: Syslog connection broken; fd='12', server='AF_INET(127.0.0.1:9200)', time_reopen='60' Issue#2: Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' For issue 1 I'm not sure what to do or how to resolve it. For issue 2, I know for certain libjvm does exist, and I've mapped the LD_LIBRARY_PATH to the directory libjvm.so resides in. Ultimately, are these two issues preventing Syslog-NG from sending to Elasticsearch or are they just separate issues to tackle after I get things cleared up, and most importantly if they're not related, how do I integrate Syslog-NG with Elasticsearch and Kibana. Documentation is not helpful and not concise. Thanks!

4 18

Periodic "Destination timed out, reaping" and disconnection on a write fifo (pipe)
by Charles Eidsness 27 Jun '19

27 Jun '19

I kind of feel like this is going to be a dumb question but I just can't figure this out. I'm trying to get syslog-ng working with sagan, but every 60 seconds syslog-ng closes the fifo it's writing to, which closes the sagan connection to the fifo. When running in verbose mode syslog-ng is throwing up a "Destination timed out, reaping..." message. I simplified my syslog-ng configuration to: source s_sys { system(); }; destination d_fifo { pipe( "/tmp/test.fifo" ); }; log { source(s_sys); destination(d_fifo); }; And then run use cat to read the fifo: cat /tmp/test.fifo It sends all of the log traffic every but every 60 seconds cat closes. Is this expected behavior? Am I missing something? I'm using syslog-ng version 3.19.1. Thanks! Charles == Charles Eidsness Chief Engineer *CCX Technologies* (613)791-9715

2 7

Sending Messages to Slack via a HTTP Proxy
by Frank Crawford 26 Jun '19

26 Jun '19

We have recently upgraded to syslog-ng v3.19 so we can use the slack hook, but have an issue that to access the URL we will need to go through a squid web proxy. How do we configure the details of the proxy in either syslog-ng or in the slack destination? Regards Frank -- [http://www.ac3.com.au/emailsigs/v2/ac3_logo.png]<http://www.ac3.com.au> the human touch in a technical world Frank Crawford Technology and Infrastructure Manager t 02 9199 0995 | m 0401 714 870 e frank.crawford(a)ac3.com.au<mailto:frank.crawford@ac3.com.au> Level 8 East, 8 Central Ave, Eveleigh NSW 2015 [http://www.ac3.com.au/emailsigs/v2/linkedin.png] <https://www.linkedin.com/company/ac3> [http://www.ac3.com.au/emailsigs/v2/twitter.png]<https://twitter.com/ac3ptyltd> [http://www.ac3.com.au/emailsigs/v2/iso.png]<http://www.ac3.com.au/about-us/news/double-iso-certification-ac3> This email is for intended recipients only. If you are not an intended recipient, you must not disclose, copy or distribute this email to others or act upon the information contained in this email in any way. This email may contain privileged or confidential information. If you have received this email in error, please notify the sender and then delete it from your system.

2 4

error on syslog-ng elasticsearch2 destination
by joao reis 26 Jun '19

26 Jun '19

Hi, I'm getting the following error when trying to configure the elasticseach2 backend: Error parsing destination statement, destination plugin java not found in block destination elasticsearch2() at /usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf:25:4:3-4:7: 1 2 #Start Block block destination elasticsearch2() at /usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf:25 3 4-----> java( 4-----> ^^^^ 5 class_path("//usr/lib64/syslog-ng/java-modules/*.jar://usr/lib64/syslog-ng/java-modules/elastic-jest-client/*.jar:/*.jar") 6 class_name("org.syslog_ng.elasticsearch_v2.ElasticSearchDestination") 7 option("index", "logserver1") 8 option("type", "syslog") 9 option("server", "localhost") This is the block that causes the error: destination d_es { elasticsearch2 ( cluster("logserver1") client-mode("http") index("logserver1") type("syslog") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ); }; root@elk-server1 conf.d]# rpm -qa|grep syslog-ng syslog-ng-3.21.1-3.el7.x86_64 syslog-ng-java-3.21.1-3.el7.x86_64 syslog-ng-http-3.21.1-3.el7.x86_64 Any ideia? Thanks in advance.

1 1

syslog-ng-3.22: unofficial Debian/Ubuntu packages
by Laszlo Budai 25 Jun '19

25 Jun '19

Hi, syslog-ng 3.22.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.22.1
by Laszlo Budai 25 Jun '19

25 Jun '19

3.22.1 Highlights * Sending SNMP traps: Using the new snmp() destination, incoming log messages can be converted to SNMP traps, as the fields of the SNMP messages can be customized with macros. (#2693<https://github.com/balabit/syslog-ng/pull/2693>) * $(template) dynamic binding: Extends the $(template) template function to allow dynamic binding. For example, the name of the template to be invoked can come from the message (name-value pairs). (#2716<https://github.com/balabit/syslog-ng/pull/2716>) * syslog(), network(): Add dynamic-window-size() option to enable dynamic flow control that distributes the specified amount of window between active connections at runtime. This can be used in low-memory environments, where only a small subset of the active clients sends messages at high rate. (#2772<https://github.com/balabit/syslog-ng/pull/2772>) Features * match(): Add support for the template() option (#2715<https://github.com/balabit/syslog-ng/pull/2715>) * add-contextual-data(): Allow using templates in name-value pairs (#2711<https://github.com/balabit/syslog-ng/pull/2711>) * Add support for floating point operations in template functions (#2742<https://github.com/balabit/syslog-ng/pull/2742>) * Add support for usec precision when parsing time (#2709<https://github.com/balabit/syslog-ng/pull/2709>) Bugfixes * Fix null pointer access when destinations are suspended (#2778<https://github.com/balabit/syslog-ng/pull/2778>) * Fix grouping-by() deadlock (#2758<https://github.com/balabit/syslog-ng/pull/2758>) * Fix a general source-related crash and enhance wildcard-file()'s bookmark handling (#2589<https://github.com/balabit/syslog-ng/pull/2589>) * Fix infinite loop (reload/reopen) (#2739<https://github.com/balabit/syslog-ng/pull/2739>) * Fix python() package/module name collision (#2438<https://github.com/balabit/syslog-ng/pull/2438>) * Fix escaped quote in block argument (#2781<https://github.com/balabit/syslog-ng/pull/2781>) * Reintroduce test on SYSLOG_NG_HAVE_TIMEZONE (#2774<https://github.com/balabit/syslog-ng/pull/2774>) * snmp(): Fix template leak (#2746<https://github.com/balabit/syslog-ng/pull/2746>) Other changes * Never drop flow-controlled messages: The meaning of log-fifo-size() has changed to avoid dropping flow-controlled messages when log-fifo-size() is misconfigured. From now on, log-fifo-size() only affects messages that are not flow-controlled. (#2753<https://github.com/balabit/syslog-ng/pull/2753>) * The -d/--debug syslog-ng command line flag no longer implies -e/--stderr. If you want to redirect internal() source to stderr, use the -e/--stderr option explicitly. (#2731<https://github.com/balabit/syslog-ng/pull/2731>) * dbld, RPM and DEB packaging improvements (#2724<https://github.com/balabit/syslog-ng/pull/2724>) * Checkpoint parser improvements (#2740<https://github.com/balabit/syslog-ng/pull/2740>) * Reset the timezone on config reload event (#2691<https://github.com/balabit/syslog-ng/pull/2691>) * geoip2(): Include IP into the error message (#2743<https://github.com/balabit/syslog-ng/pull/2743>) * Improve regexp error messages (#2796<https://github.com/balabit/syslog-ng/pull/2796>) * http(): Warn if less workers used than urls (#2757<https://github.com/balabit/syslog-ng/pull/2757>) * http(): Allow URLs to be specified by a space/comma separated string (#2699<https://github.com/balabit/syslog-ng/pull/2699>) * loggen: Change message rate at runtime using signals (#2756<https://github.com/balabit/syslog-ng/pull/2756>) * debun: add acquire_running_syslog_config function (#2752<https://github.com/balabit/syslog-ng/pull/2752>) * FreeBSD fixes for the test suite (#2783<https://github.com/balabit/syslog-ng/pull/2783>) Notes to the developers * ivykis: update to 0.42.4 (#2736<https://github.com/balabit/syslog-ng/pull/2736>) * Support generator plugins in global options (#2747<https://github.com/balabit/syslog-ng/pull/2747>) * logthrfetcher: new constants (#2766<https://github.com/balabit/syslog-ng/pull/2766>) * logthrsourcedrv: support position tracking (#2750<https://github.com/balabit/syslog-ng/pull/2750>) * Light: Support pre-commit and tox (#2725<https://github.com/balabit/syslog-ng/pull/2725>) * Enable Bison error flags: conflicts-sr/rr (#2762<https://github.com/balabit/syslog-ng/pull/2762>) * Dynamic stats constant registration (#2761<https://github.com/balabit/syslog-ng/pull/2761>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Christian Michallek, Fabien Wernli, Gabor Nagy, Kyeong Yoo, Laszlo Budai, Laszlo Szemere, László Várady, Mehul Prajapati, Norbert Takacs, Oleksii Hamov, Péter Kókai, Romain Tartière, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.22.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. - balabit/syslog-ng github.com

1 0

Re: [syslog-ng] grouping-by question
by Газин Максим Алексеевич 20 Jun '19

20 Jun '19

bypassed the problem in this way $(- $(context-length) 1) And one more question: is it possible to use the context-length instead of a timeout as the context closure value? -----Original Message----- From: Fabien Wernli [mailto:wernli@in2p3.fr] Sent: Thursday, June 20, 2019 12:25 PM To: Газин Максим Алексеевич <Maksim.Gazin(a)rt.ru> Cc: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: RE: [syslog-ng] grouping-by question Hi, On Thu, Jun 20, 2019 at 07:17:28AM +0000, Газин Максим Алексеевич wrote: > what does it mean? Is it possible to make the context-length equal to the value of the aggregated messages? In your original email, your config indicated that you were using `timeout(15)` in the grouping-by config. As far as I understand the grouping-by parser, this means that the context will only be closed (and thus context-len reinitialized) if no message matches the condition for 15 seconds.

3 2

Insider 2019-06: Python; Google Stackdriver; elasticsearch-http(); a year of syslog-ng; Red Hat Summit;
by Peter Czanik (pczanik) 20 Jun '19

20 Jun '19

Dear syslog-ng users, This is the 74th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Format your log messages in Python ---------------------------------- Sometimes getting log messages into the desired format can be a problem, but with syslog-ng you can use Python to get the exact format you need. The syslog-ng Python template function allows you to write custom templates for syslog-ng in Python. In this blog post, I will show you a simple use of the Python parser: resolving IP addresses to host names. I will also show you the logger method, a nice new feature that enables you to log to syslog-ng’s internal() log source instead of writing logs from Python to stdout. This way you can follow what your Python code is doing even if syslog-ng is running as a daemon in the background. https://www.syslog-ng.com/community/b/blog/posts/format-your-log-messages-i… Sending logs to Google Stackdriver using syslog-ng PE ----------------------------------------------------- Google Stackdriver collects and analyses logs, events and metrics of your infrastructure. Using syslog-ng PE 7.0.14 or later versions, you can send your logs to Google Stackdriver. While originally designed to quickly respond to events in the Google Cloud Platform (GCP), you can use Google Stackdriver with any other cloud providers (for example, Amazon Web Services) or on-premises data as well. This way you can view events of a hybrid infrastructure at a single location. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-google-sta… How to forward logs to Elasticsearch using the elasticsearch-http() destination in syslog-ng ------------------------------------------------------------- Starting with syslog-ng 3.21.1, you can use the elasticsearch-http() destination (based on the http() destination) to send logs to Elasticsearch. In most cases it is more efficient than the old Java-based Elasticsearch destination, and it is definitely easier to configure. https://www.syslog-ng.com/community/b/blog/posts/how-to-forward-logs-to-ela… A year of syslog-ng ------------------- With about two thirds of syslog-ng users running their logging application on Red Hat Enterprise Linux (or CentOS), it is one of our most important platforms both for the open source edition (OSE) and the premium edition (PE) of syslog-ng. Preparing for the annual Red Hat Summit, I made a list of major new features in syslog-ng since last year’s event. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-is-coming-to-red… Sudo + syslog-ng: two softwares at two conferences -------------------------------------------------- I visited two conferences recently: LOADays and Red Hat Summit. Both conferences focus on open source softwares, but there the similarity ends. LOADays in Antwerp is small, free and focuses on Linux administrators. The Red Hat Summit in Boston is huge, expensive and covers a wide variety of topics, including administration (among many others). Regardless of the differences, both events are among my favorites. https://www.syslog-ng.com/community/b/blog/posts/sudo-syslog-ng-two-softwar… CONFERENCES * Pass the SALT 2019 (https://2019.pass-the-salt.org/) syslog-ng workshop * All Things Open (https://allthingsopen.org/) WEBINARS * How to use the syslog-ng PE’s new Google Stackdriver destination https://www.syslog-ng.com/event/live-webinar-how-to-use-the-syslogng-pes-ne… Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

grouping-by question
by Газин Максим Алексеевич 20 Jun '19

20 Jun '19

Hello, I'm new to syslog-ng, and I'm having some trouble. I want to use syslog-ng to aggregate similar events from our firewalls. I use the grouping-by parser to create an aggregated event, but I have a question, how do I get only aggregated events on the output without source ones? My configuration: parser p_patterndb { db-parser(file("/etc/syslog-ng/conf.d/forti.xml")); }; log { source { file("/tmp/in.log" flags(no-parse)); }; parser { kv_parser (prefix("forti.")); grouping-by( key("${forti.devname}${forti.srcip}${forti.dstip}${forti.dstport}") scope("process") timeout(15) aggregate( value("MESSAGE" "aggegated devname=${forti.devname} srcip=${forti.srcip} dstip=${forti.dstip} dstport=${forti.dstport} count=$(context-length)") inherit-mode("context") ) ); }; destination { file("tmp/out.log"); }; }; My current output: Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="fw01" devid=" " vd="" date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:27 centos7-1 <15>Jun 10 16:10:02 fw01 fw: logver=56 devname="-fw01" devid="" vd=" " date=2019-06-04 time=15:53:34 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1559652814 srcip=172.26.104.3 srcport=62444 srcintf="ae1.2051" srcintfrole="lan" dstip=172.16.104.219 dstport=9092 dstintf="ae0.2099" dstintfrole="lan" sessionid=3465151973 proto=6 action="deny" policyid=0 policytype="policy" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" crscore=30 craction=131072 crlevel="high" Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 What I want: Jun 8 08:42:28 centos7-1 aggegated devname=fw01 srcip=172.26.104.3 dstip=172.16.104.219 dstport=9092 count=3 Any guidance would be very much appreciated. Maxim Gazin | Senior security engineer | Rostelecom

3 5

An External Application For Parsing Existing Syslog-ng Log File
by Sezer BAGLAN 18 Jun '19

18 Jun '19

Dear all, Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below: ${ISODATE} ${HOST} ${FACILITY} ${LEVEL} ${LEVEL_NUM} ${TAGS} ${MESSAGE} 1) What is your opinion about the best practices to handle this issue? Please note that I'm not talking about logging messages with filters and parsers. I'm talking about parsing an existing syslog-ng log file. 2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job? 3) Do you suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on? Thanks in advance. Kind regards, Sezer

2 1

Difficulty with Elasticsearch Destination
by Kelly Marchewa 16 Jun '19

16 Jun '19

Greetings all: I recently installed syslog-ng on my Debian 9 system. I have tested syslog-ng with a file destination and everything seemed fine. Now, I am attempting?? to set the destination to my local Elasticsearch instance. However, I am receiving the following error: /Error parsing destination, destination plugin elasticsearch-http not found in /etc/syslog-ng/syslog-ng.conf at line 44, column 1:/ /elasticsearch-http(/ /^^^^^^^^^^^^^^^^^^/ My configuration, based on the documentation: /elasticsearch-http(////?? index("syslog")////?? type("test")////?? url("http:/localhost:9200/_bulk"));////};/ Output of syslog-ng --version: /syslog-ng 3.8.1////Installer-Version: 3.8.1////Revision: 3.8.1-10////Module-Directory: /usr/lib/syslog-ng/3.8////Module-Path: /usr/lib/syslog-ng/3.8////Available-Modules: disk-buffer,redis,date,afstomp,afprog,afsocket,pseudofile,confgen,csvparser,afsql,kvformat,geoip-plugin,afuser,afsmtp,system-source,mod-python,afamqp,riemann,linux-kmsg-format,dbparser,basicfuncs,syslogformat,graphite,afmongodb,json-plugin,cryptofuncs,affile,sdjournal,cef,add-contextual-data////Enable-Debug: off////Enable-GProf: off////Enable-Memtrace: off////Enable-IPv6: on////Enable-Spoof-Source: on////Enable-TCP-Wrapper: on////Enable-Linux-Caps: off/ I saw no indication in the documentation that any special modules/plugins needed to be loaded to use Elasticsearch as the destination. Any pointers as to what the issue is would be appreciated. Thanks! -- Kelly Marchewa kelly_marchewa(a)yahoo.com

2 1

Ring buffered Log into file or Db
by Yiğit YAŞAR 13 Jun '19

13 Jun '19

Hi All, I got a limited system so i need ring buffered logging for protect disk space. I did some research but i could not found the answer. Does syslog-ng support ring buffer for logging to file or db ? Best Regards, Yigit

2 1

syslog-ng at Splunk Live in London
by Peter Czanik (pczanik) 12 Jun '19

12 Jun '19

Hi, The next Splunk Live event is tomorrow in London: https://live.splunk.com/london While there you are there you should also visit the syslog-ng booth and learn, how syslog-ng can provide you with a scalable log management layer in front of Splunk. Register for demo with my colleagues at https://www.syslog-ng.com/register/120335/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Need Assistance:error while loading shared libraries: libsyslog-ng-3.6.so.0
by Nitish Saboo 11 Jun '19

11 Jun '19

Hi, I am using syslog-ng-3.6.2 ( https://src.fedoraproject.org/lookaside/pkgs/syslog-ng/syslog-ng_3.6.2.tar.… ). I want to use syslog-ng header files in my project. I am calling syslog-ng code from my Go code. I compiled my source code in my local system ( go build -v -x main.go) and created a binary called 'main'. My binary is running successfully on my local system. When I copy the binary ('main') in another Linux box and try to run it as './main' it is giving me the following error: ubuntu@ingest-0693b91d:~$ ./main ./main: error while loading shared libraries: libsyslog-ng-3.6.so.0: cannot open shared object file: No such file or directory I have syslog-ng-3.6.2 shared files in the following location '/home/ubuntu/test/syslog/syslog-ng-3.6.2/install/lib/syslog-ng' and I have set LD_LIBRARY_PATH to {prefix}/lib/syslog-ng Can someone please guide me, why is it giving such error ? Thanks, Nitish

2 1

Generating configuration blocks for Option
by Sezer BAGLAN 23 May '19

23 May '19

Dear all, In Syslog-ng Administration Guide document there is a statement give as below: "The context option determines the type of the configuration snippet that the script generates, and must be one of the following: destination, filter , log, parser, rewrite, root, source. " >From this statement, we understand that we cannot use "option" type during auto generation of conf. blocks. Why do not we have "Option" type? How can we generate "option" blocks via scripts? Kind regards, Sezer

2 1

snmptrap handler parses SNMP traps correctly, but the file it reads from is not emptying
by Allen Pouratian 21 May '19

21 May '19

Hello - I set up snmptrapd to write to /var/log/snmptrapd.log and syslog-ng is configured to read from it like this ... source { snmptrap(filename("/var/log/snmptrapd.log")); }; ... but /var/log/snmptrapd.log is not getting emptied, so it needs to be truncated every so often to avoid filling up the filesystem. But the problem with truncating /var/log/snmptrapd.log is that we're going to lose traps, since we get a lot of them. When I was testing this syslog-ng snmptrap facility with a few traps a minute, I saw /var/log/snmptrapd.log empty, but with hundreds of traps incoming per second, it does not empty any more. I have what looks like a work-around to the syslog-ng snmptrap handler/parser where ... 1. 1) Snmptrapd 5.7.3 writes to syslog with -Lsd 2. 2) Syslog-ng 3.20 reads from syslog with system-journal() 3. 3) Rewrite the trap $MESSAGE with a series of "substitutions" (subst) into space separated key=value pairs 4. 4) point kv-parser() at $MESSAGE and specify " " as a separator 5. 5) delete the original $MESSAGE block ... but perhaps I didn't have to do that, since perhaps I'm mis-using the snmptrap facility built into syslog-ng, and thus causing /var/log/snmptrapd.log to not empty. Your comments and insights would be appreciated. - Allen -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Altice USA and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. --------------------------------------------------------

2 3

Forum upcoming on syslog-ng.com
by Hunor Voith (hvoith) 21 May '19

21 May '19

Dear syslog-ng users, The syslog-ng mailing list has been around for two decades to be a channel for newcomers, enthusiasts and developers to discuss their syslog-ng related topics. It gave place to many interesting and useful exchanges; however, it is also a semi-closed channel, lacking real public visibility. In this email, we would like to share some information about an upcoming forum on syslog-ng.com, and the future of this mailing list. We long wanted syslog-ng.com to be a sort of hub for everything syslog-ng related. A forum would be a great place to act as an ever-expanding knowledge base, complementing the extensive technical documentation of syslog-ng. Currently we have the forum up on the syslog-ng.com staging, and we would like to ask you to give it a spin and provide constructive feedback. To get access, please send an email to hunor.voith@<mailto:hunor.voith@oneidentity.com>oneidentity.com and I will share the access details with you. To pre-populate the forum, previous mailing list content has been (and will be, to an extent) migrated as threaded topics. We hope that the new platform will encourage more users to actively take part in syslog-ng related conversations. Of course, we will promote it once it is up and running to raise awareness among those who are currently not part of the mailing list community. We cannot share an exact go-live date with you yet, however, you can expect to see it on the live site in a couple weeks. We will keep you posted. What happens to the mailing list? The mailing list would be phased out after the forum goes live. The team is open to constructive feedback and there will no doubt be enhancements further down to road. -- Hunor Voith Product Marketing Manager @ syslog-ng

3 3

Fedora i386
by Czanik, Péter 21 May '19

21 May '19

Hi, Anyone still using syslog-ng on Fedora i386? I ran into some compile problems recently. If there is someone still using it, I can put some effort into a workaround (most likely disabling Java support on i386). If there is no positive response until the end of this week, I'll simply drop i386 platform support from my Copr repositories. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.21.1 RPM packages
by Czanik, Péter 21 May '19

21 May '19

Hi, The initial version of my syslog-ng 3.21.1 RPM packages for Fedora / RHEL 7 and openSUSE / SLES are now ready. Right now they are available in my git snapshot repositories. Learn how to install them from my blog at: https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-n… Unlike the development snapshots these should not eat your server for breakfast, but still they went through only minimal testing. This is especially true about the Riemann and Kafka destinations, where I packaged the latest versions of external dependencies to make sure that microsecond precision and Kafka are available. I don't have the test environment for them. Please test the packages and let me know not just if you ran into problems, but also your success stories. I'm still traveling. I plan to create a dedicated 3.21 repo once I'm back to Hungary and read some positive feedback from you! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Multiple log statements vs If/else
by Faine, Mark R. (MSFC-IS40)[NICS] 20 May '19

20 May '19

Is there any functional difference between something like: log { .... }; log { ... flags(final); }; and log { if { ... } else { ... }; I was thinking about trying to build a configuration specific for an app from a Jinja2 template in Ansible and it seems like to me that if they aren't different it would be easier to do multiple log statements if generated dynamically. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

4 10

Systemd service times out for syslog-ng 3.16.1
by Ronald Fenner 16 May '19

16 May '19

I'm trying to switch from an init.d script to a system service and having problems getting the service to run. The syslog-ng service gets timed out and never starts up. My service looks like this: [Unit] Description="Syslog-ng an enhanced syslog" Documentation=man:syslog-ng(8) After=network.target ConditionFileNotEmpty=/opt/syslog-ng/etc/syslog-ng.conf [Service] Type=notify ExecStart=/opt/syslog-ng/sbin/syslog-ng --no-caps -p /var/run/syslogd.pid -rdvt ExecReload=/bin/kill -HUP $MAINPID StandardOutput=journal StandardError=journal Restart=on-failure Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java/jre/lib/amd64/server:$LD_LIBRARY_PATH [Install] WantedBy=multi-user.target My config looks like this: ############################################################################# # Default syslog-ng.conf.partial file which collects all local logs into a # single file called /var/log/messages. # @version: 3.16 @module "mod-java" @include "scl.conf" @include "conf.d/gc-parsers.conf" source s_internal {internal();}; source s_rtl_stream { unix-stream("/var/gamecircus/log/rtl-stream.sock" flags(no-parse) log_msg_size(1000000)); }; # Backup for when syslog-ng crashes and doesn't restart queues the messages source s_rtl_backup { file("/var/gamecircus/log/rtl-backup.log" flags(no-parse) log_msg_size(1000000)); }; destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("syslog-ng-`DeployedEnvironment`") ); }; destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("${topic}.${environment}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; destination d_syslog_ng { file("/var/log/syslog-ng"); }; parser p_json { json-parser(); }; log { source(s_internal); destination(d_syslog_ng); }; log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); }; log { source(s_rtl_backup); parser(p_json); destination(d_kafka_structured); }; # Build script will add stuff below for the specific build. ############################################################################### The logging that I turned on in the service above isbn't helping. Here's the last little bit May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217091] Add path to classpath: /opt/kafka/libs/kafka-streams-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217248] Add path to classpath: /opt/kafka/libs/rocksdbjni-5.0.1.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217309] Add path to classpath: /opt/kafka/libs/kafka-streams-examples-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.361062] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365718] Running application hooks; hook='1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365930] Running application hooks; hook='3' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366137] syslog-ng starting up; version='3.16.1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366480] Worker thread started; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service start operation timed out. Terminating. May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273058] Running application hooks; hook='4' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273074] syslog-ng shutting down; version='3.16.1' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.376882] Worker thread finished; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377030] Closing log transport fd; fd='92' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377186] Java machine free; May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.378858] Running application hooks; hook='5' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Failed to start "Syslog-ng an enhanced syslog". May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Unit syslog-ng.service entered failed state. May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service failed. Any suggestions on what me causing it to timeout? I'm new to Ssytemd but if I understand the notify type it would suggest that syslog-ng isn't notifying systemd that it's started. I'm not sure what would be holding it up. Any help is much appreciated. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 5

Rest2Syslog
by C B 15 May '19

15 May '19

hello all, just wanted to share with you a new python syslog-ng plugin I was working on to support ingestion of any REST API into any syslog destination. this is off course open source and available here - https://github.com/chenbekor/Rest2Syslog I'll be happy to receive any feedback/suggestion. Also - if any of you guys know of a public page with all available plugins/extensions please let me know so I can add this nice project to the list. Regards, Chen.

2 1

Re: [syslog-ng] Troubleshooting Question
by Walter Tienken 14 May '19

14 May '19

Hi, Yes, the em1 interface is for our front-end connectivity and the em2 interface is for the F5 LB where traffic is sent through. By dual stack, I assume you mean IPV4/IPV6 ... in that case, no, only IPV4. It looks like syslog-ng is listening on all interfaces: Here is the output of netstat -ulp: https://gist.github.com/MrTink76/ade770cfe6d497aa50fe820a9c438f54 The part that's throwing me off is "distinct" ... shouldn't that be 9999? You see it is listening on all interfaces and the netstat -anp | grep 9999 here as well: https://gist.github.com/MrTink76/7741d90831fe4f331abe4630f3a8b29e Thanks much, Walter Tienken walter.tienken(a)asu.edu On 5/12/19, 8:56 AM, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: Does your box have multiple interfaces? Does it run a dual-stack? If yes, does the program (nc or syslogng) listen on the correct interface? (netstat -ulp)

2 1

How to create two independent patternDB instances in syslog-ng
by Nitish Saboo 13 May '19

13 May '19

I understand PatternDB is a syslog-ng pluginand its dynamic library can be found in as libdbparser.so. I went through the following tests in ' https://github.com/balabit/syslog-ng/blob/master/modules/dbparser/patterndb… ' I want to create two independent pattern_db instances having two different ruleset. My queries: 1) How can I create two independent instances of patternDB having two different ruleset ? 2)How can I get the handle of these two independent PatternDB instances ? Can someone please help me on this ? Thanks

1 0

Is it possible to create Two instances of syslog-ng engine ?
by Nitish Saboo 13 May '19

13 May '19

Hi, I have installed syslog-ng on my Linux box and I am planning you use syslog-ng parser and wanted to initialise it's engine for parsing the data. so initialise() method in my Go code calls C code that is wrapper around syslog-ng C header files. So ,I am making use of cgo. Following is my Go code: func (obj Syslogparser) initialise{ X := C.CString(obj.X) defer C.free(unsafe.Pointer(X)) Y := C.CString(obj.Y) defer C.free(unsafe.Pointer(Y)) C.initialize_engine(X, Y, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo))); } And following is my C method 'initialize_engine': int initialize_engine(const gchar* X, const gchar* Y, key_value_cb cb) // cb is callback function { module_path = _module_path; app_startup(); msg_init(TRUE); configuration = cfg_new(0x0302); plugin_load_module("basicfuncs", configuration, NULL); plugin_load_module("syslogformat", configuration, NULL); pattern_db_global_init(); if(patterndb != NULL){ pattern_db_free(patterndb); } patterndb = pattern_db_new(); pattern_db_reload_ruleset(patterndb, configuration, filename); pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb); return 0; } My queries: 1)What is pattern_db_global_init, 'patterndb = pattern_db_new()', 'pattern_db_reload_ruleset', 'pattern_db_set_emit_func' doing in the following case ? 2)Can I make two independent instances of syslog-ng engines? 3)How can I call two instances of syslog-ng engines independently from my Go code? Thanks

2 2

Troubleshooting Question
by Walter Tienken 12 May '19

12 May '19

Hello All, I'd like to see if we can get some troubleshooting help with Syslog-NG OSE. Here's some background: Our environment collects logs from various network locations (F5, checkpoint, plixer, etc) and sends them to our Syslog-NG cluster through two F5 load balancers. We have four netlog boxes in round-robin that are running version 5 of the PE version of Syslog-NG on RHEL6. We also have a dev server running rhel7 and the 3.19.1 OSE edition of Syslog-NG from the COPR repository. On each of these servers, we run the Splunk Universal Forwarder that then sends the logs over to our indexer cluster. The problem we are having is that the dev server will not listen to any traffic except via localhost. We can see the traffic just fine on a TCPDUMP as it comes out of the load balancer into the dev box, but watching Syslog-NG in the foreground with -Fevd the traffic never registers at all. We can send test messages with loggen or netcat, etc, from localhost and Syslog-NG will see it and log it to disk as expected. Anywhere else it just never sees the traffic nor logs anything to disk even though we have confirmed that Syslog-NG is listening on the 9999 port to UDP with netstat. Here is our syslog-ng.conf file from the working RHEL6 boxes: https://gist.github.com/MrTink76/9ee1e88f93a313f953e4033560af463a This is the syslog-ng.conf file on the OSE box: https://gist.github.com/MrTink76/e181f4c0bf052077440d7bdfaf418e02 This is an example of our testing CONF file located in conf.d on the OSE box: https://gist.github.com/MrTink76/a0433b9e908ba683e36cb6199f9cc43f We send our test traffic to the F5 load balancer vip using UDP 9999. Like I said above, when we send on 9999 localhost with loggen or netcat, Syslog-NG sees it just fine and logs it to disk, but anywhere else it never registers nor records the test message to disk. We currently have SELinux disabled and there is no firewall running on the dev box (we see the traffic fine via tcpdump). Any help/suggestions would be greatly appreciated. Please let me know if I need to provide further information. Thanks much, Walter Tienken walter.tienken(a)asu.edu<https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeB…> Cloud and Advanced Network Engineering Services

3 4

unofficial syslog-ng packages for syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

Hi, syslog-ng 3.21.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 (dependencies are still building) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

3.21.1 Highlights * Add an alternative, native, librdkafka based kafka-c() destination in parallel of the existing Java implementation, that provides the same configuration interface. Eventually, we expect this to replace the Java one (#2496<https://github.com/balabit/syslog-ng/pull/2496>) * Add a native, http() based destination based driver for elasticsearch called elasticsearch-http(), as an alternative of the Java one. Eventually, we expect this to replace the Java implementation. (#2509<https://github.com/balabit/syslog-ng/pull/2509>) * Add the ability to automatically determine the timezone value for an incoming log entry as long as the incoming stream is close to real time and the timezone information is missing from the timestamp. Enable this function by using flags(guess-timezone) for sources and the date-parser(). (#2517<https://github.com/balabit/syslog-ng/pull/2517>, #2673<https://github.com/balabit/syslog-ng/pull/2673>) Features * syslog(): Add the ability to work with messages larger than log-msg-size() in the source driver by using the trim-large-messages(yes) option. The characters over the limit will be truncated. Previously messages longer than the limit caused the connection to be closed abruptly. (#2644<https://github.com/balabit/syslog-ng/pull/2644>) * amqp(): add support for heartbeats and the "external" authentication mechanism. (#2676<https://github.com/balabit/syslog-ng/pull/2676>, #2626<https://github.com/balabit/syslog-ng/pull/2626>) * graylog2(): add support for TLS and UDP. (#2657<https://github.com/balabit/syslog-ng/pull/2657>) * udp(): Add spoof-source-max-msglen() option to allow setting the maximum spoofed datagram size, which was hard-wired to 1024 previously. (#2535<https://github.com/balabit/syslog-ng/pull/2535>) * db-parser(): add an option program-template() that customizes the value used for matching the PROGRAM field. (#2651<https://github.com/balabit/syslog-ng/pull/2651>) * pdbtool: Add sort option to pdbtool merge (#2664<https://github.com/balabit/syslog-ng/pull/2664>) * $(implode) and $(explode): add template functions to split and join strings based on a simple separator. The exploded array is represented as a syslog-ng list that can be manipulated with the $(list-*) template functions. (#2700<https://github.com/balabit/syslog-ng/pull/2700>) * Add an --omit-empty-values option for value-pairs based destinations & template functions. (#2519<https://github.com/balabit/syslog-ng/pull/2519>) * grouping-by() parser: add sort-key() option (#2701<https://github.com/balabit/syslog-ng/pull/2701>) Support for non-syslog or non-standard formats in SCL * apache-accesslog-parser(): support for vhost:port as the first field in common/combined log formats (#2688<https://github.com/balabit/syslog-ng/pull/2688>) * Add application adapter for Junos classification (#2684<https://github.com/balabit/syslog-ng/pull/2684>) * Add parser and adapter for CheckPoint LogExporter output (#2665<https://github.com/balabit/syslog-ng/pull/2665>) Bugfixes * Fix race condition of idle timer and scheduled I/O job (#2650<https://github.com/balabit/syslog-ng/pull/2650>) * Few leaks find via sanitizer (#2696<https://github.com/balabit/syslog-ng/pull/2696>) * syslogformat: set $MSG even if the incoming message is empty (#2672<https://github.com/balabit/syslog-ng/pull/2672>) * Fix double-free error in logproto unit tests (#2662<https://github.com/balabit/syslog-ng/pull/2662>) * groupingby: identical persist name (#2659<https://github.com/balabit/syslog-ng/pull/2659>) * stats: deindex pruned counters/clusters (#2648<https://github.com/balabit/syslog-ng/pull/2648>) * Type hinting should not accept empty values (#2639<https://github.com/balabit/syslog-ng/pull/2639>) * app-parser, pseudofile: fix crash with grammar error (#2640<https://github.com/balabit/syslog-ng/pull/2640>) * python: set_timestamp normalization (#2643<https://github.com/balabit/syslog-ng/pull/2643>) * db-parser: fix memory leak (#2652<https://github.com/balabit/syslog-ng/pull/2652>) * grouping-by: use after free, memory leak, missing init calls of filters (#2655<https://github.com/balabit/syslog-ng/pull/2655>) * amqp: fixing double connect (#2660<https://github.com/balabit/syslog-ng/pull/2660>) * old style definition warning fixes (#2680<https://github.com/balabit/syslog-ng/pull/2680>) * Fix "!=" filter (#2683<https://github.com/balabit/syslog-ng/pull/2683>) * dbparser: fix memleak (#2706<https://github.com/balabit/syslog-ng/pull/2706>) * nondumpable-allocator: fixing mmap error handling (#2666<https://github.com/balabit/syslog-ng/pull/2666>) * Fix timeutils warning (#2604<https://github.com/balabit/syslog-ng/pull/2604>) * Fix old style include statement compatibility (#2600<https://github.com/balabit/syslog-ng/pull/2600>) * Fix config revert (threaded destinations) (#2596<https://github.com/balabit/syslog-ng/pull/2596>) * Add warning on old style include statement (#2592<https://github.com/balabit/syslog-ng/pull/2592>) Other changes * cfg-parser: add aliases for yesno (#2671<https://github.com/balabit/syslog-ng/pull/2671>) * Include json-c in the dist tarball (#2590<https://github.com/balabit/syslog-ng/pull/2590>) * cmake: disable_all_modules support (#2647<https://github.com/balabit/syslog-ng/pull/2647>) * Cmake clang sanitizer (#2562<https://github.com/balabit/syslog-ng/pull/2562>) * timeutils refactor (#2483<https://github.com/balabit/syslog-ng/pull/2483>) * Expedite threaded flush at reload (#2656<https://github.com/balabit/syslog-ng/pull/2656>) * elasticsearch2: Added deprecation warning (#2628<https://github.com/balabit/syslog-ng/pull/2628>) * Astyle fixes (#2624<https://github.com/balabit/syslog-ng/pull/2624>) * Force C99 with GNU (#2623<https://github.com/balabit/syslog-ng/pull/2623>) * Make rewording and other small edits to README (#2608<https://github.com/balabit/syslog-ng/pull/2608>) * Port tests to Criterion (#2607<https://github.com/balabit/syslog-ng/pull/2607>, #2661<https://github.com/balabit/syslog-ng/pull/2661>, #2621<https://github.com/balabit/syslog-ng/pull/2621>, #2620<https://github.com/balabit/syslog-ng/pull/2620>, #2619<https://github.com/balabit/syslog-ng/pull/2619>, #2618<https://github.com/balabit/syslog-ng/pull/2618>, #2617<https://github.com/balabit/syslog-ng/pull/2617>, #2616<https://github.com/balabit/syslog-ng/pull/2616>, #2615<https://github.com/balabit/syslog-ng/pull/2615>, #2599<https://github.com/balabit/syslog-ng/pull/2599>, #2594<https://github.com/balabit/syslog-ng/pull/2594>, #2593<https://github.com/balabit/syslog-ng/pull/2593>, #2591<https://github.com/balabit/syslog-ng/pull/2591>, #2586<https://github.com/balabit/syslog-ng/pull/2586>, #2584<https://github.com/balabit/syslog-ng/pull/2584>, #2583<https://github.com/balabit/syslog-ng/pull/2583>) * test_reliable_backlog: fix random failure (#2668<https://github.com/balabit/syslog-ng/pull/2668>) * Fix unit test with function pointer dereference in case of ASLR, Criterion (#2669<https://github.com/balabit/syslog-ng/pull/2669>) * test-stats-query: fix unit test (#2603<https://github.com/balabit/syslog-ng/pull/2603>) Notes to the developers * Version from git describe (#2627<https://github.com/balabit/syslog-ng/pull/2627>) * light: example-msg-generator support (#2571<https://github.com/balabit/syslog-ng/pull/2571>) * light: test app parser applications (#2686<https://github.com/balabit/syslog-ng/pull/2686>) * light: Switch to native logger (#2546<https://github.com/balabit/syslog-ng/pull/2546>) * light: Remove SetupTestcase() dependency (#2587<https://github.com/balabit/syslog-ng/pull/2587>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Chris Spencer, David Liew, Fabien Wernli, Gabor Nagy, Laszlo Budai, Laszlo Szemere, Layne, László Várady, Mehul Prajapati, Nik Ambrosch, Parth Wazurkar, Péter Kókai, Terez Nemes, Victor Ma, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. - balabit/syslog-ng github.com

1 0

feature request idea
by Faine, Mark R. (MSFC-IS40)[NICS] 09 May '19

09 May '19

How hard would it be to make --syntax-only accept a path to a configuration file as an argument? This would be useful to me since I use an Ansible template module to install the file and it has a validate parameter but it requires an argument for the filename. It could of course still function normally If no path to a configuration file was provided. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

2 2

UDP alerts in syslog
by Swordfish Binary 09 May '19

09 May '19

Hi, I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes. we have syslog version 3.5.6. Questions - Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make? Environment: - RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk Errors: Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors - The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors. Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };

5 4

GSoC Improve Developer Guide Project
by Victor Ma 09 May '19

09 May '19

Hi, I will be working on a plugin programming guide for this year's Google Summer of Code. It will cover source drivers, parsers, filter functions, and template functions. Each section will come with a hopefully-useful-on-its-own example plugin to aid in the explanation. My proposal can be found here: https://github.com/balabit/syslog-ng/wiki/GSoC-2019-Proposal:-Improve-Devel… I am a rising junior at the University of Toronto. I plan to graduate with a major in computer science and minors in philosophy and creative writing. Up-and-coming logger, Victor Ma

1 0

Information regarding Syslog-ng
by Nitish Saboo 08 May '19

08 May '19

Hi, I am not sure if this is the right forum to ask these generic questions.Let me know if I have to post these questions on some other forum to get the correct information. 1) What is the latest syslog-ng version ? 2) Does syslog-ng has an enterprise edition , I mean a licensed version or It has only community edition ? 3)Does the latest version contain all the bug-fixes ? Thanks, Nitish

2 1

Syslog-ng options bad_hostname & check_hostname
by Delon Lee Di Lun 08 May '19

08 May '19

Hi, Anybody used the syslog-ng options bad_hostname & check_hostname? How does that works? I have syslog-ng listening on UDP, and found that there are some gibberish logs in the folder. Was thinking if using the above two options would help reduce the gibberish. Yours Sincerely, Delon

2 4