- syslog-ng - lists.balabit.hu

drop log lines having keyword "INTERNET"
by Sachchidanand Upadhyay 04 Aug '22

04 Aug '22

Hi, I want to drop all log lines having keyword "INTERNET", I tried following scenario: scenario-1 filter f_log { facility(local3); }; filter f_nointernet { not message("INTERNET"); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; scenario-2 filter f_internet { message("INTERNET"); }; filter f_log { facility(local3); and not filter(f_internet); }; log { source(s_sys); filter(f_log); destination(d_log); }; scenario-3 filter f_log { facility(local3); }; filter f_internet { message("INTERNET"); }; filter f_nointernet { not filter(f_internet); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; But none of them is working. Please help. Regards, Sachchidanand

1 0

Kafka-c issues
by Sanjay Patel 15 Jul '22

15 Jul '22

I am trying to setup Syslog-NG t send output to Kakka. I am running ubuntu 18.04, Syslog-NG version 3.37. This is my syslog-ng.conf ( I have #scl.conf because if I don't it tries to install kafka-java) @version: 3.37 @define kafka-implementation kafka-c #@include "scl.conf" destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("{incoming}") ); }; This is the error when I try and run sudo syslog-ng Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/syslog-ng.conf:6:3-6:8: 1 @version: 3.37 2 @define kafka-implementation kafka-c 3 #@include "scl.conf" 4 5 destination d_kafka { 6-----> kafka( 6-----> ^^^^^ 7 bootstrap-servers("127.0.0.1:9092") 8 topic("{incoming}") 9 ); 10 }; Any help will be appreciated. SKP

4 8

Insider 2022-06: RHEL 9; disk-buffer; Microsoft Linux;
by Peter Czanik (pczanik) 14 Jul '22

14 Jul '22

Dear syslog-ng users, This is the 103rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS RHEL 9 syslog-ng news --------------------- Red Hat Enterprise Linux 9 became generally available recently. Version 3.35 of syslog-ng has been part of EPEL 9 (the semi-official extra software repo for RHEL maintained by Fedora packagers) for a while and now I enabled a few more destination drivers. I also enabled RHEL 9 support in my unofficial Git snapshot packages, so I can support RHEL 9 together with other RHEL and Fedora versions on the next syslog-ng release. https://www.syslog-ng.com/community/b/blog/posts/rhel-9-syslog-ng-news How does the syslog-ng disk-buffer work? ---------------------------------------- Last time, we had an overview of the syslog-ng disk-buffer. This time, we dig a bit deeper and take a quick look at how it works, and a recent major change that helped speed up the reliable disk-buffer considerably. https://www.syslog-ng.com/community/b/blog/posts/how-does-the-syslog-ng-dis… Installing syslog-ng on Microsoft Linux --------------------------------------- Yes, Microsoft has its own Linux distribution, called CBL-Mariner. It is an internal Linux distribution by Microsoft used for cloud infrastructure and edge products and services. And even if it is not installed in the OS by default, CBL-Mariner also includes syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-on-mi… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Missing Depends libjson-c4
by Joe Craig 06 Jul '22

06 Jul '22

The following packages have unmet dependencies: syslog-ng-core : Depends: libjson-c4 (>= 0.13.1) but it is not installable Depends: libssl1.1 (>= 1.1.1) but it is not installable >From research it looks like libjson-c4 has been replaced in the Debian repositories by libjson-c5 is there a way to utilize the libjson-c5 version or how should I resolve this issue? FYI this was working on Ubuntu 22.04 Jammy up until the past week. I ran updates on the system to resolve another issue and after that I'm getting an error starting the daemon × syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-07-05 15:39:48 UTC; 4s ago Docs: man:syslog-ng(8) Process: 1342 ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS (code=exited, status=2) Main PID: 1342 (code=exited, status=2) Status: "Starting up... (Tue Jul 5 15:39:48 2022" CPU: 29ms Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jul 05 15:39:48 <hostname> systemd[1]: Failed to start System Logger Daemon. Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 5. Jul 05 15:39:48 <hostname> systemd[1]: Stopped System Logger Daemon. Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Start request repeated too quickly. Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jul 05 15:39:48 <hostname> systemd[1]: Failed to start System Logger Daemon. The rabbit hole lead me to this current state. Any assistance would be appreciated. Joseph Craig

2 2

Environment variables in syslog
by Faisal Chishti 28 Jun '22

28 Jun '22

Hellow, Please see below the snippet from my syslog-ng.conf file. ABC_LOG_PATH is the environment variable on the host that contains the path. I am trying to access the env variable within the config. It does not throw any error and the configuration is valid but it does not work properly. I don't see the file agent.log being used at all. source s_agent { file("`ABC_LOG_PATH`/agent.log" follow-freq(1) flags(no-parse)); }; root@syslog-ng-784fdc497-pgdzr:/# syslog-ng --version syslog-ng 3 (3.30.1) Config version: 3.29 Installer-Version: 3.30.1 Revision: Compile-Date: Apr 12 2022 20:15:37 Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: add-contextual-data,afamqp,affile,afprog,afsocket,afsql,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,mod-python,pseudofile,redis,secure-logging,stardate,syslogformat,system-source,tags-parser,timestamp,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off root@syslog-ng-784fdc497-pgdzr:/# uname -a Linux syslog-ng-784fdc497-pgdzr 4.18.0-147.3.1.el8_1.x86_64 #1 SMP Fri Jan 3 23:55:26 UTC 2020 x86_64 GNU/Linux root@syslog-ng-784fdc497-pgdzr:/# -- Regards, Faisal

2 1

I am a novice on syslog-ng needs help.
by Alexandre Adao 20 Jun '22

20 Jun '22

Hello, I am a novice on syslog-ng. I need some assistance on installing syslog-ng and looking for a good recommendation on step-by-step instructions for Ubuntu (prefered OS) or Windows environment. I googled it but I My goal is to send the logs to splunk. Any help would be appreciated. Thanks, --Alex

3 3

Insider 2022-06: 3.37; hardware; Apache; disk-buffer;
by Peter Czanik (pczanik) 16 Jun '22

16 Jun '22

Dear syslog-ng users, This is the 102nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.37 released ----------------------- Balázs Scheidler, founder of the syslog-ng project, describes the latest syslog-ng release: “syslog-ng 3.37 has just been released, packages available in various platforms this week. You can get the detailed release notes on the github releases page, however I felt this would be a good opportunity to revisit my draft on the syslog-ng long term objectives and how this release builds in that direction.” Read the rest of the blog at https://syslog-ng-future.blog/syslog-ng-3-37-released/ When (not) to use the syslog-ng disk-buffer ------------------------------------------- The syslog-ng disk buffer is one of the most often used syslog-ng options to ensure message delivery. However, it is not always necessary and using the safest variant has serious performance impacts. If you utilize disk-buffer in your syslog-ng configuration, it is worth to make sure that you use a recent syslog-ng version. >From this blog, you can learn when to use the disk-buffer option, the main differences between reliable and non-reliable disk-buffer, and why is it worth to use the latest syslog-ng version. https://www.syslog-ng.com/community/b/blog/posts/when-not-to-use-the-syslog… Hardware for a syslog-ng server ------------------------------- What hardware to use for a syslog-ng server? It is a frequent question with no definite answer. It depends on many factors: the number and type of sources, the number of logs, the way logs are processed, and so on. My experience is that for the majority users even a Raspberry Pi would be enough. But of course, not for everyone. https://www.syslog-ng.com/community/b/blog/posts/hardware-for-a-syslog-ng-s… Analyzing Apache HTTPD logs in syslog-ng ---------------------------------------- Recently, I started my own blog, and as Google Analytics seems to miss a good part of visitors, I wanted to analyze my web server logs myself. I use syslog-ng to read Apache logs, process them, and store them to Elasticsearch. Along the way, I resolve the IP address using a Python parser, analyze the Agent field of the logs, and also use GeoIP to locate the user on the map. >From this blog, you can learn how I built my configuration. Note that once I was ready, I realized that my configuration is not GDPR compliant, so I also show you which parts to remove from the final configuration :-). https://www.syslog-ng.com/community/b/blog/posts/analyzing-apache-httpd-log… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Test email.
by Tamás Pál 12 Jun '22

12 Jun '22

Please ignore this email.

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 206, Issue 2
by Александр Масленников 10 Jun '22

10 Jun '22

i'm not sure that *format-flat-json* works with lists same as with nested dicts. My example uses format-flat-json, but without an additional filter on pothon, I was unable to flatten it There is original message { "operation": "CLIENT_RESPONSE", "identity": "ns-server.example.com", "family": "INET", "protocol": "TCP", "query-ip": "192.168.xx.zz", "query-port": "51000", "response-ip": "192.168.yy.zz", "response-port": "53", "length": 691, "rcode": "NOERROR", "qname": "_dnsaddr.bootstrap.libp2p.io", "qtype": "TXT", "latency": "0.000000", "timestamp-rfc3339": "2022-06-06T08:47:58.797332215Z", "answers": [{ "name": "_dnsaddr.bootstrap.libp2p.io", "rdatatype": "TXT", "ttl": 600, "rdata": "dnsaddr=/dnsaddr/ams-2.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb" }, { "name": "_dnsaddr.bootstrap.libp2p.io", "rdatatype": "TXT", "ttl": 600, "rdata": "dnsaddr=/dnsaddr/sjc-1.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN" }, { "name": "_dnsaddr.bootstrap.libp2p.io", "rdatatype": "TXT", "ttl": 600, "rdata": "dnsaddr=/dnsaddr/ewr-1.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa" }, { "name": "_dnsaddr.bootstrap.libp2p.io", "rdatatype": "TXT", "ttl": 600, "rdata": "dnsaddr=/dnsaddr/ams-rust.bootstrap.libp2p.io/p2p/12D3KooWEZXjE41uU4EL2gpkAQeDXYok6wghN7wwNVPF5bwkaNfS" }, { "name": "_dnsaddr.bootstrap.libp2p.io", "rdatatype": "TXT", "ttl": 600, "rdata": "dnsaddr=/dnsaddr/sjc-2.bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp" }, { "name": "_dnsaddr.bootstrap.libp2p.io", "rdatatype": "TXT", "ttl": 600, "rdata": "dnsaddr=/dnsaddr/nrt-1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt" }], "country-isocode": "-" } I would be very grateful to you if you have a solution using the built-in functions of syslog-ng. пт, 10 июн. 2022 г. в 15:00, <syslog-ng-request(a)lists.balabit.hu>: > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. need help with parser to make flat nested json list of > dictionaries (????????? ???????????) > 2. Re: need help with parser to make flat nested json list of > dictionaries (Peter Kokai (pkokai)) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 10 Jun 2022 11:02:55 +0300 > From: ????????? ??????????? <alexander.a.maslennikov(a)gmail.com> > To: syslog-ng(a)lists.balabit.hu > Subject: [syslog-ng] need help with parser to make flat nested json > list of dictionaries > Message-ID: > <CA+G0nAjp1b6_50LbCROVPje1_B4R_AzNYiZ-_dT0m= > fXcqwmHA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > hi all > i have a json message that contains a nested json list of dicts > > {"a":1,"b":[{"c":1},{"c":2},{"c":3}]} > > i want to flat that message, so expected result looks like { > "a": 1, > "b_0_c": 1, > "b_1_c": 2, > "b_2_c": 3 > } > > My approach is a python implemented parser. > Is it possible to achieve the same result using the built-in syslog-ng > tools? > My solution below > > @define kafka-implementation kafka-c > > python { > > import collections > import json > > class FlattenedJson(object): > > def parse(self, log_message, flat_message=None): > def flatten(d, parent_key='', sep='_'): > items = [] > for k, v in d.items(): > new_key = parent_key + sep + k if parent_key else k > if isinstance(v, collections.MutableMapping): > items.extend(flatten(v, new_key, sep=sep).items()) > elif isinstance(v, list): > for idx, value in enumerate(v): > items.extend(flatten(value, new_key + sep + > str(idx), sep).items()) > else: > items.append((new_key, v)) > return dict(items) > try: > decoded_msg = > json.loads(log_message['MESSAGE'].decode('utf-8')) > flat_message = flatten(decoded_msg) > final_message = > str(json.dumps(flat_message)).encode(encoding='utf-8') > log_message['MESSAGE'] = final_message > except Exception as error: > log_message['python_error'] = 'An exception occurred: > {}'.format(error) > return True > }; > > destination d_kafka_dnstap { > kafka( > topic("mytopic") > bootstrap-servers("localhost:9092") > message("$(format-flat-json --scope all-nv-pairs > application_name=myapp @timestamp=${ISODATE} )") > ); > }; > > source s_net_dnstap { network( transport(udp) port(514) flags(no-parse) ); > }; > > parser p_dnstap { channel { > parser { python(class("FlattenedJson")); }; > parser { json-parser(prefix("dnstap.")); }; > }; > }; > > log { source(s_net_dnstap); parser(p_dnstap); destination(d_kafka_dnstap); > }; >

2 1

need help with parser to make flat nested json list of dictionaries
by Александр Масленников 10 Jun '22

10 Jun '22

hi all i have a json message that contains a nested json list of dicts {"a":1,"b":[{"c":1},{"c":2},{"c":3}]} i want to flat that message, so expected result looks like { "a": 1, "b_0_c": 1, "b_1_c": 2, "b_2_c": 3 } My approach is a python implemented parser. Is it possible to achieve the same result using the built-in syslog-ng tools? My solution below @define kafka-implementation kafka-c python { import collections import json class FlattenedJson(object): def parse(self, log_message, flat_message=None): def flatten(d, parent_key='', sep='_'): items = [] for k, v in d.items(): new_key = parent_key + sep + k if parent_key else k if isinstance(v, collections.MutableMapping): items.extend(flatten(v, new_key, sep=sep).items()) elif isinstance(v, list): for idx, value in enumerate(v): items.extend(flatten(value, new_key + sep + str(idx), sep).items()) else: items.append((new_key, v)) return dict(items) try: decoded_msg = json.loads(log_message['MESSAGE'].decode('utf-8')) flat_message = flatten(decoded_msg) final_message = str(json.dumps(flat_message)).encode(encoding='utf-8') log_message['MESSAGE'] = final_message except Exception as error: log_message['python_error'] = 'An exception occurred: {}'.format(error) return True }; destination d_kafka_dnstap { kafka( topic("mytopic") bootstrap-servers("localhost:9092") message("$(format-flat-json --scope all-nv-pairs application_name=myapp @timestamp=${ISODATE} )") ); }; source s_net_dnstap { network( transport(udp) port(514) flags(no-parse) ); }; parser p_dnstap { channel { parser { python(class("FlattenedJson")); }; parser { json-parser(prefix("dnstap.")); }; }; }; log { source(s_net_dnstap); parser(p_dnstap); destination(d_kafka_dnstap); };

2 1