- syslog-ng - lists.balabit.hu

Can elasticsearch_http send data to ES datastreams ?
by Russell Fulton 19 Dec '20

19 Dec '20

Datastreams behave like indexes *except* when you use _bulk then instead of “index” you must use “create”. Datastreams are set up to handle timeseries log data and they make managing such data must easier. R

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 188, Issue 15
by Németh Balázs 19 Dec '20

19 Dec '20

Hi, You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.: https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/… Regards, Balázs <syslog-ng-request(a)lists.balabit.hu> ezt írta (időpont: 2020. dec. 19., Szo, 13:00): > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > Today's Topics: > > 1. elasticsearch and ILM (Russell Fulton) > > > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 01:40:42 +0000 > Subject: [syslog-ng] elasticsearch and ILM > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. > Most of the documentation seems to assume you are using filebeat, logstash > or datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell_______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

What am I doing wrong?
by Dan Egli 16 Dec '20

16 Dec '20

Help me understand this, please? I have ISC dhcpd configured to log to syslog.local7 (since I don't see an option to force it into it's own log file). So I went into my syslog-ng file and created two filters, just like on the example page of syslog-ng.com: filter dhcpmsgs { facility(23) ); filter non_dhcp { NOT filter(dhcpmsgs) ) I quoted almost directly from the example page on syslog-ng.com, but I keep getting this error when I reload syslog-ng's config: Error parsing filter expression, filter plugin NOT not found OR you may not used double quotes in your filter expression in /etc/syslog-ng/syslog-ng.conf:25:18-25:21: What did I do wrong? Here's the lines I modified from the syslog-ng page: filter demo_filter { host("example") and match("deny" value("MESSAGE")) }; filter inverted_demo_filter { NOT filter(demo_filter) } You can see the page at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… -- Dan Egli From my Test Server

2 6

syslog-ng and DTLS
by Secure-Logging, Generic 12 Dec '20

12 Dec '20

Dear syslog-ng users and developers, I know that syslog-ng is able to communicate using TCP and also supports TLS connections for improved security. Does syslog-ng also support support secure connections based UDP using DTLS? I am trying to implement a secure connection on a system with a network stack that does not support TCP because of resource constraints. Can I use syslog-ng for this? As syslog-ng security is based on OpenSSL and OpenSSL supports DTLS, I hope that this will be possible but I rely on your feedback concerning this topic. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

2 1

Insider 2020-12: web interfaces; Grafana Loki; Amazon Linux 2;
by Peter Czanik (pczanik) 10 Dec '20

10 Dec '20

Dear syslog-ng users, This is the 87th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Web interfaces for your syslog server – an overview --------------------------------------------------- This is the 2020 edition of my most read blog entry about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011, only a single logging as a service solution was available, while nowadays, I regularly run into others. Also, while some software disappeared, the number of logging-related GUIs is growing. This is why in this post, I will mostly focus on generic log management and open source instead of highly specialized software, like SIEMs. https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy… Sending logs from syslog-ng to Grafana Loki ------------------------------------------- Loki is one of the latest applications that lets you aggregate and query log messages, and of course to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC5424 syslog protocol. This is where syslog-ng can send its log messages. >From this blog, you can learn a minimal Loki & Promtail setup. We will send logs from syslog-ng, and as a first step, will check them with logcli, a command line utility for Loki. Once it works, we will also install Grafana in a container and query Loki from there. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… Installing syslog-ng in Amazon Linux 2, including Graviton2 ----------------------------------------------------------- Amazon Linux 2 is just one of many Linux distributions and other operating systems running in AWS. As Amazon Linux 2 is a close relative of RHEL 7 / CentOS 7, the user experience is also very close to these operating systems. All you need to learn is an extra command for enabling EPEL (Extra Packages for Enterprise Linux). Best of all: the very same commands work both on x86 and Graviton2 (Arm/Aarch64) instances. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-in-am… WEBINARS * Using syslog-ng with Google Pub/Sub: https://www.syslog-ng.com/event/using-syslogng-with-google-pubsub-8145754/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Poll: RHEL 6 / CentOS 6 packages
by Peter Czanik (pczanik) 09 Dec '20

09 Dec '20

Hi, Now, that CentOS 6 (RHEL 6) is end of life and removed from mirrors (https://twitter.com/Arrfab/status/1334064890261147650) it is time to think what to do with my unofficial CentOS 6 syslog-ng packages on Copr (https://copr.fedorainfracloud.org/coprs/czanik/) They definitely do not receive any updates. Personally I would delete the CentOS 6 syslog-ng repositories today to free up resources, but before doing so I am asking your opinion. The repositories will be deleted by Copr anyway in about half a year (180 days after a release becomes EoL). You can answer here or my poll on Twitter: https://twitter.com/PCzanik/status/1334071192748167168 Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Filter matching not working
by Saqib M 09 Dec '20

09 Dec '20

Greetings - I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m(a)cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.

4 5

Syslog-ng Windows Agent
by Nik Ambrosch 08 Dec '20

08 Dec '20

Hi, I was wondering if there is an OSE syslog-ng windows agent or if that's PE only? The "officially supported binaries" link is broken (bottom of the page) - https://www.syslog-ng.com/products/open-source-log-management/3rd-party-bin… We have some windows servers that I'd like to integrate into our log management system and the built-in syslog support is... not great. Thanks.

2 1

Don't change permissions
by Faine, Mark R. (MSFC-IS40)[NICS] 08 Dec '20

08 Dec '20

I'm monitoring the ACL on log files with monit and comparing them to a known good ACL to make sure everything is correct and running a correction script in response to a failure, my problem is that my compare keeps failing because syslog-ng is changing the effective permissions. Is there a way I can disable this? Thanks, -Mark

2 1

Re: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 01 Dec '20

01 Dec '20

Hello Laci, thanks for your support and sorry for the delay. After some debugs and installing the old version 3.5.6 on the host again, which was also not working I found the problem in the secondary ip address we used on the host. After removing the secondary address syslog-ng is working again without a problem. At the moment we will use this old version, but I will try to use the docker version on a test system to check if it works again and we can use the new features of the 3.29 version. Thanks again and kind regards Kind regards Uwe Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 13. November 2020 12:04 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng) CAUTION: This e-mail is from an external sender. Do not click links or open attachments unless you recognize the sender and know the content is safe. Never provide your password! Hello Uwe, first of all: thank you for sharing your config and exact Syslog-ng versions. It helps a lot. At first look I cannot see any problem with your config. However, there were a lot of changes between version 3.5.6 and 3.26.1 (You might want to take a look at "cisco-parser()") If you suspect this is a filter problem, may I suggest starting Syslog-ng in a debug mode? example: https://github.com/balabit/syslog-ng-docker/issues/58#issuecomment-680674916 In this case you should see messages like: >>>>>> filter rule evaluation begin; rule='foobar', location='/conf/syslog-ng.conf:15:16', msg='0x7efd38015c40' <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='foobar', location='/conf/syslog-ng.conf:15:16', msg='0x7efd38015c40' If this does not help finding the problematic filter, I might need some example logs to trace down any parsing issues. Best regards, Laci ________________________________ From: syslog-ng on behalf of Martin, Uwe Sent: Friday, November 13, 2020 11:15 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng) CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe Festo SE & Co. KG Uwe Martin Abteilung IM-IHN IT Communication Networks Gottlieb-Stoll-Straße 29 66386 St. Ingbert Deutschland Telefon +49(6894)591-6323 Telefax +49(711)34754-6323 http://www.festo.com Der Inhalt dieser E-Mail und moeglicher Anhaenge sind ausschliesslich fuer den bezeichneten Adressaten bestimmt. Jede Form der Kenntnisnahme, Veroeffentlichung, Vervielfaeltigung oder Weitergabe des Inhalts dieser E-Mail und moeglicher Anhaenge durch unberechtigte Dritte ist unzulaessig. Wir bitten Sie, sich mit dem Absender der E-Mail in Verbindung zu setzen, falls Sie nicht der Adressat dieser E-Mail sind sowie das Material von Ihrem Computer zu loeschen. This e-mail and any attachments are confidential and intended solely for the addressee. The perusal, publication, copying or dissemination of the contents of this e-mail by unauthorised third parties is prohibited. If you are not the intended recipient of this e-mail, please delete it and immediately notify the sender. Rechtsform: Kommanditgesellschaft, Sitz: Esslingen a.N., Registergericht Stuttgart HRA 211583, Umsatzsteuerident-Nummer: DE 145339206 Persoenlich haftende Gesellschafterin: Festo Management SE, Sitz: Frankfurt a.M., Registergericht Frankfurt a.M., HRB 115998 Vorstand: Dipl.-Ing. Gerhard Borho, Dipl.-Ing. Dr. h.c. Oliver D. Jung (Vorsitzender), Dr. Ansgar Kriwet, Dr. Frank Melzer, Dipl.-Ing. (FH) Frank Notz, Dr. Jaroslav Patka Aufsichtsratsvorsitzender: Dr. Friedrich Eichiner Bitte beachten Sie: die Festo AG & Co. KG firmiert seit 31.01.2020 unter Festo SE & Co. KG. Please note, on 2020-01-31 Festo AG & Co. KG changed its company name to Festo SE & Co. KG.

1 0