- syslog-ng - lists.balabit.hu

Polling additional machines
by Dan Egli 23 Jan '21

23 Jan '21

I have an interesting setup that was proposed to me, and while I _THINK_ syslog-ng can handle it I don't know for sure, let alone HOW. There are multiple service-specific machines in the network, behind the firewall. I'll just use the services they provide as their hostnames in this example. So firewall fw is a small linux machine running syslog-ng and fail2ban. Now fail2ban works off of the log files, but it also wants to apply changes to the firewall rules locally. So either www has its firewall rules while smtp has its rules and imap has a third set (ugly) or fail2ban can run on fw and modify the rules there. Trick is, in order to do that, all the logs have to be accessible in real time so that fail2ban can see them. My thought was to have syslog-ng on each sub machine somehow report to the syslog-ng on fw. Then fw's syslog-ng can write the files that fail2ban wants to read from. What would be the best way to go about this? I am still learning syslog-ng so if this is something simple, I'm sorry.

2 1

syslog-ng on CentOS 7 Aarch64
by Peter Czanik (pczanik) 20 Jan '21

20 Jan '21

Hi, I just learned that RHEL 7 on Aarch64 (64bit Arm) is end of life. Copr, where I build syslog-ng packages, removes RHEL/CentOS 7 Aarch64 packages in about half a year and it is not possible to build new packages there any more. CentOS 7 on the other hand keeps going on Aarch64 (using CentOS 8 kernel and CentOS 7 userland). There might be a workaround for the situation (using a different build service), but it's quite a lot extra effort. So, before investing any time and effort into keep building CentOS 7 Aarch64, I want to know, if there is a significant number of syslog-ng users, who keep using CentOS 7 on Aarch64. You can respond here or to my Twitter poll at: https://twitter.com/PCzanik/status/1351816930345291776 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Signed Syslog Messages RFC 5848
by Alexandre Santos 20 Jan '21

20 Jan '21

Hi, Are there any plans to add Signed Syslog Messages RFC 5848 to syslog-ng? Best Regards, Alex

3 4

Signed Syslog Messages RFC 5848
by Secure-Logging, Generic 19 Jan '21

19 Jan '21

Hi Alexandre, as a replacement for signed messages you might have a look at the secure logging module of syslog-ng. This module authentically encrypts each message, so you get an integrity guarantee which is stronger than a pure signature. You can then forward the message to a backend for verification and further processing. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

Failover/Hi Availability
by Faine, Mark R. (MSFC-IS40)[NICS] 18 Jan '21

18 Jan '21

If I have two Syslog-ng servers receiving logs from the same source and writing to the exact same file system destination, is there a way to do this without running into issues with the two instances clobbering the files or file locking issues? Thanks, -Mark

2 4

Filtering Destination by Source
by Peter Griggs 12 Jan '21

12 Jan '21

Hello, We have a lot of network logs all being pointed to a central syslog however this is a mix of vendors (Cisco / Juniper / Checkpoint) etc. is there a way of splitting the destination file by vendor type / or source IP address? We ingest this data into Splunk so want to get the source typing right however I am unable to get the sources to point to various listeners and I would prefer. Thanks Peter.

3 3

Disk file truncation issue
by Raghunath Adhyapak 20 Dec '20

20 Dec '20

Hi, We have configured the destination with reliable(yes). We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated. This syslog-ng service is running on 18.04 ubuntu VM. =============================================== Our destination configuration is as below: destination d_msg_syslog_01a_06 { tcp( "172.XX.XX.XX" port(514) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(50000) # throttle to max 5k eps throttle(3000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(75161927680) # 70GB reliable(yes) # directory location to persist messages dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/") ) persist-name(d_msg_syslog_01a_06) ); }; =============================================== $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf Here are log snippets when we ran syslog-ng in debug mode. =============================================== [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat' [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846' [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' =============================================== We are unable to figure out why the disk buffer file is not getting truncated. We have restarted syslog-ng multiple times. Please advise. Thanks Raghu

2 3

Re: [syslog-ng] elasticsearch and ILM
by Németh Balázs 20 Dec '20

20 Dec '20

Hi, Originally I responsed to a digest message so you might did not get it. You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.:https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsea… Regards, Balázs > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 21:47:12 +0000 > Subject: Re: [syslog-ng] elasticsearch and ILM > Russell wishes to delete this post :) > > Russell did what he should have done before posting! He ran tcpdump and > found the error response from ES! > > Apologies for the noise. The original question is still open! > > On 20/12/2020, at 10:38 AM, Russell Fulton <r.fulton(a)auckland.ac.nz> > wrote: > > Caution - Forged Internal Domain! > This e-mail cannot be validated and may not have been sent by the sender > shown in the 'From' field. > If you were not expecting to receive this e-mail we recommend you contact > the sender to confirm that they sent it. > If you believe this email was legitimately sent, we suggest the sender > notify the Staff Service Centre that it has been received as a forged > (fake) e-mail. > Please contact the Staff Service Centre on extension 86000 if you require > further assistance. > > > > On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: > > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. Most > of the documentation seems to assume you are using filebeat, logstash or > datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell > > > This is getting weirder ! > > ES is now failing to create my new index at all in spite of the > destination definition being practically identical to one that works: > > destination d_auth_elastic { > elasticsearch_http( > index("auth_${YEAR}.${MONTH}.${DAY}") > type("_doc") > persist-name("auth") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR > > > --key ISODATE)\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > destination d_authm_elastic { > elasticsearch_http( > index("auth-000001") > type("") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR )\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > > > log { > source( s_loghost ); > parser( p_patterns ); > filter( f_classified ); > filter( f_ping999 ); > filter( f_forti_stats ); > rewrite( r_user1 ); > rewrite( r_user2 ); > rewrite( r_srcip ); > parser( p_srcip ); > rewrite( r_srcip_country ); > rewrite( r_srcip_city); > rewrite( r_src_loc); > rewrite( r_ISODATE ); > destination( d_auth_elastic ); > destination( d_authm_elastic); > flags( flow-control); > }; > > d_auth_elastic works fine but d_authm_elastic never gets created. No > errors on the ES end. > > Anyone have any clues as to what is going on? > > I am at the point of starting tcpdump and looking at what is going over > the network. > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

elasticsearch and ILM
by Russell Fulton 19 Dec '20

19 Dec '20

I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic. The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams. So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”. If anyone has a working set up that they are happy to share it would be wonderful. Russell

1 2

Can elasticsearch_http send data to ES datastreams ?
by Russell Fulton 19 Dec '20

19 Dec '20

Datastreams behave like indexes *except* when you use _bulk then instead of “index” you must use “create”. Datastreams are set up to handle timeseries log data and they make managing such data must easier. R

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 188, Issue 15
by Németh Balázs 19 Dec '20

19 Dec '20

Hi, You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.: https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/… Regards, Balázs <syslog-ng-request(a)lists.balabit.hu> ezt írta (időpont: 2020. dec. 19., Szo, 13:00): > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > Today's Topics: > > 1. elasticsearch and ILM (Russell Fulton) > > > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 01:40:42 +0000 > Subject: [syslog-ng] elasticsearch and ILM > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. > Most of the documentation seems to assume you are using filebeat, logstash > or datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell_______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

What am I doing wrong?
by Dan Egli 16 Dec '20

16 Dec '20

Help me understand this, please? I have ISC dhcpd configured to log to syslog.local7 (since I don't see an option to force it into it's own log file). So I went into my syslog-ng file and created two filters, just like on the example page of syslog-ng.com: filter dhcpmsgs { facility(23) ); filter non_dhcp { NOT filter(dhcpmsgs) ) I quoted almost directly from the example page on syslog-ng.com, but I keep getting this error when I reload syslog-ng's config: Error parsing filter expression, filter plugin NOT not found OR you may not used double quotes in your filter expression in /etc/syslog-ng/syslog-ng.conf:25:18-25:21: What did I do wrong? Here's the lines I modified from the syslog-ng page: filter demo_filter { host("example") and match("deny" value("MESSAGE")) }; filter inverted_demo_filter { NOT filter(demo_filter) } You can see the page at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… -- Dan Egli From my Test Server

2 6

syslog-ng and DTLS
by Secure-Logging, Generic 12 Dec '20

12 Dec '20

Dear syslog-ng users and developers, I know that syslog-ng is able to communicate using TCP and also supports TLS connections for improved security. Does syslog-ng also support support secure connections based UDP using DTLS? I am trying to implement a secure connection on a system with a network stack that does not support TCP because of resource constraints. Can I use syslog-ng for this? As syslog-ng security is based on OpenSSL and OpenSSL supports DTLS, I hope that this will be possible but I rely on your feedback concerning this topic. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

2 1

Insider 2020-12: web interfaces; Grafana Loki; Amazon Linux 2;
by Peter Czanik (pczanik) 10 Dec '20

10 Dec '20

Dear syslog-ng users, This is the 87th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Web interfaces for your syslog server – an overview --------------------------------------------------- This is the 2020 edition of my most read blog entry about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011, only a single logging as a service solution was available, while nowadays, I regularly run into others. Also, while some software disappeared, the number of logging-related GUIs is growing. This is why in this post, I will mostly focus on generic log management and open source instead of highly specialized software, like SIEMs. https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy… Sending logs from syslog-ng to Grafana Loki ------------------------------------------- Loki is one of the latest applications that lets you aggregate and query log messages, and of course to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC5424 syslog protocol. This is where syslog-ng can send its log messages. >From this blog, you can learn a minimal Loki & Promtail setup. We will send logs from syslog-ng, and as a first step, will check them with logcli, a command line utility for Loki. Once it works, we will also install Grafana in a container and query Loki from there. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… Installing syslog-ng in Amazon Linux 2, including Graviton2 ----------------------------------------------------------- Amazon Linux 2 is just one of many Linux distributions and other operating systems running in AWS. As Amazon Linux 2 is a close relative of RHEL 7 / CentOS 7, the user experience is also very close to these operating systems. All you need to learn is an extra command for enabling EPEL (Extra Packages for Enterprise Linux). Best of all: the very same commands work both on x86 and Graviton2 (Arm/Aarch64) instances. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-in-am… WEBINARS * Using syslog-ng with Google Pub/Sub: https://www.syslog-ng.com/event/using-syslogng-with-google-pubsub-8145754/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Poll: RHEL 6 / CentOS 6 packages
by Peter Czanik (pczanik) 09 Dec '20

09 Dec '20

Hi, Now, that CentOS 6 (RHEL 6) is end of life and removed from mirrors (https://twitter.com/Arrfab/status/1334064890261147650) it is time to think what to do with my unofficial CentOS 6 syslog-ng packages on Copr (https://copr.fedorainfracloud.org/coprs/czanik/) They definitely do not receive any updates. Personally I would delete the CentOS 6 syslog-ng repositories today to free up resources, but before doing so I am asking your opinion. The repositories will be deleted by Copr anyway in about half a year (180 days after a release becomes EoL). You can answer here or my poll on Twitter: https://twitter.com/PCzanik/status/1334071192748167168 Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Filter matching not working
by Saqib M 09 Dec '20

09 Dec '20

Greetings - I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m(a)cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.

4 5

Syslog-ng Windows Agent
by Nik Ambrosch 08 Dec '20

08 Dec '20

Hi, I was wondering if there is an OSE syslog-ng windows agent or if that's PE only? The "officially supported binaries" link is broken (bottom of the page) - https://www.syslog-ng.com/products/open-source-log-management/3rd-party-bin… We have some windows servers that I'd like to integrate into our log management system and the built-in syslog support is... not great. Thanks.

2 1

Don't change permissions
by Faine, Mark R. (MSFC-IS40)[NICS] 08 Dec '20

08 Dec '20

I'm monitoring the ACL on log files with monit and comparing them to a known good ACL to make sure everything is correct and running a correction script in response to a failure, my problem is that my compare keeps failing because syslog-ng is changing the effective permissions. Is there a way I can disable this? Thanks, -Mark

2 1

Re: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 01 Dec '20

01 Dec '20

Hello Laci, thanks for your support and sorry for the delay. After some debugs and installing the old version 3.5.6 on the host again, which was also not working I found the problem in the secondary ip address we used on the host. After removing the secondary address syslog-ng is working again without a problem. At the moment we will use this old version, but I will try to use the docker version on a test system to check if it works again and we can use the new features of the 3.29 version. Thanks again and kind regards Kind regards Uwe Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 13. November 2020 12:04 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng) CAUTION: This e-mail is from an external sender. Do not click links or open attachments unless you recognize the sender and know the content is safe. Never provide your password! Hello Uwe, first of all: thank you for sharing your config and exact Syslog-ng versions. It helps a lot. At first look I cannot see any problem with your config. However, there were a lot of changes between version 3.5.6 and 3.26.1 (You might want to take a look at "cisco-parser()") If you suspect this is a filter problem, may I suggest starting Syslog-ng in a debug mode? example: https://github.com/balabit/syslog-ng-docker/issues/58#issuecomment-680674916 In this case you should see messages like: >>>>>> filter rule evaluation begin; rule='foobar', location='/conf/syslog-ng.conf:15:16', msg='0x7efd38015c40' <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='foobar', location='/conf/syslog-ng.conf:15:16', msg='0x7efd38015c40' If this does not help finding the problematic filter, I might need some example logs to trace down any parsing issues. Best regards, Laci ________________________________ From: syslog-ng on behalf of Martin, Uwe Sent: Friday, November 13, 2020 11:15 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng) CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe Festo SE & Co. KG Uwe Martin Abteilung IM-IHN IT Communication Networks Gottlieb-Stoll-Straße 29 66386 St. Ingbert Deutschland Telefon +49(6894)591-6323 Telefax +49(711)34754-6323 http://www.festo.com Der Inhalt dieser E-Mail und moeglicher Anhaenge sind ausschliesslich fuer den bezeichneten Adressaten bestimmt. Jede Form der Kenntnisnahme, Veroeffentlichung, Vervielfaeltigung oder Weitergabe des Inhalts dieser E-Mail und moeglicher Anhaenge durch unberechtigte Dritte ist unzulaessig. Wir bitten Sie, sich mit dem Absender der E-Mail in Verbindung zu setzen, falls Sie nicht der Adressat dieser E-Mail sind sowie das Material von Ihrem Computer zu loeschen. This e-mail and any attachments are confidential and intended solely for the addressee. The perusal, publication, copying or dissemination of the contents of this e-mail by unauthorised third parties is prohibited. If you are not the intended recipient of this e-mail, please delete it and immediately notify the sender. Rechtsform: Kommanditgesellschaft, Sitz: Esslingen a.N., Registergericht Stuttgart HRA 211583, Umsatzsteuerident-Nummer: DE 145339206 Persoenlich haftende Gesellschafterin: Festo Management SE, Sitz: Frankfurt a.M., Registergericht Frankfurt a.M., HRB 115998 Vorstand: Dipl.-Ing. Gerhard Borho, Dipl.-Ing. Dr. h.c. Oliver D. Jung (Vorsitzender), Dr. Ansgar Kriwet, Dr. Frank Melzer, Dipl.-Ing. (FH) Frank Notz, Dr. Jaroslav Patka Aufsichtsratsvorsitzender: Dr. Friedrich Eichiner Bitte beachten Sie: die Festo AG & Co. KG firmiert seit 31.01.2020 unter Festo SE & Co. KG. Please note, on 2020-01-31 Festo AG & Co. KG changed its company name to Festo SE & Co. KG.

1 0

poll: web interface for syslog-ng
by Peter Czanik (pczanik) 27 Nov '20

27 Nov '20

Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 3

Re: [syslog-ng] poll: web interface for syslog-ng
by Clayton Dukes 25 Nov '20

25 Nov '20

Well naturally, my vote is for LogZilla 😉 Peter, I should probably get you an updated description of the product, it has grown a lot! Here are a few useful links: LogZilla in 4 minutes https://www.logzilla.net/logzilla-in-4-minutes.html Automation/Orchestration https://www.logzilla.net/tcat.html ELK Revealed https://www.logzilla.net/the-real-cost-of-elk.html [photo] [signature_543644359] Clayton Dukes CEO, LogZilla Corporation [signature_672330100]<http://www.linkedin.com/in/lzcdukes/> [signature_753363620]<http://twitter.com/logzilla> [signature_14708953] (855) LogZilla<tel:(855)%20LogZilla> [signature_1738705128] 919-600-3198<tel:919-600-3198> [signature_579604246] cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> [signature_1641853981] logzilla.net<http://logzilla.net/> [signature_1306807084] 4819 Emperor Blvd., Suite 400, Raleigh, NC, 27703 [signature_560762648]<https://calendly.com/teamneo/logzilla-neo-discovery> Schedule a 15 minute demo<https://calendly.com/teamneo/logzilla-neo-discovery> [signature_457621985]<https://youtu.be/yGQqkXegdoo> [signature_39202885]LZ in 4 Minutes<https://youtu.be/yGQqkXegdoo> LogZilla Corporation delivers a Centralized Log Management (CLM) platform, designed for IT Operations, Security, and Risk Management leaders who want to gain better incident investigation capabilities by capturing all network and security related logs into a single log collection platform. [Logo Description automatically generated] From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of "Peter Czanik (pczanik)" <Peter.Czanik(a)oneidentity.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Date: Wednesday, November 25, 2020 at 9:46 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] poll: web interface for syslog-ng Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslo…>) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsyslog-ng…> https://twitter.com/PCzanik<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Logzilla Corporation may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.

1 0

syslog-ng 3.30 Debian/Ubuntu packages (OBS)
by Laszlo Budai 20 Nov '20

20 Nov '20

Hi, syslog-ng 3.30.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 * Ubuntu 20.10 (build is still in progress in OBS) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-geat install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.30.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

issues with starting syslog-ng from systemd
by Russell Fulton 20 Nov '20

20 Nov '20

version 3.26.1 — ubuntu 14.04 I can start syslog-ng from the command line without issues (apart form a warning to stderr) but when I try and start it from systemd it starts up (it prints the warning message to the logs) and then exits silently. This version was compiled locally i.e. it was not installed from a package. The .service file was adapted from a sample in the contrib directory. I also had the same behaviour from and other .service file that I put together myself. It is presumably some to do with the environment. I did try starting it with -i (interactive) in the hope that it would spit something out on the console but instead it hung and I had to ^C it Any thoughts on how to figure out what is going on? ===================================== here are two different attempts (first commented) rful011@secmgrprd02:~$ cat /etc/systemd/system/syslog-ng.service #[Unit] #Description=syslog next generation #After=network.target auditd.service #[Service] #EnvironmentFile=/etc/default/syslog-ng #ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf #ExecReload=/usr/local/sbin/syslog-ng-ctl reload #Restart=on-failure #RestartPreventExitStatus=255 #Type=notify #[Install] #WantedBy=multi-user.target #Alias=syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) Conflicts=emergency.service emergency.target Wants=network.target network-online.target After=network.target network-online.target [Service] Type=notify EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf ExecReload=/usr/local/sbin/syslog-ng-ctl reload StandardOutput=journal StandardError=journal Restart=on-failure [Install] WantedBy=multi-user.target

3 2

syslog-ng-3.30.1
by Laszlo Budai 19 Nov '20

19 Nov '20

3.30.1 Highlights * filter template function * support proxy-protocol<https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> Features * kafka (C implementation): * Added template support to topic(). * Added fallback-topic() option, which will be used, if the templated topic() yields an invalid topic name. (#3372<https://github.com/syslog-ng/syslog-ng/pull/3372>) * transport: add proxy-protocol support http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt Example config, click to expand! ([#3437<https://github.com/syslog-ng/syslog-ng/pull/3437>](https://github.com//pull/3437<https://github.com/syslog-ng/syslog-ng/pull/3437>)) * filter: new template function The new introduced filter template function will allow filtering lists based on a filter expression. For example this snippet removes odd numbers log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * file, network, program destinations: : new truncate_size option introduced to truncate an output message to a specified max size. default value is -1 (disabled). network("127.0.0.1" truncate_size(100)); new stats counters: dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_count;1 dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_bytes;1 (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * network: add FreeBSD support for the so_reuseport(yes) the same as in linux SO_REUSEPORT (FreeBSD uses SO_REUSEPORT_LB flag). (#3438<https://github.com/syslog-ng/syslog-ng/pull/3438>) * date-parser: %z accepts local timezone std format as well (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * syslog-format: accepting longer sdata keys Triggered by #3197<https://github.com/syslog-ng/syslog-ng/issues/3197> At the end of the discussion in the mentioned issue, we decided to change the parser and accept longer than 32 character ID's. (#3244<https://github.com/syslog-ng/syslog-ng/pull/3244>) * systemd-journal: add namespace() option This option accepts a string which is identical to the --namespace option of journalctl. For systems defining this option with a systemd version older than v245 a warning is issued. (#3358<https://github.com/syslog-ng/syslog-ng/pull/3358>) Bugfixes * date-parse: %Z should parse the same timezones as %z not just local and gmt (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * python: printing the exception instead of None (if compiled with clang) (#3405<https://github.com/syslog-ng/syslog-ng/pull/3405>) * network/udp: message was lost (not sent) if it was too large, and a time reopen amount of time needed to expire to send the next message lowering the thoughtput. now it is truncated at 65507. (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * tlscontext: support IPv6 X509v3 Subject Alternative Name Fixes #3465<https://github.com/syslog-ng/syslog-ng/issues/3465> (#3466<https://github.com/syslog-ng/syslog-ng/pull/3466>) * map: pass $_ to if correctly. Prior this patchset, if did not receive $_ correctly. After this change, these configurations will work: log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(map $(if ('$(% $_ 2)' eq '0') 'even' 'odd') $INPUT)'\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * systemd-journal: add namespace to the persist name (#3407<https://github.com/syslog-ng/syslog-ng/pull/3407>) * syslog-ng: fixed numerous spelling mistakes in messages generated by syslog-ng (#3398<https://github.com/syslog-ng/syslog-ng/pull/3398>) * network: fix TLS certificate hostname verification when using failover() servers For TLS certificate hostname verification, the certificate's hostname needs to be compared to the configured hostname of the primary and each failover server. syslog-ng used always the primary server's name incorrectly. (#3447<https://github.com/syslog-ng/syslog-ng/pull/3447>) * afsocket: syslog-ng fails to bind() after config revert When having a program source or destination and a network destination in the config, if we reload with an invalid config, syslog-ng crashes, as it cannot init the old network source, because its address is in use. (#3416<https://github.com/syslog-ng/syslog-ng/pull/3416>) * syslog-ng-ctl: when syslog-ng gets stuck on executing a heavy stats-ctl command, should be able to do a graceful shutdown when it is requested. (#3349<https://github.com/syslog-ng/syslog-ng/pull/3349>) * json-parser: fix parsing 64 bit numbers (currently 32 bit was a limit) (#3403<https://github.com/syslog-ng/syslog-ng/pull/3403>) * usertty(): on each tty open error an error mesage and a 10 minutes long disabling of the usertty() destination has been added. Until now, the usertty() destination were only disabled for blocking write() calls. (#3473<https://github.com/syslog-ng/syslog-ng/pull/3473>) Notes to developers * Proxy protocol support added to loggen. Four new options added to loggen to suppport the proxy protocol: * --proxied : Generate PROXY protocol v1 header * --proxy-src-ip : Set the source IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-dst-ip : Set the destination IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-src-port : Set the source port for the PROXY protocol v1 header. If not specified a random port generated in the range 5000-10000. * --proxy-dst-port : Set the destination port for the PROXY protocol v1 header. If not specified the port number 514 will be used. (#3462<https://github.com/syslog-ng/syslog-ng/pull/3462>) * bison: Minimum required version is now 3.4.2. You still only need bison, if you are building from git source or changing the grammar in the released source tarball. (#2526<https://github.com/syslog-ng/syslog-ng/pull/2526>) * Template evaluation related function signatures changed. A new structure LogTemplateEvalOptions is introduced to group parameters together. (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) Other changes * json-parser: change every per message logs that was higher than debug to debug (#3401<https://github.com/syslog-ng/syslog-ng/pull/3401>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Boris Korzun, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Viktor Juhasz, Vivin Peris, Zoltan Pallagi, bjoe2k4

1 0

syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 13 Nov '20

13 Nov '20

Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe

2 1