- syslog-ng - lists.balabit.hu

Failover/Hi Availability
by Faine, Mark R. (MSFC-IS40)[NICS] 18 Jan '21

18 Jan '21

If I have two Syslog-ng servers receiving logs from the same source and writing to the exact same file system destination, is there a way to do this without running into issues with the two instances clobbering the files or file locking issues? Thanks, -Mark

2 4

Filtering Destination by Source
by Peter Griggs 12 Jan '21

12 Jan '21

Hello, We have a lot of network logs all being pointed to a central syslog however this is a mix of vendors (Cisco / Juniper / Checkpoint) etc. is there a way of splitting the destination file by vendor type / or source IP address? We ingest this data into Splunk so want to get the source typing right however I am unable to get the sources to point to various listeners and I would prefer. Thanks Peter.

3 3

Disk file truncation issue
by Raghunath Adhyapak 20 Dec '20

20 Dec '20

Hi, We have configured the destination with reliable(yes). We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated. This syslog-ng service is running on 18.04 ubuntu VM. =============================================== Our destination configuration is as below: destination d_msg_syslog_01a_06 { tcp( "172.XX.XX.XX" port(514) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(50000) # throttle to max 5k eps throttle(3000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(75161927680) # 70GB reliable(yes) # directory location to persist messages dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/") ) persist-name(d_msg_syslog_01a_06) ); }; =============================================== $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf Here are log snippets when we ran syslog-ng in debug mode. =============================================== [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat' [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846' [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' =============================================== We are unable to figure out why the disk buffer file is not getting truncated. We have restarted syslog-ng multiple times. Please advise. Thanks Raghu

2 3

Re: [syslog-ng] elasticsearch and ILM
by Németh Balázs 20 Dec '20

20 Dec '20

Hi, Originally I responsed to a digest message so you might did not get it. You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.:https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsea… Regards, Balázs > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 21:47:12 +0000 > Subject: Re: [syslog-ng] elasticsearch and ILM > Russell wishes to delete this post :) > > Russell did what he should have done before posting! He ran tcpdump and > found the error response from ES! > > Apologies for the noise. The original question is still open! > > On 20/12/2020, at 10:38 AM, Russell Fulton <r.fulton(a)auckland.ac.nz> > wrote: > > Caution - Forged Internal Domain! > This e-mail cannot be validated and may not have been sent by the sender > shown in the 'From' field. > If you were not expecting to receive this e-mail we recommend you contact > the sender to confirm that they sent it. > If you believe this email was legitimately sent, we suggest the sender > notify the Staff Service Centre that it has been received as a forged > (fake) e-mail. > Please contact the Staff Service Centre on extension 86000 if you require > further assistance. > > > > On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: > > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. Most > of the documentation seems to assume you are using filebeat, logstash or > datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell > > > This is getting weirder ! > > ES is now failing to create my new index at all in spite of the > destination definition being practically identical to one that works: > > destination d_auth_elastic { > elasticsearch_http( > index("auth_${YEAR}.${MONTH}.${DAY}") > type("_doc") > persist-name("auth") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR > > > --key ISODATE)\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > destination d_authm_elastic { > elasticsearch_http( > index("auth-000001") > type("") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR )\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > > > log { > source( s_loghost ); > parser( p_patterns ); > filter( f_classified ); > filter( f_ping999 ); > filter( f_forti_stats ); > rewrite( r_user1 ); > rewrite( r_user2 ); > rewrite( r_srcip ); > parser( p_srcip ); > rewrite( r_srcip_country ); > rewrite( r_srcip_city); > rewrite( r_src_loc); > rewrite( r_ISODATE ); > destination( d_auth_elastic ); > destination( d_authm_elastic); > flags( flow-control); > }; > > d_auth_elastic works fine but d_authm_elastic never gets created. No > errors on the ES end. > > Anyone have any clues as to what is going on? > > I am at the point of starting tcpdump and looking at what is going over > the network. > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

elasticsearch and ILM
by Russell Fulton 19 Dec '20

19 Dec '20

I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic. The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams. So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”. If anyone has a working set up that they are happy to share it would be wonderful. Russell

1 2

Can elasticsearch_http send data to ES datastreams ?
by Russell Fulton 19 Dec '20

19 Dec '20

Datastreams behave like indexes *except* when you use _bulk then instead of “index” you must use “create”. Datastreams are set up to handle timeseries log data and they make managing such data must easier. R

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 188, Issue 15
by Németh Balázs 19 Dec '20

19 Dec '20

Hi, You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.: https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/… Regards, Balázs <syslog-ng-request(a)lists.balabit.hu> ezt írta (időpont: 2020. dec. 19., Szo, 13:00): > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > Today's Topics: > > 1. elasticsearch and ILM (Russell Fulton) > > > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 01:40:42 +0000 > Subject: [syslog-ng] elasticsearch and ILM > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. > Most of the documentation seems to assume you are using filebeat, logstash > or datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell_______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

What am I doing wrong?
by Dan Egli 16 Dec '20

16 Dec '20

Help me understand this, please? I have ISC dhcpd configured to log to syslog.local7 (since I don't see an option to force it into it's own log file). So I went into my syslog-ng file and created two filters, just like on the example page of syslog-ng.com: filter dhcpmsgs { facility(23) ); filter non_dhcp { NOT filter(dhcpmsgs) ) I quoted almost directly from the example page on syslog-ng.com, but I keep getting this error when I reload syslog-ng's config: Error parsing filter expression, filter plugin NOT not found OR you may not used double quotes in your filter expression in /etc/syslog-ng/syslog-ng.conf:25:18-25:21: What did I do wrong? Here's the lines I modified from the syslog-ng page: filter demo_filter { host("example") and match("deny" value("MESSAGE")) }; filter inverted_demo_filter { NOT filter(demo_filter) } You can see the page at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… -- Dan Egli From my Test Server

2 6

syslog-ng and DTLS
by Secure-Logging, Generic 12 Dec '20

12 Dec '20

Dear syslog-ng users and developers, I know that syslog-ng is able to communicate using TCP and also supports TLS connections for improved security. Does syslog-ng also support support secure connections based UDP using DTLS? I am trying to implement a secure connection on a system with a network stack that does not support TCP because of resource constraints. Can I use syslog-ng for this? As syslog-ng security is based on OpenSSL and OpenSSL supports DTLS, I hope that this will be possible but I rely on your feedback concerning this topic. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

2 1

Insider 2020-12: web interfaces; Grafana Loki; Amazon Linux 2;
by Peter Czanik (pczanik) 10 Dec '20

10 Dec '20

Dear syslog-ng users, This is the 87th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Web interfaces for your syslog server – an overview --------------------------------------------------- This is the 2020 edition of my most read blog entry about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011, only a single logging as a service solution was available, while nowadays, I regularly run into others. Also, while some software disappeared, the number of logging-related GUIs is growing. This is why in this post, I will mostly focus on generic log management and open source instead of highly specialized software, like SIEMs. https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy… Sending logs from syslog-ng to Grafana Loki ------------------------------------------- Loki is one of the latest applications that lets you aggregate and query log messages, and of course to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC5424 syslog protocol. This is where syslog-ng can send its log messages. >From this blog, you can learn a minimal Loki & Promtail setup. We will send logs from syslog-ng, and as a first step, will check them with logcli, a command line utility for Loki. Once it works, we will also install Grafana in a container and query Loki from there. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… Installing syslog-ng in Amazon Linux 2, including Graviton2 ----------------------------------------------------------- Amazon Linux 2 is just one of many Linux distributions and other operating systems running in AWS. As Amazon Linux 2 is a close relative of RHEL 7 / CentOS 7, the user experience is also very close to these operating systems. All you need to learn is an extra command for enabling EPEL (Extra Packages for Enterprise Linux). Best of all: the very same commands work both on x86 and Graviton2 (Arm/Aarch64) instances. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-in-am… WEBINARS * Using syslog-ng with Google Pub/Sub: https://www.syslog-ng.com/event/using-syslogng-with-google-pubsub-8145754/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0