December 2020 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Disk file truncation issue
by Raghunath Adhyapak 20 Dec '20

20 Dec '20

Hi, We have configured the destination with reliable(yes). We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated. This syslog-ng service is running on 18.04 ubuntu VM. =============================================== Our destination configuration is as below: destination d_msg_syslog_01a_06 { tcp( "172.XX.XX.XX" port(514) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(50000) # throttle to max 5k eps throttle(3000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(75161927680) # 70GB reliable(yes) # directory location to persist messages dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/") ) persist-name(d_msg_syslog_01a_06) ); }; =============================================== $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf Here are log snippets when we ran syslog-ng in debug mode. =============================================== [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat' [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846' [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' =============================================== We are unable to figure out why the disk buffer file is not getting truncated. We have restarted syslog-ng multiple times. Please advise. Thanks Raghu

2 3

Re: [syslog-ng] elasticsearch and ILM
by Németh Balázs 20 Dec '20

20 Dec '20

Hi, Originally I responsed to a digest message so you might did not get it. You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.:https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsea… Regards, Balázs > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 21:47:12 +0000 > Subject: Re: [syslog-ng] elasticsearch and ILM > Russell wishes to delete this post :) > > Russell did what he should have done before posting! He ran tcpdump and > found the error response from ES! > > Apologies for the noise. The original question is still open! > > On 20/12/2020, at 10:38 AM, Russell Fulton <r.fulton(a)auckland.ac.nz> > wrote: > > Caution - Forged Internal Domain! > This e-mail cannot be validated and may not have been sent by the sender > shown in the 'From' field. > If you were not expecting to receive this e-mail we recommend you contact > the sender to confirm that they sent it. > If you believe this email was legitimately sent, we suggest the sender > notify the Staff Service Centre that it has been received as a forged > (fake) e-mail. > Please contact the Staff Service Centre on extension 86000 if you require > further assistance. > > > > On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: > > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. Most > of the documentation seems to assume you are using filebeat, logstash or > datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell > > > This is getting weirder ! > > ES is now failing to create my new index at all in spite of the > destination definition being practically identical to one that works: > > destination d_auth_elastic { > elasticsearch_http( > index("auth_${YEAR}.${MONTH}.${DAY}") > type("_doc") > persist-name("auth") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR > > > --key ISODATE)\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > destination d_authm_elastic { > elasticsearch_http( > index("auth-000001") > type("") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR )\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > > > log { > source( s_loghost ); > parser( p_patterns ); > filter( f_classified ); > filter( f_ping999 ); > filter( f_forti_stats ); > rewrite( r_user1 ); > rewrite( r_user2 ); > rewrite( r_srcip ); > parser( p_srcip ); > rewrite( r_srcip_country ); > rewrite( r_srcip_city); > rewrite( r_src_loc); > rewrite( r_ISODATE ); > destination( d_auth_elastic ); > destination( d_authm_elastic); > flags( flow-control); > }; > > d_auth_elastic works fine but d_authm_elastic never gets created. No > errors on the ES end. > > Anyone have any clues as to what is going on? > > I am at the point of starting tcpdump and looking at what is going over > the network. > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

elasticsearch and ILM
by Russell Fulton 19 Dec '20

19 Dec '20

I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic. The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams. So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”. If anyone has a working set up that they are happy to share it would be wonderful. Russell

1 2

Can elasticsearch_http send data to ES datastreams ?
by Russell Fulton 19 Dec '20

19 Dec '20

Datastreams behave like indexes *except* when you use _bulk then instead of “index” you must use “create”. Datastreams are set up to handle timeseries log data and they make managing such data must easier. R

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 188, Issue 15
by Németh Balázs 19 Dec '20

19 Dec '20

Hi, You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.: https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/… Regards, Balázs <syslog-ng-request(a)lists.balabit.hu> ezt írta (időpont: 2020. dec. 19., Szo, 13:00): > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > Today's Topics: > > 1. elasticsearch and ILM (Russell Fulton) > > > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 01:40:42 +0000 > Subject: [syslog-ng] elasticsearch and ILM > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. > Most of the documentation seems to assume you are using filebeat, logstash > or datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell_______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

What am I doing wrong?
by Dan Egli 16 Dec '20

16 Dec '20

Help me understand this, please? I have ISC dhcpd configured to log to syslog.local7 (since I don't see an option to force it into it's own log file). So I went into my syslog-ng file and created two filters, just like on the example page of syslog-ng.com: filter dhcpmsgs { facility(23) ); filter non_dhcp { NOT filter(dhcpmsgs) ) I quoted almost directly from the example page on syslog-ng.com, but I keep getting this error when I reload syslog-ng's config: Error parsing filter expression, filter plugin NOT not found OR you may not used double quotes in your filter expression in /etc/syslog-ng/syslog-ng.conf:25:18-25:21: What did I do wrong? Here's the lines I modified from the syslog-ng page: filter demo_filter { host("example") and match("deny" value("MESSAGE")) }; filter inverted_demo_filter { NOT filter(demo_filter) } You can see the page at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… -- Dan Egli From my Test Server

2 6

syslog-ng and DTLS
by Secure-Logging, Generic 12 Dec '20

12 Dec '20

Dear syslog-ng users and developers, I know that syslog-ng is able to communicate using TCP and also supports TLS connections for improved security. Does syslog-ng also support support secure connections based UDP using DTLS? I am trying to implement a secure connection on a system with a network stack that does not support TCP because of resource constraints. Can I use syslog-ng for this? As syslog-ng security is based on OpenSSL and OpenSSL supports DTLS, I hope that this will be possible but I rely on your feedback concerning this topic. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

2 1

Insider 2020-12: web interfaces; Grafana Loki; Amazon Linux 2;
by Peter Czanik (pczanik) 10 Dec '20

10 Dec '20

Dear syslog-ng users, This is the 87th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Web interfaces for your syslog server – an overview --------------------------------------------------- This is the 2020 edition of my most read blog entry about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011, only a single logging as a service solution was available, while nowadays, I regularly run into others. Also, while some software disappeared, the number of logging-related GUIs is growing. This is why in this post, I will mostly focus on generic log management and open source instead of highly specialized software, like SIEMs. https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy… Sending logs from syslog-ng to Grafana Loki ------------------------------------------- Loki is one of the latest applications that lets you aggregate and query log messages, and of course to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC5424 syslog protocol. This is where syslog-ng can send its log messages. >From this blog, you can learn a minimal Loki & Promtail setup. We will send logs from syslog-ng, and as a first step, will check them with logcli, a command line utility for Loki. Once it works, we will also install Grafana in a container and query Loki from there. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… Installing syslog-ng in Amazon Linux 2, including Graviton2 ----------------------------------------------------------- Amazon Linux 2 is just one of many Linux distributions and other operating systems running in AWS. As Amazon Linux 2 is a close relative of RHEL 7 / CentOS 7, the user experience is also very close to these operating systems. All you need to learn is an extra command for enabling EPEL (Extra Packages for Enterprise Linux). Best of all: the very same commands work both on x86 and Graviton2 (Arm/Aarch64) instances. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-in-am… WEBINARS * Using syslog-ng with Google Pub/Sub: https://www.syslog-ng.com/event/using-syslogng-with-google-pubsub-8145754/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Poll: RHEL 6 / CentOS 6 packages
by Peter Czanik (pczanik) 09 Dec '20

09 Dec '20

Hi, Now, that CentOS 6 (RHEL 6) is end of life and removed from mirrors (https://twitter.com/Arrfab/status/1334064890261147650) it is time to think what to do with my unofficial CentOS 6 syslog-ng packages on Copr (https://copr.fedorainfracloud.org/coprs/czanik/) They definitely do not receive any updates. Personally I would delete the CentOS 6 syslog-ng repositories today to free up resources, but before doing so I am asking your opinion. The repositories will be deleted by Copr anyway in about half a year (180 days after a release becomes EoL). You can answer here or my poll on Twitter: https://twitter.com/PCzanik/status/1334071192748167168 Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1