- syslog-ng - lists.balabit.hu

syslog-ng won't talk to systemd?
by Dan Egli 26 Jan '21

26 Jan '21

I'm trying to setup syslog-ng on a systemd system. But even though I have several things writing to syslog, syslog-ng doesn't write ANYTHING to the /var/log/messages except kernel messages and it's own info. So I searched around, and found the systemd-journal() source. So I tried it. And I get an error that says syslog-ng can't initialize the logger plugin: # syslog-ng -F [2021-01-23T23:25:36.525097] The configuration must not contain more than one systemd-journal() source; [2021-01-23T23:25:36.525157] Error initializing journal_reader; [2021-01-23T23:25:36.525173] Error initializing message pipeline; plugin_name='systemd-journal', location='/etc/syslog-ng/syslog-ng.conf:26:5' Here's the config, basically copied from the syslog-ng.com page: @version: 3.26 # # Syslog-ng default configuration file for Gentoo Linux # https://bugs.gentoo.org/426814 @include "scl.conf" options { threaded(yes); chain_hostnames(no); stats_freq(43200); mark_freq(3600); }; source s_journald { systemd-journal(prefix(".SDATA.journald.")); }; source src { system(); internal(); file("/proc/kmsg"); }; destination messages { file("/var/log/messages"); }; destination console_all { file("/dev/tty12"); }; log { source(src); destination(messages); }; log { source(s_journald); destination(messages); }; log { source(src); destination(console_all); }; What am I doing wrong? I am POSITIVE things are sending to the syslog sub-system. I have ensured several programs have sent something. But without the systemd-journal line syslog-ng loads fine, recording /proc/kmsg and syslog-ng messages but nothing else. I am completely stumped here.

2 3

Polling additional machines
by Dan Egli 23 Jan '21

23 Jan '21

I have an interesting setup that was proposed to me, and while I _THINK_ syslog-ng can handle it I don't know for sure, let alone HOW. There are multiple service-specific machines in the network, behind the firewall. I'll just use the services they provide as their hostnames in this example. So firewall fw is a small linux machine running syslog-ng and fail2ban. Now fail2ban works off of the log files, but it also wants to apply changes to the firewall rules locally. So either www has its firewall rules while smtp has its rules and imap has a third set (ugly) or fail2ban can run on fw and modify the rules there. Trick is, in order to do that, all the logs have to be accessible in real time so that fail2ban can see them. My thought was to have syslog-ng on each sub machine somehow report to the syslog-ng on fw. Then fw's syslog-ng can write the files that fail2ban wants to read from. What would be the best way to go about this? I am still learning syslog-ng so if this is something simple, I'm sorry.

2 1

syslog-ng on CentOS 7 Aarch64
by Peter Czanik (pczanik) 20 Jan '21

20 Jan '21

Hi, I just learned that RHEL 7 on Aarch64 (64bit Arm) is end of life. Copr, where I build syslog-ng packages, removes RHEL/CentOS 7 Aarch64 packages in about half a year and it is not possible to build new packages there any more. CentOS 7 on the other hand keeps going on Aarch64 (using CentOS 8 kernel and CentOS 7 userland). There might be a workaround for the situation (using a different build service), but it's quite a lot extra effort. So, before investing any time and effort into keep building CentOS 7 Aarch64, I want to know, if there is a significant number of syslog-ng users, who keep using CentOS 7 on Aarch64. You can respond here or to my Twitter poll at: https://twitter.com/PCzanik/status/1351816930345291776 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Signed Syslog Messages RFC 5848
by Alexandre Santos 20 Jan '21

20 Jan '21

Hi, Are there any plans to add Signed Syslog Messages RFC 5848 to syslog-ng? Best Regards, Alex

3 4

Signed Syslog Messages RFC 5848
by Secure-Logging, Generic 19 Jan '21

19 Jan '21

Hi Alexandre, as a replacement for signed messages you might have a look at the secure logging module of syslog-ng. This module authentically encrypts each message, so you get an integrity guarantee which is stronger than a pure signature. You can then forward the message to a backend for verification and further processing. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

Failover/Hi Availability
by Faine, Mark R. (MSFC-IS40)[NICS] 18 Jan '21

18 Jan '21

If I have two Syslog-ng servers receiving logs from the same source and writing to the exact same file system destination, is there a way to do this without running into issues with the two instances clobbering the files or file locking issues? Thanks, -Mark

2 4

Filtering Destination by Source
by Peter Griggs 12 Jan '21

12 Jan '21

Hello, We have a lot of network logs all being pointed to a central syslog however this is a mix of vendors (Cisco / Juniper / Checkpoint) etc. is there a way of splitting the destination file by vendor type / or source IP address? We ingest this data into Splunk so want to get the source typing right however I am unable to get the sources to point to various listeners and I would prefer. Thanks Peter.

3 3

Disk file truncation issue
by Raghunath Adhyapak 20 Dec '20

20 Dec '20

Hi, We have configured the destination with reliable(yes). We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated. This syslog-ng service is running on 18.04 ubuntu VM. =============================================== Our destination configuration is as below: destination d_msg_syslog_01a_06 { tcp( "172.XX.XX.XX" port(514) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(50000) # throttle to max 5k eps throttle(3000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(75161927680) # 70GB reliable(yes) # directory location to persist messages dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/") ) persist-name(d_msg_syslog_01a_06) ); }; =============================================== $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf Here are log snippets when we ran syslog-ng in debug mode. =============================================== [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat' [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846' [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' =============================================== We are unable to figure out why the disk buffer file is not getting truncated. We have restarted syslog-ng multiple times. Please advise. Thanks Raghu

2 3

Re: [syslog-ng] elasticsearch and ILM
by Németh Balázs 20 Dec '20

20 Dec '20

Hi, Originally I responsed to a digest message so you might did not get it. You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.:https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsea… Regards, Balázs > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 21:47:12 +0000 > Subject: Re: [syslog-ng] elasticsearch and ILM > Russell wishes to delete this post :) > > Russell did what he should have done before posting! He ran tcpdump and > found the error response from ES! > > Apologies for the noise. The original question is still open! > > On 20/12/2020, at 10:38 AM, Russell Fulton <r.fulton(a)auckland.ac.nz> > wrote: > > Caution - Forged Internal Domain! > This e-mail cannot be validated and may not have been sent by the sender > shown in the 'From' field. > If you were not expecting to receive this e-mail we recommend you contact > the sender to confirm that they sent it. > If you believe this email was legitimately sent, we suggest the sender > notify the Staff Service Centre that it has been received as a forged > (fake) e-mail. > Please contact the Staff Service Centre on extension 86000 if you require > further assistance. > > > > On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: > > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. Most > of the documentation seems to assume you are using filebeat, logstash or > datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell > > > This is getting weirder ! > > ES is now failing to create my new index at all in spite of the > destination definition being practically identical to one that works: > > destination d_auth_elastic { > elasticsearch_http( > index("auth_${YEAR}.${MONTH}.${DAY}") > type("_doc") > persist-name("auth") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR > > > --key ISODATE)\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > destination d_authm_elastic { > elasticsearch_http( > index("auth-000001") > type("") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR )\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > > > log { > source( s_loghost ); > parser( p_patterns ); > filter( f_classified ); > filter( f_ping999 ); > filter( f_forti_stats ); > rewrite( r_user1 ); > rewrite( r_user2 ); > rewrite( r_srcip ); > parser( p_srcip ); > rewrite( r_srcip_country ); > rewrite( r_srcip_city); > rewrite( r_src_loc); > rewrite( r_ISODATE ); > destination( d_auth_elastic ); > destination( d_authm_elastic); > flags( flow-control); > }; > > d_auth_elastic works fine but d_authm_elastic never gets created. No > errors on the ES end. > > Anyone have any clues as to what is going on? > > I am at the point of starting tcpdump and looking at what is going over > the network. > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

elasticsearch and ILM
by Russell Fulton 19 Dec '20

19 Dec '20

I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic. The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams. So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”. If anyone has a working set up that they are happy to share it would be wonderful. Russell

1 2