- syslog-ng - lists.balabit.hu

Quoting in filters
by Faine, Mark R. (MSFC-IS40)[NICS] 05 Aug '19

05 Aug '19

What is the policy on quoting values in filters? In the docs it seems that sometimes they are quoted and other times they are not. For example, are these equivalent: netmask (127.0.0.1/32) and netmask("127.0.0.1/32") I think that maybe some filters, the ones that take strings essentially, are quoted, and not the ones that take numeric data. Is that correct? Thanks, -Mark

2 1

new install of OSE 3.22 fails on ubuntu 16,04
by Russell Fulton 05 Aug '19

05 Aug '19

Running on ubuntu 16.04 I have just compiled and installed syslog-ng version 3.22. When started it prints: iv_tls_user_register: called after iv_init and then exits. Google tells me there was a bug that caused this back in 2013 but there are not recent posts. Any idea what is going on? BTW I can’t use the binary packages as there are conflicts between the rsyslog and syslog-ng packages and I am stuck with having the rsyslog package by corporate policy. Russell

2 1

Fwd: Multithreaded inputs with same output file : is it safe ?
by Jean-Baptiste Fuzier 02 Aug '19

02 Aug '19

Hello, I have a few questions related to multithreading. >From the doc, my understanding is that inputs are multithreaded this way : 1/ Tcp connections are sent to different thread when multithreading is enabled, even if there is a single source enabled in config with a single tcp port listening 2/ UDP datagrams are NOT sent to different thread if there is a single source even if there are multiple listening ports in the source config 3/ UDP datagrams are sent to different threads if there are different sources in config (1 source -> 1 thread), so basically to benefit from multithreading with UDP one needs to configure several sources. Is my understanding correct ? If we have several UDP source with some round robin balancing of the input to different source on the same daemon process, that means that several host with the same name can send their syslog data to different source. For the output, I am using files with the name of the sending host in the name. Is there an issue if several source (= from different threads) ends up being outputed in the same file. Another way to turn the question is: are all the data from the source thread then send to some common thread so that output to a file is always handled by the same thread even if log is received from different source threads ? I tried to ask the same question with a schematic : Case1 : +-------------------------+ +-------------------+ +-----------------------------+ +-----------+ +----------+ | | | | | | | host foo +------->+ |X----->+ thread1 some foo log +----->+ +--->+ thread handling foo.log +------>foo.log +-----------+ | iptables |X XX| some bar log | | black box magic | | | | round | XXXX +-------------------------+ | | +-----------------------------+ | robin | XXXXX | | | |XX XX+-------------------------+ | | +-----------------------------+ +-----------+ | X------>+ | | | | | | host bar +------->+ | | thread2 some foo log +----->+ +--->+ thread handling bar.log +------>bar.log +-----------+ +----------+ | some bar log | | | | | +-------------------------+ +-------------------+ +-----------------------------+ Case2 : +-------------------------+ +-----------+ +----------+ | | | host foo +------->+ |X----->+ thread1 some foo log +---------------------> +--------+ +-----------+ | iptables |X XX| some bar log +----------| |foo.log | | round | XXXX +-------------------------+ +------------> +--------+ 2 different thread writting | robin | XXXXX | | to same file | |XX XX+-------------------------+ | +---------->----------+ likely bad ! +-----------+ | X------>+ +--------+ |bar.log | | host bar +------->+ | | thread2 some foo log +--------------------->----------+ +-----------+ +----------+ | some bar log | +-------------------------+ One example config : # Notice that host name is used in output file name # There is no afinity between 1 host <> 1 udp port => logs from host foo are send to every source/udp port in a round robin fashion source s_syslog__udp5141{ network(ip(192.168.1.1) transport("udp") port(5141) flags("threaded")); }; source s_syslog__udp5142{ network(ip(192.168.1.1) transport("udp") port(5142) flags("threaded")); }; source s_syslog__udp5143{ network(ip(192.168.1.1) transport("udp") port(5143) flags("threaded")); }; source s_syslog__udp5144{ network(ip(192.168.1.1) transport("udp") port(5144) flags("threaded")); }; destination d_network_fw_misc { file("`syslog_log_root_path`/fw/fw/$FULLHOST/`naming_scheme`" group("logs_fw") dir-group("logs_fw") log-fifo-size(500000)); }; destination d_network_wifi { file("`syslog_log_root_path`/wifi/$FULLHOST/`naming_scheme`" group("logs_wifi") dir-group("logs_wifi") log-fifo-size(500000)); }; destination d_network_debug { file("`syslog_log_root_path`/debug/`naming_scheme`" group("logs_network_debug") dir-group("logs_network_debug") log-fifo-size(600000)); }; log { source(s_syslog__udp5141); source(s_syslog__udp5142); source(s_syslog__udp5143); source(s_syslog__udp5144); # Every channel of the junction received a copy of the message, unless there is a final flag before # Final flags are applied only within the junction and not globally junction { channel {filter(f_debug); destination(d_network_debug);}; channel {filter(f_network_wifi); destination(d_network_wifi); flags(final);}; channel {destination(d_network_fw_misc); flags(final);}; }; }; #Round robin with iptables : iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 4 --packet 0 -j REDIRECT --to-port 5141 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 3 --packet 0 -j REDIRECT --to-port 5142 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 2 --packet 0 -j REDIRECT --to-port 5143 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 1 --packet 0 -j REDIRECT --to-port 5144 Thanks for your help -- Jean-Baptiste Fuzier

2 1

cisco templetes
by ciprian niculescu 31 Jul '19

31 Jul '19

Hello, i'm building a syslog relay to collect and duplicate the flows to multiple destinations. but the relayed messages are strange looking. my source are cisco network devices (catalyst, nexus, asa) and i want to relay to a Solarwinds, Splunk and a linux-syslog for archiving. i search the net for a templete but found none. What i got so far is that the catalyst is sending in syslog bsd format, but with the relay configured to source bsd and destination bsd, the end message is different (the date is doubled, the relay add his IP) any help is appreciated. Regards, Ciprian

2 1

Tips for handling large message load
by Faine, Mark R. (MSFC-IS40)[NICS] 30 Jul '19

30 Jul '19

I have several Splunk log aggregators that gets thousands of messages per second but we are seeing issues with dropping messages from UDP sources. I've read the section in the docs about handling large message load and we've made many of those changes. Do you have any other suggestions to improve performance? We are using flow control. We have made the following sysctl changes: - net.core.rmem_max = 268435456 - net.core.netdev_max_backlog = 2000 We have increased flush-lines to 100 We are also looking to increase the initial window size below and max connections as well as the so_rcvbuf. network(port(514) transport("tcp") max-connections(100) log_iw_size(10000) flags(syslog-protocol)); network(port(514) transport("udp") max-connections(100) log_iw_size(10000) flags(syslog-protocol)); network(ip(0.0.0.0) transport("tls") port(10514) max-connections(100) log_iw_size(10000) flags(syslog-protocol) I'm currently showing about 50 TCP connections and almost 1000 UDP connections, though it's early on Monday morning so it is likely to increase as the day goes on. Most of the log messages are coming from the VPNs (hence so many UDP connections). The servers are quite beefy with 64GB of RAM and 24 Xeon cores @ 2.4GHz (Dell PowerEdge R530). We are using two separate network interfaces one for TCP and the other for UDP. I'd appreciate any suggestions on how to further increase performance, also, any general rules or calculations I can use to determine optimal values for these parameters from available system metrics would be very helpful. Thanks, -Mark

3 2

SYSLOG to SNMP TRAP relay server, how can I spoof source address
by Armando Martires 23 Jul '19

23 Jul '19

Hello, I set up a Syslog-ng relay server to relay syslog messages as SNMP traps. The relay is working sending a SNMP TRAP whenever a SYSLOG message is received, but the source seems to be the syslog-ng server and not the client host that originated the syslog message. This is my config: options { time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); chain_hostnames (no); }; source s_labs_itsm { syslog(ip("192.168.50.20") transport("udp")); }; destination d_local { file("/var/log/messages_${HOST}"); }; destination d_ss { snmp( host("192.168.50.10") version("v2c") community("public") trap-obj('.1.3.6.1.6.3.1.1.4.1.0', 'Objectid', '.1.3.6.1.4.1.18372.3.1.1.1.2.1') snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.1.0', 'Octetstring', '${MESSAGE}') snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.2.0', 'Octetstring', 'admin') snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.3.0', 'Ipaddress', '${HOST}') ); }; log { source(s_labs_itsm); destination(d_local); destination(d_ss); }; Can anyone help me out understanding what I'm doing wrong? Thanks! -- Armando Mártires

2 1

Syslog-ng header & rfc5424
by Florian Goulais 19 Jul '19

19 Jul '19

Hi. I'am trying to understand why syslog-ng and an header (timestamp host) to log as the rfc specify another header. Someone can explain me the reason plz ? Regards Florian

3 6

Get local log files written immediately even if remote log server is unresponsive
by Jim Segrave 16 Jul '19

16 Jul '19

We are running syslog-ng 3.16.1 on Centos 7.4.1708 on a central logging host. We have a large (nearly 1,000) servers also running the same version of syslog-ng on the same Centos release. The servers are configured to log locally and also forward logs to the central logging host. This morning we encountered a problem - syslog-ng was running on the logging host, but was not processing incoming logs or locally generated ones. The servers forwarding to the central host did not write anything to their local log files, a small but significant portion of them had syslog crash, after which is was restarted by systemd, but still no logs were written until syslog-ng was forcibly stopped on the central server and then restarted. Connections to the central server weren't failing in the sense of TCP close or reset, but logs were accumulating on all the servers, including the central one, in the cache file for buffering logs. For our purposes, we need to have up-to-the moment logs available on the individual servers, so an admin going in to troubleshoot on a server who only has console access still has recent logs to consult if needed. Is there a way to tell syslog-ng to write local logs immediately even if it's currently buffering logs for sending to a non-responsive remote server?

3 3

Missing messages
by Klimek, Bryan J. 16 Jul '19

16 Jul '19

First time poster, so be gentle. We run syslog-ng in our environment as a centralized syslog manager for the archiving of syslog data. We have over 2000+ Unix/Linux systems sending their syslog data with a daily ingest rate of about 10GB per day. It was recently pointed out to me that one particular message from one specific host is not getting persisted to the files on our syslog-ng server all the time. That is to say, it is intermittent. If one person can find one message that is not making into our syslog-ng archive, I can only assume that we are dropping other messages as well. How can I debug if and when messages are being lost and not making into the files on my centralized syslog server? Bryan Klimek Mayo Clinic

4 10

Cannot send Syslog-ng to Elasticsearch
by Allen Olivas 12 Jul '19

12 Jul '19

Hello, Recently I've tried following along with the Syslog-NG to Elasticsearch and Kibana blog posts and Admin Documentation for integrating Syslog-NG into Elasticsearch but I'm unable to integrate the two. I see in the .conf files the destination calls for creating and Index Pattern for Syslog-NG but when I curl the existing indices I do not see syslog-ng. Also, I'm now receiving two errors. The first I'm fairly certain we need to resolve but I've not been able to find adequate documentation on how to identify the issue let along resolve it, and the second I'm not sure if we actually need to fix. The two issues: Issue#1: I/O error occurred syslog-ng[26432]: Syslog connection established; fd='12', server='AF_INET(127.0.0.1:9200)', local='AF_INET(0.0.0.0:0)' syslog-ng[26432]: I/O error occurred while writing; fd='12', error='Broken pipe (32)' syslog-ng[26432]: Syslog connection broken; fd='12', server='AF_INET(127.0.0.1:9200)', time_reopen='60' Issue#2: Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' For issue 1 I'm not sure what to do or how to resolve it. For issue 2, I know for certain libjvm does exist, and I've mapped the LD_LIBRARY_PATH to the directory libjvm.so resides in. Ultimately, are these two issues preventing Syslog-NG from sending to Elasticsearch or are they just separate issues to tackle after I get things cleared up, and most importantly if they're not related, how do I integrate Syslog-NG with Elasticsearch and Kibana. Documentation is not helpful and not concise. Thanks!

4 18