May 2017 - syslog-ng - lists.balabit.hu

Do stats cause memory leak
by Evan Rempel 06 Jun '17

06 Jun '17

We are running git head download post 3.9.1 and the memory footprint seems to continually grow. Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195 It it possible that all of these statistics take up a non-trivial amount of RAM? USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 10139760 ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng That is a 12GB footprint after 1 month of uptime. Evan.

4 8

syslog 1024 byte limit work around
by Corey Davelaar 31 May '17

31 May '17

Hello, I'm new to the group and hopefully it's not a repeat question. I was guided to this mailing list by a balabit tech support rep. We use splunk on the back end of syslog-ng and we have some syslog sources that utilize very long syslog messages, that end up being over 1024 bytes. We switched from udp to tcp as the transport and now we are seeing the tail end of the syslog messages, but splunk gets confused by the inputs and treats each line as a separate syslog message and completely unrelated to the previous one, because that’s how syslog-ng is treating it. I was wondering if there is a way to tell syslog-ng that if a new packet doesn’t start with a standard syslog timestamp to treat it as a continuation of the previous message or somehow combine it with a previous message, utilizing a buffer type of setting. Thanks, Corey

3 5

Sawmill and Syslog-NG template() string
by Francesco Rolando 31 May '17

31 May '17

Hi all, I'm working with Sawmill log collector that it's able to import the Syslog-NG logs. I have to export logs with a specific template() to have them corretly imported into Sawmill. Here the regular expression used to match syslog-ng lines inside the Sawmill parser (that seem coded by "BalaBit IT Security"): '^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])T([0-9][0-9]:[0-9][0-9]:[0-9][0-9])[-+][0-9][0-9]:*[0-9][0-9] ([^ ]+) ([^ ]+) ([^ ]+) (.*)$' where matches are imported as: 1- date 2- time 3- logging_device 4- syslog_message_type 5- syslog_priority 6- v.syslog_message I have partially re-create the template() string but I still have a couple of doubts: template("${YEAR}-${MONTH}-${DAY}T${HOUR}:${MIN}:${SEC}+??:?? ${SOURCEIP} ${PROGRAM} ${PRIORITY} ${MSG}\n") Make sense $PROGRAM as "message type"? And do you have any ideas for the question marks? Any help is appreciated. Thank you. -- Saluti, Francesco.

2 2

Insider 2017-05: osquery; SELinux; disk queue; upcoming events
by Czanik, Péter 26 May '17

26 May '17

Dear syslog-ng users, This is the 58th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS osquery and syslog-ng --------------------- osquery allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. From this post, you will learn how to send log messages to osquery, read osquery logs using syslog-ng, and how to parse the JSON-based log messages of osquery, so selected fields can be forwarded to Elasticsearch or other destinations expecting name-value pairs. https://www.balabit.com/blog/endpoint-visibility-and-monitoring-using-osque… SELinux in enforcing mode ------------------------- Security-Enhanced Linux (SELinux) is a set of kernel and user-space tools enforcing strict access control policies. It is also the tool behind at least half of the syslog-ng problem reports. SELinux rules in Linux distributions cover all aspects of the syslog-ng configuration coming in the syslog-ng package available in the distribution. But as soon as an unusual port number or directory name is specified in the configuration, syslog-ng fails to work even with a perfectly legitimate configuration. While preventing unusual access is the main feature of SELinux, it also causes lots of headaches for unsuspecting administrators. Learn how you can use syslog-ng with SELinux in enforcing mode. https://www.balabit.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/ Processing messages stuck in the disk queue files ------------------------------------------------- When you change the configuration of a syslog-ng host that uses disk-based buffering (also called disk queue), syslog-ng may start new disk buffer files for the destinations that you have changed. Learn how you can flush log messages from the orphaned disk queue files from our new document: https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-dis… UPCOMING EVENTS You can learn about syslog-ng at a growing number of events: Big Data Universe: https://bdu.hu/ openSUSE conference: https://events.opensuse.org/conference/oSC17 Libre Software Meeting: https://2017.rmll.info/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 2

Re: [syslog-ng] Syslog-ng input for beats ? [SUMMARY01]
by Scot 26 May '17

26 May '17

Hi All, Just wanted to report my progress on this subject, and say thanks for your input. I'm pretty happy with this setup so far I'll try to keep it short but wanted to share. *tl;dnr* HOWTO send Windows log data through syslog-ng and get it perfectly parsed in splunk and elasticsearch. *Purpose*: Funnel Windows events from enterprise or dmz zones through an remote aggregation relay into syslog-ng, allowing syslog-ng to replicate and broker log data based on business needs. *Components*: *Beats*- Winlogbeat is one of the beats agents by Elastic.co (Elasticsearch) that run on windows hosts to offload events to Logstash in this case. *Logstash*: Like beats it is able to take plugin based input and convert it to a plugin based output. I use it like a syslog relay but for compressed json streams. *syslog-ng:* Primary data aggregation and rule based distribution to SPLUNK,ES and Archive file. Beats support plugin based outputs, the primary two are es and logstash. This solution uses a logstash instance running on the same VM as my rsyslog relay. The output from winlogbeat, running on all windows servers, is a compressed json data stream to its designated logstash relay IP:PORT. This looks like this winlogbeat winlogbeat winlogbeat ---> TCP:5044 --> Logstash --> TCP:9150 --> syslog-ng winlogbeat winlogbeat Systems local to the syslog-ng server also go through a logstash hop. Logstash ads a single aggregator tag to each json message and can also take rule based actions like VictorOps, or massage data inline. syslog-ng then directs a copy of the json stream to a splunk forwarder, archive file, elasticsearch and any other output supported by Balabit. *Advantage*: I can write rules to tier my data to best of class or lower cost data analytic solutions or even new ones with a syslog-ng. *sylog-ng.conf * # This listens for data from logstash output. source s_BEATS {network(port(9150) log-msg-size(65536) flags(no-parse));}; # Define the output to splunkforwarder. destination d_splunk_BEAT {network("splunkforwarderhost" template("$(format-json --scope nv-pairs -p @timestamp=${ISODATE} -x MESSAGE)\n") port(5140));}; # Send a copy to SPLUNK or filter f_splunk_beat log { source(s_BEATS); parser {json-parser();}; destination(d_splunk_BEAT); }; *logstash beats-pipeline.conf output to syslog-ng* output { tcp { host => "syslog-ng host" port => "9150" mode => "client" codec => "json_lines" } }

2 1

How to export syslogng to elasticsearch
by Mik J 19 May '17

19 May '17

Hello, I've read many documentation about how to export syslogng to elasticsearch but still don't get it # syslog-ng -V syslog-ng 3.7.2 Installer-Version: 3.7.2 Revision: Compile-Date: May 8 2017 10:54:55 Available-Modules: afprog,afsocket,afsql,afuser,basicfuncs,confgen,csvparser,dbparser,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,syslogformat,system-source,affile,cryptofuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off In syslog-ng.conf I need to add: @module mod-java @include "scl.conf" In scl.conf I have (I kept the default configuration) @define scl-root "`syslog-ng-data`/include/scl" @define include-path "`include-path`:`syslog-ng-data`/include" @include 'scl/*/*.conf' I don't understand really what should I configure. I also downloaded gradle # ls /usr/local/gradle-3.5 And Java # /usr/local/jdk-1.8.0/bin/java -version openjdk version "1.8.0_121" OpenJDK Runtime Environment (build 1.8.0_121-b13) OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode) Back to syslog-ng.conf I added # Test Elasticsearch filter f_MyTest { host("1.1.1.1"); }; log { source(s_net); filter(f_MyTest); destination(d_MyTest); }; destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("syslog-ng") client_mode("transport") custom_id("${UNIQID}") flush-limit("10000") ); }; log { source(s_net); filter(f_MyTest); destination(d_elastic); flags(flow-control); }; But starting the daemon gives me this error # syslog-ng [2017-05-18T16:43:42.465496] Plugin module not found in 'module-path'; module-path='/usr/local/lib/syslog-ng', module='mod-java' Error parsing destination, destination plugin elasticsearch not found in /etc/syslog-ng/syslog-ng.conf at line 171, column 2: elasticsearch( ^^^^^^^^^^^^^

3 6

syslog-ng with SELinux in enforcing mode
by Czanik, Péter 19 May '17

19 May '17

Hi, Recently a large part of syslog-ng problem reports were related to SELinux not allowing access to some custom network ports or directories. You can read the basics of customizing SELinux (and Firewalld) rules for your environment in my latest blog: https://www.balabit.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

use of Syslog-ng to filter logs for forwarding to Splunk
by Stuart Martin 18 May '17

18 May '17

Hi We are looking to collect logs from servers and devices in our DMZ and then filter the logs of unnecessary information to then be sent to our internal Splunk instance. Would Syslog-ng OSE edition be capable of this task? I've started to read through your excellent documentation for the OSE edition, I was wondering what route I should take to configure it for the scenario described above? The link I am thinking is correct is the "To configure syslog-ng OSE as a relay that receives log messages from client hosts and forwards them to a central logserver, see Procedure 4.2, Configuring syslog-ng on server hosts<https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…>." Does that sound correct? Many Thanks Kind Regards Stuart Martin Infrastructure Security Engineer Surrey Satellite Technology Ltd Tycho House, 20 Stephenson Road, Surrey Research Park, Guildford, GU2 7YE Tel: +44 (0)1483 803803 | Fax: +44 (0)1483 803804 | Email: s.martin(a)sstl.co.uk<mailto:s.martin@sstl.co.uk> www.sstl.co.uk<http://www.sstl.co.uk/> http://twitter.com/SurreySat P Please consider the environment before printing this e-mail This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal. The information contained in this message is intended for the addressee only and may contain sensitive information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organisation. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended.

3 5

DMZ Relay Config (multiple ports)
by wiskbroom＠hotmail.com 18 May '17

18 May '17

Hello! I am trying to build a syslog-ng.conf for a DMZ relay that will listen on several TCP and UDP ports, all working just great. My config essentially captures traffic, then redirects each connection internally from itself. What happens if my internal host goes down? I am hoping to store locally if connectivity is down, but not sure where or how that would get defined. I need to keep a direct relationship between ports, as my internal syslog-ng treats these ports differently, so redirecting to the same as received is key. Below is my DMZ relay syslog-NG config file, any offers to make it better greatly appreciated. -Vadim -------------------------syslog-ng.conf------------------------------ @version:3.5 @include "scl.conf" @include "/etc/syslog-ng/conf.d/*.conf" options { time-reap(30); mark-freq(10); keep-hostname(yes); chain_hostnames (off); flush_lines (0); time_reopen (10); log_fifo_size (1000); use_dns (yes); use_fqdn (no); create_dirs (no); }; source s_udp-relay { udp(ip(0.0.0.0) port(514) so_rcvbuf(425984)); }; source s_tcp-relay-514 { tcp(ip(0.0.0.0) port(514) max-connections(250) so_rcvbuf(425984) log_iw_size(25000) so_keepalive(yes) log_fetch_limit(100)); }; source s_tcp-relay-7514 { tcp(ip(0.0.0.0) port(7514) max-connections(250) so_rcvbuf(425984) log_iw_size(25000) so_keepalive(yes) log_fetch_limit(100)); }; destination d_syslog_udp { udp("10.5.5.5" port(514)); }; destination d_syslog_tcp-514 { tcp("10.5.5.5" port(514)); }; destination d_syslog_tcp-7514 { tcp("10.5.5.5" port(7514)); }; log { source(s_udp-relay); destination(d_syslog_udp); }; log { source(s_tcp-relay-514); destination(d_syslog_tcp-514); }; log { source(s_tcp-relay-7514); destination(d_syslog_tcp-7514); };

3 2

syslog-ng-3.9.1-1 is crashing
by Harish Shetty 17 May '17

17 May '17

Hi Peter I have deployed syslog-ng-3.9.1-1.el6.x86_64 , where I am seeing one issue. Where there is disk usage in greater than 90%, It coredumps. Instead of that shouldnt it be handle the syslog-ng service gracefully? I have pasted part of core here and attached the coredump full file gdb) bt full #0 0x0000003b52a325e5 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x0000003b52a33dc5 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000003206c7506b in iv_fatal (fmt=<value optimized out>) at iv_fatal.c:57 ap = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 0x7f35c5aadac0, reg_save_area = 0x7f35c5aada00}} msg = "iv_init: can't initialize event dispatcher\000\006\062\000\ 000\000\177\000\000\bZ\003\000\000\000\000`KL\234\065\177\ 000\000?M006\062\000\000\000@֪\177\000\000@֪\177\000\000\ 177\000\000\b\000\000\000\000\000\000\000L\234\065\177\000\ 000L\234\065\177\000\000\000KL\234\065\177\000\000Ye006\062\000\000\000\001\ 000\000\000\000\000\000\000ت\177\000\000ت\177\000\000\177\000\000ت\177", '\000' <repeats 46 times>, "\004", '\000' <repeats 35 times>... #3 0x0000003206c76f2a in iv_fd_init (st=0x7f359c945850) at iv_fd.c:129 No locals. #4 0x0000003206c77499 in iv_init () at iv_main_posix.c:74 st = 0x7f359c945850 #5 0x0000003206c75b8e in iv_work_thread (_thr=0x996c540) at iv_work.c:177 thr = 0x996c540 pool = 0x1df6820 #6 0x0000003206c77def in iv_thread_handler (_thr=0x52933d0) at iv_thread_posix.c:142 __clframe = {__cancel_routine = 0x3206c77e50 <iv_thread_cleanup_handler>, __cancel_arg = 0x52933d0, __do_it = 1, __cancel_type = 0} thr = 0x52933d0 #7 0x0000003b53207aa1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #8 0x0000003b52ae8aad in clone () from /lib64/libc.so.6 No symbol table info available. Can you please check and let me know how can we fix this issue? Regards Harish shetty

4 9