- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Filter Not Working (too many or's?)
by james.r.hendrick 09 Jun '17

09 Jun '17

Not exactly what you asked but I have used netmask to simplify some filters. Although it probably won't be able to solve your problem. Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Szalai, Attila" <Attila.Szalai(a)morganstanley.com> Date: 6/6/17 8:07 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Filter Not Working (too many or's?) Hi, First of all, the content of the host() is a regular expression, so adding .* to the beginning and/or to the end of the expression adds nothing, just pain/slowness. Second, it would help a lot if we can see the actual error message. I found no obvious mistake, but because this is not the original line, maybe something lost in the translation. From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of wiskbroom(a)hotmail.com Sent: Tuesday, June 06, 2017 12:59 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] Filter Not Working (too many or's?) Here is an example of what I am trying to do, these hostnames are not real; the real ones have no common pattern. filter f_xyz { host("*.abca.*") or host(".*abcb.*") or host(".*abcc.*") or host(".*abcd.*") or host(".*abce.*") or host(".*abcf.*") or host(".*abcg.*") or host(".*abch.*"); }; The filter above is for any host containing a hostname with what is contained within the .* and *.; i.e. hostabca01 will be matched by host("*.abca.*") When I have this filter in my config, syslog fails to restart. Eyes hurt, obvious mistake? NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.

5 4

Duplicate header ?
by Scot 08 Jun '17

08 Jun '17

I seem to be getting a duplicate host header in my udp syslog input where the IP is printed twice. IP/IP any ideas where it comes from ? Same result for either of these. #source s_net_udp {udp(ip(0.0.0.0) port(514) keep_hostname(yes) so_rcvbuf(262142));}; #source s_net_udp {syslog(ip(10.189.252.62) port(514) transport("udp") flags(no-hostname) so_rcvbuf(262142));}; Jun 8 13:55:21 *192.168.10.10/192.168.10.10 <http://192.168.10.10/192.168.10.10> * fw-aplha %ASA-4-106 ..............

3 3

Filter Not Working (too many or's?)
by wiskbroom＠hotmail.com 06 Jun '17

06 Jun '17

Here is an example of what I am trying to do, these hostnames are not real; the real ones have no common pattern. filter f_xyz { host("*.abca.*") or host(".*abcb.*") or host(".*abcc.*") or host(".*abcd.*") or host(".*abce.*") or host(".*abcf.*") or host(".*abcg.*") or host(".*abch.*"); }; The filter above is for any host containing a hostname with what is contained within the .* and *.; i.e. hostabca01 will be matched by host("*.abca.*") When I have this filter in my config, syslog fails to restart. Eyes hurt, obvious mistake?

3 3

RPM packages from syslog-ng git head
by Czanik, Péter 06 Jun '17

06 Jun '17

Hi, Test the latest syslog-ng features and bug fixes! They are available for RPM platforms for easy installation. I build packages from git head snapshots regularly for the following Linux distribution versions: openSUSE SLES 12 Fedora RHEL/CentOS 7 You can read how to get started with these packages at: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Do stats cause memory leak
by Evan Rempel 06 Jun '17

06 Jun '17

We are running git head download post 3.9.1 and the memory footprint seems to continually grow. Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195 It it possible that all of these statistics take up a non-trivial amount of RAM? USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 10139760 ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng That is a 12GB footprint after 1 month of uptime. Evan.

4 8

docker based, local builds
by Scheidler, Balázs 05 Jun '17

05 Jun '17

Hi, I've just submitted a PR that makes it easy to build rpms and debs locally, out of a freshly checked out syslog-ng tree and/or a release tarball. If you download a release tarball (one that contains debian packaging files), all you need to build it to Ubuntu or Debian is: # this one builds the Ubuntu build image locally $ dbld/rules image-zesty $ dbld/rules deb-zesty But rpm-centos7 and deb-jessie also work. Check out the help screen for dbld/rules and the PR. This should make it pretty trivial to get binaries that you can deploy in production. This is just a bare-bones stuff, later on support can be added to a lot more distros. If you want to use it in a freshly cloned copy of syslog-ng, checked out from github, that will not have debian packaging (at least for now), you'll need to merge the "release" branch in order to get them. # this one gets the debian/rpm packaging locally $ git clone git@github.com:balabit/syslog-ng.git $ cd syslog-ng $ git merge origin/release $ dbld/rules image-centos7 $ dbld/rules rpm-centos7 I still don't like that the "release" branch lives separate, but I can't change that without discussing it with our Debian maintainer first. -- Bazsi

1 0

syslog 1024 byte limit work around
by Corey Davelaar 31 May '17

31 May '17

Hello, I'm new to the group and hopefully it's not a repeat question. I was guided to this mailing list by a balabit tech support rep. We use splunk on the back end of syslog-ng and we have some syslog sources that utilize very long syslog messages, that end up being over 1024 bytes. We switched from udp to tcp as the transport and now we are seeing the tail end of the syslog messages, but splunk gets confused by the inputs and treats each line as a separate syslog message and completely unrelated to the previous one, because that’s how syslog-ng is treating it. I was wondering if there is a way to tell syslog-ng that if a new packet doesn’t start with a standard syslog timestamp to treat it as a continuation of the previous message or somehow combine it with a previous message, utilizing a buffer type of setting. Thanks, Corey

3 5

Sawmill and Syslog-NG template() string
by Francesco Rolando 31 May '17

31 May '17

Hi all, I'm working with Sawmill log collector that it's able to import the Syslog-NG logs. I have to export logs with a specific template() to have them corretly imported into Sawmill. Here the regular expression used to match syslog-ng lines inside the Sawmill parser (that seem coded by "BalaBit IT Security"): '^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])T([0-9][0-9]:[0-9][0-9]:[0-9][0-9])[-+][0-9][0-9]:*[0-9][0-9] ([^ ]+) ([^ ]+) ([^ ]+) (.*)$' where matches are imported as: 1- date 2- time 3- logging_device 4- syslog_message_type 5- syslog_priority 6- v.syslog_message I have partially re-create the template() string but I still have a couple of doubts: template("${YEAR}-${MONTH}-${DAY}T${HOUR}:${MIN}:${SEC}+??:?? ${SOURCEIP} ${PROGRAM} ${PRIORITY} ${MSG}\n") Make sense $PROGRAM as "message type"? And do you have any ideas for the question marks? Any help is appreciated. Thank you. -- Saluti, Francesco.

2 2

Insider 2017-05: osquery; SELinux; disk queue; upcoming events
by Czanik, Péter 26 May '17

26 May '17

Dear syslog-ng users, This is the 58th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS osquery and syslog-ng --------------------- osquery allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. From this post, you will learn how to send log messages to osquery, read osquery logs using syslog-ng, and how to parse the JSON-based log messages of osquery, so selected fields can be forwarded to Elasticsearch or other destinations expecting name-value pairs. https://www.balabit.com/blog/endpoint-visibility-and-monitoring-using-osque… SELinux in enforcing mode ------------------------- Security-Enhanced Linux (SELinux) is a set of kernel and user-space tools enforcing strict access control policies. It is also the tool behind at least half of the syslog-ng problem reports. SELinux rules in Linux distributions cover all aspects of the syslog-ng configuration coming in the syslog-ng package available in the distribution. But as soon as an unusual port number or directory name is specified in the configuration, syslog-ng fails to work even with a perfectly legitimate configuration. While preventing unusual access is the main feature of SELinux, it also causes lots of headaches for unsuspecting administrators. Learn how you can use syslog-ng with SELinux in enforcing mode. https://www.balabit.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/ Processing messages stuck in the disk queue files ------------------------------------------------- When you change the configuration of a syslog-ng host that uses disk-based buffering (also called disk queue), syslog-ng may start new disk buffer files for the destinations that you have changed. Learn how you can flush log messages from the orphaned disk queue files from our new document: https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-dis… UPCOMING EVENTS You can learn about syslog-ng at a growing number of events: Big Data Universe: https://bdu.hu/ openSUSE conference: https://events.opensuse.org/conference/oSC17 Libre Software Meeting: https://2017.rmll.info/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 2

Re: [syslog-ng] Syslog-ng input for beats ? [SUMMARY01]
by Scot 26 May '17

26 May '17

Hi All, Just wanted to report my progress on this subject, and say thanks for your input. I'm pretty happy with this setup so far I'll try to keep it short but wanted to share. *tl;dnr* HOWTO send Windows log data through syslog-ng and get it perfectly parsed in splunk and elasticsearch. *Purpose*: Funnel Windows events from enterprise or dmz zones through an remote aggregation relay into syslog-ng, allowing syslog-ng to replicate and broker log data based on business needs. *Components*: *Beats*- Winlogbeat is one of the beats agents by Elastic.co (Elasticsearch) that run on windows hosts to offload events to Logstash in this case. *Logstash*: Like beats it is able to take plugin based input and convert it to a plugin based output. I use it like a syslog relay but for compressed json streams. *syslog-ng:* Primary data aggregation and rule based distribution to SPLUNK,ES and Archive file. Beats support plugin based outputs, the primary two are es and logstash. This solution uses a logstash instance running on the same VM as my rsyslog relay. The output from winlogbeat, running on all windows servers, is a compressed json data stream to its designated logstash relay IP:PORT. This looks like this winlogbeat winlogbeat winlogbeat ---> TCP:5044 --> Logstash --> TCP:9150 --> syslog-ng winlogbeat winlogbeat Systems local to the syslog-ng server also go through a logstash hop. Logstash ads a single aggregator tag to each json message and can also take rule based actions like VictorOps, or massage data inline. syslog-ng then directs a copy of the json stream to a splunk forwarder, archive file, elasticsearch and any other output supported by Balabit. *Advantage*: I can write rules to tier my data to best of class or lower cost data analytic solutions or even new ones with a syslog-ng. *sylog-ng.conf * # This listens for data from logstash output. source s_BEATS {network(port(9150) log-msg-size(65536) flags(no-parse));}; # Define the output to splunkforwarder. destination d_splunk_BEAT {network("splunkforwarderhost" template("$(format-json --scope nv-pairs -p @timestamp=${ISODATE} -x MESSAGE)\n") port(5140));}; # Send a copy to SPLUNK or filter f_splunk_beat log { source(s_BEATS); parser {json-parser();}; destination(d_splunk_BEAT); }; *logstash beats-pipeline.conf output to syslog-ng* output { tcp { host => "syslog-ng host" port => "9150" mode => "client" codec => "json_lines" } }

2 1