April 2018 - syslog-ng - lists.balabit.hu

Insider 2018-04: RHEL6; Windows Event Log; Patched RPM; HEC; Conferences;
by Czanik, Péter 12 Apr '18

12 Apr '18

Dear syslog-ng users, This is the 66th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng is available for RHEL 6 / CentOS 6 again -------------------------------------------------- After more than a year, the latest version of syslog-ng compiles again on Red Hat Enterprise Linux version 6. This is thanks to a patch from Balázs Scheidler which resolves compatibility problems with older glib releases by implementing the missing functions within syslog-ng. https://syslog-ng.com/blog/latest-syslog-ng-available-rhel-6-centos-6/ Agentless Windows logging ------------------------- With syslog-ng Premium Edition, you can collect Windows event logs without installing any third-party application on your Windows-based computer. Learn more about it from our blog at https://syslog-ng.com/blog/how-to-collect-windows-event-logs-with-syslog-ng… or join our webinar where you can also ask questions: https://www.brighttalk.com/webcast/16207/314789?utm_source=OSE/ Build your own syslog-ng RPM ---------------------------- When a user reports a problem with syslog-ng, developers create code to fix it. The fix is called a patch or a pull request (PR) in Git terminology. Often this code does not enter the main source code automatically but developers ask the user to test the fix before the code is merged. A new challenge starts here: figuring out how to test a proposed fix. This blog shows you how to download and patch the syslog-ng source code and build easy-to-install (and remove) RPM packages from it. https://syslog-ng.com/blog/build-syslog-ng-rpm-patched-git-sources-rhel-cen… While this blog introduces you to package building in depth, a lot simpler method is also available for Ubuntu packages and should be working soon again for RHEL / CentOS as well: https://github.com/balabit/syslog-ng/tree/master/dbld Sending logs to Splunk HEC -------------------------- Recently Splunk started to recommend the use of the HTTP Event Collector (HEC) instead of forwarders. syslog-ng supports this in multiple ways. In a previous blog, we showed you how to use the http() destination of syslog-ng. This time, we introduce you to another possibility: using an external python script to send logs to HEC. Read our blog at https://syslog-ng.com/blog/splunk-hec-sending-logs-using-program-destinatio… or watch the webinar at https://www.brighttalk.com/webcast/16207/308061 Conferences: CentOS Dojo, FOSDEM and SCALE ------------------------------------------ Recently syslog-ng was presented at some popular open source conferences. Read about the presentations and experiences in related blogs: CentOS Dojo & FOSDEM: https://syslog-ng.com/blog/centos-dojo-fosdem-2018/ SCALE: https://syslog-ng.com/blog/syslog-ng-scale-2018/ UPCOMING EVENTS LOADays: http://loadays.org/ (Antwerp) Red Hat Summit: https://www.redhat.com/en/summit/2018 (San Francisco) Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

What Im missing to enable the wildcard-file module?
by Eric Anderson 12 Apr '18

12 Apr '18

Im trying to use the wildcard-file module, but the syslog-ng Module: wildcard-file is missing # service syslog-ng start Error parsing source, source plugin wildcard-file not found in /etc/syslog-ng/conf.d/export-client.conf at line 4, column 29: included from /etc/syslog-ng/syslog-ng.conf line 71, column 1 source s_export_recursive { wildcard-file( ^^^^^^^^^^^^^ I upgraded to the latest syslog-ng # syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: kvformat,afprog,afsocket,basicfuncs,linux-kmsg-format,graphite,cef,curl,add-contextual-data,date,json-plugin,pseudofile,afstomp,dbparser,confgen,cryptofuncs,syslogformat,disk-buffer,afuser,afamqp,csvparser,affile,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on But I still don’t see where the wildcard-file is. What Im missing to enable the wildcard-file module? The conf file Ive created where I want to use it looks like this: # syslog-ng conf file for central logging CLIENT: export # # export source source s_export_recursive { wildcard-file( base-dir("/var/www/capstan/shared/log/exports/") filename-pattern("*.log") recursive(yes) follow-freq(1) flags(no-parse) log-fetch-limit(100) ); }; # export destination destination d_export_app1 { tcp(10.0.32.97 port(601)); }; # export log log { source(s_export_recursive); destination(d_export_app1); }; Eric Anderson | DevOps Engineer E: eric.anderson(a)stonecroptech.com<mailto:michaelg@stonecroptech.com> | O: 212.500.2017 | M: 650.743.4598 [cid:image001.png@01D3D1B1.4BE3C030]

2 1

in-list for message
by Garbacik, Joe 11 Apr '18

11 Apr '18

Can syslog-ng parse the message field for a partial match using the in-list function? For instance, I am querying DNS logs and want to filter out common domains (i.e. Microsoft.com, Redhat.com) In my list file I just have the domains, one per line and in my config file, if have the following: filter f_dns_noise_oklist { in-list("/etc/syslog-ng/lists.d/dns_noise.list", value("MESSAGE")); }; but it doesn’t seem to be working.

2 1

NG 6.0.9 UDP Forwarding & Spoof Source
by Schoonover, Mark E HHHH 11 Apr '18

11 Apr '18

Hello, I'm using NG to forward via UDP to QRadar platform. We've noticed that long messages get truncated to 1024 bytes. I thought it was because of forwarding using RFC3164 which has a limit of 1024 but forwarding using RFC5424 does not have a message limit. In the manual though for the spoof-source option there's this warning: When using the spoof-source option, syslog-ng PE automatically truncates long messages to 1024 bytes, regardless of the settings of log-msg-size(). Does this mean no matter what, the max UDP forwarded message spoofing the source is 1024 bytes regardless of RFC? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of Cigna. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2018 Cigna. ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2018 Cigna ==============================================================================

2 1

Problem activating: Centos7 + Syslog-ng OSE + Elasticsearch + kibana
by André Grehs 09 Apr '18

09 Apr '18

Hi group! I installed a fresh Centos 7 setup to use it in a initial log management Project. I´m made the procedures described at thw wizard https://syslog-ng.com/blog/syslog-ng-and-elasticsearch-6-getting-started-on- rhelcentos/ All is fine, but at the moment of the creation of the es.conf file of this procedure, the syslog-ng service could not start anymore. If i remove the es.conf file, the syslog-ng service start normally. Can you help me with this problem? P.S: Sorry for my english Best Regards! Andre.

5 10

JSON Filter
by Scot 04 Apr '18

04 Apr '18

Sorry if I'm missing this but whats the best way to implement a json filter like these in syslog-ng, Patterndb? This is my logstash filter that sends data to a specific syslog-ng PORT for each condition which may grow and become a management headache when there are 6 logstash hosts that will need to support a filter for each breakout needed. I'd like to just send everything to a single syslog-ng port and have syslog-ng do the logic. which would then become. input { beats { port => 5044 } } output{ tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } *Logstash bloated output filters. * output{ if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }

3 4