- syslog-ng - lists.balabit.hu

my syslog-ng-githead repo now depends on Java 1.8
by Czanik, Péter 25 Apr '18

25 Apr '18

Hi, Recently I updated my experimental syslog-ng repo built from git head for RHEL/CentOS 7 & Fedora to use openjdk 1.8 instead of 1.7, as it might support Oracle Java packages better (no need to install both openjdk and Oracle java). @Evan: please give it a try, if it fixes your problem: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ @everyone: would you mind, if I update the dependency also for my unofficial packages for the next release? (3.15) As far as I can see, it's not openjdk 1.7 but 1.8 which is installed by default on CentOS 7. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Bug report: syslog-ng requests too many capabilities
by Russenberger Dominik 24 Apr '18

24 Apr '18

Hi List, I recently noticed something very strange: although I run syslog-ng as an unprivileged user (with -u log -g log), newly created logfiles were owned by root. syslog-ng shows up running as user log in ps, as expected. The culprit turned out to be linux capabilities. No problem I thought, I just add a "--caps 'CAP_NET_BIND_SERVICE=epi'" command line parameter (binding ports <1024 is the only capability I need). But then I was getting flooded by messages like this: syslog-ng[25657]: Error managing capability set, cap_set_proc returned an error; caps='= cap_net_bind_service+eip cap_fowner+e', error='Operation not permitted (1)' In my opinion, there are 2 bugs in syslog-ng: * if I tell a daemon to run as unprivileged user I do not expect it to write files as user root. What syslog-ng is doing basically is faking being an unprivileged user, while retaining capabilities which are equivalent to full root permissions. Syslog-ng should imho either run as root, with capabilities; OR as unprivileged user without capabilities (except those explicitly given in --caps) * syslog-ng drops to the capabilities it gets told in --caps, but later g_process_cap_modify() ignores what was specified. Some notes about my environment: - the OS is an Ubuntu remix. I'm recompiling syslog-ng 3.11.1 from Ubuntu Artful, with some changed configure flags (see below) - there are multiple syslog-ng instances running. One for the system itself (as root), and multiple other syslogs receiving incoming logs over udp/tcp (running as user log) My syslog-ng version: syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Apr 10 2017 14:06:00 Module-Directory: /usr/lib/syslog-ng/3.11 Module-Path: /usr/lib/syslog-ng/3.11 Available-Modules: disk-buffer,afsocket,afprog,system-source,kvformat,cef,affile,date,cryptofuncs,basicfuncs,confgen,pseudofile,csvparser,afuser,syslogformat,dbparser,linux-kmsg-format Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Systemd: off Thanks, Dominik

4 4

Need clarification on different behavior of forwarding rules in syslog-ng
by Kavita Mohite (kmohite) 18 Apr '18

18 Apr '18

Hello Support Team, In the syslog-ng configuration, we have 2 forward rules as follows: 1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug. 2. Second rule reads from source (port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule. Now we have syslogs coming in at a rate of 400 eps. Following is our observation: A. If the syslog matches the filter in point 2 above, then following happens · It is forwarded to the destination provided in point 2. · Also at the same time it is written to a file as per rule in point 1. B. If the syslog does not match the filter in point 2 above, then following happens · It is NOT forwarded to the destination provided in point 2 as there is no match. · It is suppose to still write to file as the rule in 1 matches. But this does not happen. We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ? Below is the snap of the config values we are using in the syslog-ng.conf options { flush_lines (100); log_fetch_limit(1000); log_iw_size(10000); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); threaded(yes); }; Thanks Kavita

2 1

Need clarification on different behavior of forwarding rules in syslog-ng
by Kavita Salve 18 Apr '18

18 Apr '18

Hello Support Team, In the syslog-ng configuration, we have 2 forward rules as follows: 1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug. 2. Second rule reads from source (port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule. Now we have syslogs coming in at a rate of 400 eps. Following is our observation: A. If the syslog matches the filter in point 2 above, then following happens - It is forwarded to the destination provided in point 2. - Also at the same time it is written to a file as per rule in point 1. B. If the syslog does not match the filter in point 2 above, then following happens - It is NOT forwarded to the destination provided in point 2 as there is no match. - It is suppose to still write to file as the rule in 1 matches. *But this does not happen.* We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. *Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ?* Below is the snap of the config values we are using in the syslog-ng.conf options { flush_lines (100); log_fetch_limit(1000); log_iw_size(10000); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); threaded(yes); }; Thanks Kavita Mohite

1 1

Re: [syslog-ng] Support for Open Source Syslog-ng
by Naveen Bhalla (nbhalla) 14 Apr '18

14 Apr '18

Team, Could you pls help us on below P1 situation for us? Regards, [http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signatur…] Naveen Bhalla | Manager.Technical Support CMS Platform Operations Cell: +91-9880362157 Desk: +91-80-44260795 From: Naveen Bhalla (nbhalla) Sent: 13 April 2018 09:14 PM To: 'support(a)balabit.com' <support(a)balabit.com<mailto:support@balabit.com>> Subject: Support for Open Source Syslog-ng Hello Support Team, We have a situation in our platform where syslog-ng is dropping part of syslog traffic coming into our server. The syslog-ng has around 750 match rules in its configuration. Based on these rules the syslogs are forwarded to the destinations. Also, there is one rule to write all the received syslogs to disk. We are receiving syslogs at the rate of 300 eps. The issue is that we are seeing that syslog-ng is not able to process the syslogs and forward them to the destinations. It is not writing to the disk also. We are seeing that there is a big delay after which some syslogs are getting written to the disk. We are seeing loss of UDP packets. The UDP buffer size is big enough. net.ipv4.tcp_rmem = 4096 4194304 16777216 net.ipv4.tcp_wmem = 98304 4194304 16777216 net.core.rmem_default = 234217728 net.core.wmem_default = 234217728 net.core.rmem_max = 234217728 net.core.wmem_max = 234217728 net.ipv4.tcp_window_scaling = 1 net.ipv4.ip_local_port_range = 32768 61000 fs.file-max = 2097152 net.core.optmem_max = 40960 net.core.netdev_max_backlog = 50000 net.ipv4.udp_rmem_min = 8192 net.ipv4.udp_wmem_min = 8192 We need help to resolve this issue. We are using open-source syslog-ng in our setup. Regards, [http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signatur…] Naveen Bhalla | Manager.Technical Support CMS Platform Operations Cell: +91-9880362157 Desk: +91-80-44260795

3 2

Insider 2018-04: RHEL6; Windows Event Log; Patched RPM; HEC; Conferences;
by Czanik, Péter 12 Apr '18

12 Apr '18

Dear syslog-ng users, This is the 66th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng is available for RHEL 6 / CentOS 6 again -------------------------------------------------- After more than a year, the latest version of syslog-ng compiles again on Red Hat Enterprise Linux version 6. This is thanks to a patch from Balázs Scheidler which resolves compatibility problems with older glib releases by implementing the missing functions within syslog-ng. https://syslog-ng.com/blog/latest-syslog-ng-available-rhel-6-centos-6/ Agentless Windows logging ------------------------- With syslog-ng Premium Edition, you can collect Windows event logs without installing any third-party application on your Windows-based computer. Learn more about it from our blog at https://syslog-ng.com/blog/how-to-collect-windows-event-logs-with-syslog-ng… or join our webinar where you can also ask questions: https://www.brighttalk.com/webcast/16207/314789?utm_source=OSE/ Build your own syslog-ng RPM ---------------------------- When a user reports a problem with syslog-ng, developers create code to fix it. The fix is called a patch or a pull request (PR) in Git terminology. Often this code does not enter the main source code automatically but developers ask the user to test the fix before the code is merged. A new challenge starts here: figuring out how to test a proposed fix. This blog shows you how to download and patch the syslog-ng source code and build easy-to-install (and remove) RPM packages from it. https://syslog-ng.com/blog/build-syslog-ng-rpm-patched-git-sources-rhel-cen… While this blog introduces you to package building in depth, a lot simpler method is also available for Ubuntu packages and should be working soon again for RHEL / CentOS as well: https://github.com/balabit/syslog-ng/tree/master/dbld Sending logs to Splunk HEC -------------------------- Recently Splunk started to recommend the use of the HTTP Event Collector (HEC) instead of forwarders. syslog-ng supports this in multiple ways. In a previous blog, we showed you how to use the http() destination of syslog-ng. This time, we introduce you to another possibility: using an external python script to send logs to HEC. Read our blog at https://syslog-ng.com/blog/splunk-hec-sending-logs-using-program-destinatio… or watch the webinar at https://www.brighttalk.com/webcast/16207/308061 Conferences: CentOS Dojo, FOSDEM and SCALE ------------------------------------------ Recently syslog-ng was presented at some popular open source conferences. Read about the presentations and experiences in related blogs: CentOS Dojo & FOSDEM: https://syslog-ng.com/blog/centos-dojo-fosdem-2018/ SCALE: https://syslog-ng.com/blog/syslog-ng-scale-2018/ UPCOMING EVENTS LOADays: http://loadays.org/ (Antwerp) Red Hat Summit: https://www.redhat.com/en/summit/2018 (San Francisco) Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

What Im missing to enable the wildcard-file module?
by Eric Anderson 12 Apr '18

12 Apr '18

Im trying to use the wildcard-file module, but the syslog-ng Module: wildcard-file is missing # service syslog-ng start Error parsing source, source plugin wildcard-file not found in /etc/syslog-ng/conf.d/export-client.conf at line 4, column 29: included from /etc/syslog-ng/syslog-ng.conf line 71, column 1 source s_export_recursive { wildcard-file( ^^^^^^^^^^^^^ I upgraded to the latest syslog-ng # syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: kvformat,afprog,afsocket,basicfuncs,linux-kmsg-format,graphite,cef,curl,add-contextual-data,date,json-plugin,pseudofile,afstomp,dbparser,confgen,cryptofuncs,syslogformat,disk-buffer,afuser,afamqp,csvparser,affile,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on But I still don’t see where the wildcard-file is. What Im missing to enable the wildcard-file module? The conf file Ive created where I want to use it looks like this: # syslog-ng conf file for central logging CLIENT: export # # export source source s_export_recursive { wildcard-file( base-dir("/var/www/capstan/shared/log/exports/") filename-pattern("*.log") recursive(yes) follow-freq(1) flags(no-parse) log-fetch-limit(100) ); }; # export destination destination d_export_app1 { tcp(10.0.32.97 port(601)); }; # export log log { source(s_export_recursive); destination(d_export_app1); }; Eric Anderson | DevOps Engineer E: eric.anderson(a)stonecroptech.com<mailto:michaelg@stonecroptech.com> | O: 212.500.2017 | M: 650.743.4598 [cid:image001.png@01D3D1B1.4BE3C030]

2 1

in-list for message
by Garbacik, Joe 11 Apr '18

11 Apr '18

Can syslog-ng parse the message field for a partial match using the in-list function? For instance, I am querying DNS logs and want to filter out common domains (i.e. Microsoft.com, Redhat.com) In my list file I just have the domains, one per line and in my config file, if have the following: filter f_dns_noise_oklist { in-list("/etc/syslog-ng/lists.d/dns_noise.list", value("MESSAGE")); }; but it doesn’t seem to be working.

2 1

NG 6.0.9 UDP Forwarding & Spoof Source
by Schoonover, Mark E HHHH 11 Apr '18

11 Apr '18

Hello, I'm using NG to forward via UDP to QRadar platform. We've noticed that long messages get truncated to 1024 bytes. I thought it was because of forwarding using RFC3164 which has a limit of 1024 but forwarding using RFC5424 does not have a message limit. In the manual though for the spoof-source option there's this warning: When using the spoof-source option, syslog-ng PE automatically truncates long messages to 1024 bytes, regardless of the settings of log-msg-size(). Does this mean no matter what, the max UDP forwarded message spoofing the source is 1024 bytes regardless of RFC? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of Cigna. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2018 Cigna. ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2018 Cigna ==============================================================================

2 1

Problem activating: Centos7 + Syslog-ng OSE + Elasticsearch + kibana
by André Grehs 09 Apr '18

09 Apr '18

Hi group! I installed a fresh Centos 7 setup to use it in a initial log management Project. I´m made the procedures described at thw wizard https://syslog-ng.com/blog/syslog-ng-and-elasticsearch-6-getting-started-on- rhelcentos/ All is fine, but at the moment of the creation of the es.conf file of this procedure, the syslog-ng service could not start anymore. If i remove the es.conf file, the syslog-ng service start normally. Can you help me with this problem? P.S: Sorry for my english Best Regards! Andre.

5 10