- syslog-ng - lists.balabit.hu

What Im missing to enable the wildcard-file module?
by Eric Anderson 12 Apr '18

12 Apr '18

Im trying to use the wildcard-file module, but the syslog-ng Module: wildcard-file is missing # service syslog-ng start Error parsing source, source plugin wildcard-file not found in /etc/syslog-ng/conf.d/export-client.conf at line 4, column 29: included from /etc/syslog-ng/syslog-ng.conf line 71, column 1 source s_export_recursive { wildcard-file( ^^^^^^^^^^^^^ I upgraded to the latest syslog-ng # syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: kvformat,afprog,afsocket,basicfuncs,linux-kmsg-format,graphite,cef,curl,add-contextual-data,date,json-plugin,pseudofile,afstomp,dbparser,confgen,cryptofuncs,syslogformat,disk-buffer,afuser,afamqp,csvparser,affile,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on But I still don’t see where the wildcard-file is. What Im missing to enable the wildcard-file module? The conf file Ive created where I want to use it looks like this: # syslog-ng conf file for central logging CLIENT: export # # export source source s_export_recursive { wildcard-file( base-dir("/var/www/capstan/shared/log/exports/") filename-pattern("*.log") recursive(yes) follow-freq(1) flags(no-parse) log-fetch-limit(100) ); }; # export destination destination d_export_app1 { tcp(10.0.32.97 port(601)); }; # export log log { source(s_export_recursive); destination(d_export_app1); }; Eric Anderson | DevOps Engineer E: eric.anderson(a)stonecroptech.com<mailto:michaelg@stonecroptech.com> | O: 212.500.2017 | M: 650.743.4598 [cid:image001.png@01D3D1B1.4BE3C030]

2 1

in-list for message
by Garbacik, Joe 11 Apr '18

11 Apr '18

Can syslog-ng parse the message field for a partial match using the in-list function? For instance, I am querying DNS logs and want to filter out common domains (i.e. Microsoft.com, Redhat.com) In my list file I just have the domains, one per line and in my config file, if have the following: filter f_dns_noise_oklist { in-list("/etc/syslog-ng/lists.d/dns_noise.list", value("MESSAGE")); }; but it doesn’t seem to be working.

2 1

NG 6.0.9 UDP Forwarding & Spoof Source
by Schoonover, Mark E HHHH 11 Apr '18

11 Apr '18

Hello, I'm using NG to forward via UDP to QRadar platform. We've noticed that long messages get truncated to 1024 bytes. I thought it was because of forwarding using RFC3164 which has a limit of 1024 but forwarding using RFC5424 does not have a message limit. In the manual though for the spoof-source option there's this warning: When using the spoof-source option, syslog-ng PE automatically truncates long messages to 1024 bytes, regardless of the settings of log-msg-size(). Does this mean no matter what, the max UDP forwarded message spoofing the source is 1024 bytes regardless of RFC? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of Cigna. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2018 Cigna. ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2018 Cigna ==============================================================================

2 1

Problem activating: Centos7 + Syslog-ng OSE + Elasticsearch + kibana
by André Grehs 09 Apr '18

09 Apr '18

Hi group! I installed a fresh Centos 7 setup to use it in a initial log management Project. I´m made the procedures described at thw wizard https://syslog-ng.com/blog/syslog-ng-and-elasticsearch-6-getting-started-on- rhelcentos/ All is fine, but at the moment of the creation of the es.conf file of this procedure, the syslog-ng service could not start anymore. If i remove the es.conf file, the syslog-ng service start normally. Can you help me with this problem? P.S: Sorry for my english Best Regards! Andre.

5 10

JSON Filter
by Scot 04 Apr '18

04 Apr '18

Sorry if I'm missing this but whats the best way to implement a json filter like these in syslog-ng, Patterndb? This is my logstash filter that sends data to a specific syslog-ng PORT for each condition which may grow and become a management headache when there are 6 logstash hosts that will need to support a filter for each breakout needed. I'd like to just send everything to a single syslog-ng port and have syslog-ng do the logic. which would then become. input { beats { port => 5044 } } output{ tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } *Logstash bloated output filters. * output{ if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }

3 4

perf testing problem
by Czanik, Péter 31 Mar '18

31 Mar '18

Hi, When I try to use loggen with more than ~20 active connections I get the following error: "Send error Connection reset by peer, results may be skewed." The network source has max_connections(128) already configured, so it should be handle that many connections without any trouble. Do you have any suggestions where to look either in syslog-ng or at OS level? It is Ubuntu 17.10 and syslog-ng 3.11 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

Linux or OS fingerprint filter
by Scot 29 Mar '18

29 Mar '18

Pretty sure I know the answer but just maybe.. Anyone have a solution OS fingerprint type filters ? Solaris, Linux vs Cisco for example. Immediate need is to pluck all Linux host from 514. Thanks ! Scot

3 2

Re: [syslog-ng] Syslog-ng shipping logs through AMQP with huge memory leaks
by Nagy, Gábor 29 Mar '18

29 Mar '18

I'm resending this letter to the mail list as we hit the message size limit. :) Hello Michal! I have some good news. We have found the reason for the increasing memory usage of syslog-ng in amqp() driver. It is due to our internal memory handling method (aka scratch_buffers) which we use in performance critical paths to prevent the overhead of memory allocations/deallocations of GStrings. It is using memory from a pre-allocated pool instead of allocating/freeing. As I have described before, the memory usage only increases if the number of queued messages reaches a new maximum. It is not per message and does not happen every for every queued message. We have several options how to fix this issue and will soon share one. Regards, Gabor

3 6

syslog-ng-java RPM requirements will not use Oracle Java
by Evan Rempel 27 Mar '18

27 Mar '18

Using the copr builds of syslog-ng 3.14 The package syslog-ng-java has RPM requirements that include libjvm.so()(64bit) libjvm.so(SUNWprivate_1.1)(64bit) The Java packages from Oracle do not provide these items. % rpm -q --provides -p jdk-8u162-linux-x64.rpm jdk jaxp_parser_impl xml-commons-apis java java-1.8.0 java-fonts jre jre-1.8.0 jdk1.8 = 2000:1.8.0_162-fcs % rpm -q --provides -p jre-8u162-linux-x64.rpm jaxp_parser_impl xml-commons-apis java java-1.8.0 java-fonts jre jre-1.8.0 jre1.8 = 1.8.0_161-fcs [% rpm -q --provides jdk-9.0.4_linux-x64_bin.rpm jdk jaxp_parser_impl xml-commons-apis java java-9.0.4 java-fonts jre jre-9.0.4 jdk-9.0.4 = 2000:9.0.4-ga jdk-9.0.4(x86-64) = 2000:9.0.4-ga Under Redhat 7 in order to get these I need libjvm.so()(64bit) --> java-1.8.0-ibm libjvm.so(SUNWprivate_1.1)(64bit) --> java-1.8.0-openjdk-headless The Oracle provided java (jdk or jre) do not provide these things. Both of these java packages provides the lib jvm.so % rpm -ql java-1.8.0-ibm | grep libjvm /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/bin/classic/libjvm.so /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/bin/j9vm/libjvm.so /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/lib/amd64/classic/libjvm.so /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/lib/amd64/compressedrefs/libjvm.so /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/lib/amd64/default/libjvm.so /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/lib/amd64/j9vm/libjvm.so % rpm -ql java-1.8.0-openjdk-headless | grep libjvm /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so Both of these package provide the following potentials for a dependency jre = 1:1.8.0 java = 1:1.8.0 or if you want to tie to a specific version of java jre-1.8.0 java-1.8.0 If these requirements were used in the RPM build then I could use Oracle Java to satisfy the RPM requirements. Am I missing something obvious here? Comments? -- Evan Rempel erempel(a)uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria

3 4

syslog-ng is skipping syslog events with no PRI
by Asif Iqbal 26 Mar '18

26 Mar '18

syslog-ng is *NOT* writing syslog like this to a file which has no <*PRI*> 23:49:48.306587 IP 192.168.1.100.39567 > 192.168.100.100.514: [|syslog] E.....@.>..U...g..........T.2018-03-19T23:49:48+0000 alarmLog, applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2, alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568, alarmText="delay:9 msec", siteName=Branch-UC1 ................ syslog-ng is writing syslog like this to a file *OK * 23:50:26.930023 IP 192.168.1.100.55078 > 192.168.100.100.514: SYSLOG mail.info, length: 76 E..h.B@.>......g.....&...Tt.<22>Mar 19 23:50:26 SVL-remotehost-02 root: this is third test alarmLog................ Here is my syslog-ng config source s_udp { udp(ip(0.0.0.0) port(514)); }; destination d_alarm { file("/var/log/alarms.log"); }; filter f_alarm { match("alarmLog" value("MESSAGE")); }; log { source(s_udp); filter(f_alarm); destination(d_alarm); }; I am using syslog-ng version 3.5.6 on centos 7 Any idea why syslog-ng is writing the first log event into a file? Appreciate any help! -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

4 12