- syslog-ng - lists.balabit.hu

Insider 2020-10: LaaS; PAN-OS; Security Onion; New Relic;
by Peter Czanik (pczanik) 12 Nov '20

12 Nov '20

Dear syslog-ng users, This is the 86th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS How to use syslog-ng with LaaS and why? --------------------------------------- The first Logging as a Service (LaaS) I learned about many years ago was created by Loggly. Of course there are many more LaaS providers now. While most services also provide their own clients for sending log messages, many of them also document sending log messages to them using syslog-ng. >From this blog, you can learn about the advantages of using syslog-ng with LaaS, or with any cloud based SIEM solution or a Managed Security Service Provider (MSSP), and also how easy it is to create a configuration block for a new provider. While creating a configuration block takes some time, the simplification it provides can make your life easier on the long term. https://www.syslog-ng.com/community/b/blog/posts/how-to-use-syslog-ng-with-… Parsing PAN-OS logs using syslog-ng ----------------------------------- Version 3.29 of syslog-ng was released recently including a user-contributed feature: the panos-parser(). It is parsing log messages from PAN-OS (Palo Alto Networks Operating System). Unlike some other networking devices, the message headers of PAN-OS syslog messages are standards-compliant. However, if you want to act on your messages (filtering, alerting), you still need to parse the message part. The panos-parser() helps you create name-value pairs from the message part of the logs. >From this blog you can learn why it is useful to parse PAN-OS log messages and how to use the panos-parser(). https://www.syslog-ng.com/community/b/blog/posts/parsing-pan-os-logs-using-… Syslog-ng and Security Onion ---------------------------- One of the most interesting projects utilizing syslog-ng is Security Onion, a free and open source Linux distribution for threat hunting, enterprise security monitoring, and log management. It is utilizing syslog-ng for log collection and log transfer and uses the Elastic stack to store and search log messages. Even if you do not use its advanced security features, you can still use it for centralized log collection and as a nice web interface for your logs. But it is also worth getting acquainted with its security monitoring features, as it can show you useful insights about your network. Best of all, Security Onion is completely free and open source, with commercial support available for it. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-security-oni… How to Use Syslog-ng with New Relic ----------------------------------- Learn how to use the http() destination to send logs to New Relic: https://blog.newrelic.com/product-news/how-to-use-syslog-ng-with-new-relic/ WEBINARS * Improve SIEM performance and cut SIEM costs with log management solutions: https://www.syslog-ng.com/event/improve-siem-performance-and-cut-siem-costs… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Using usertty
by Alexandre Santos 04 Nov '20

04 Nov '20

Hi, I am trying to use usertty(*) to send log all messages with severity equal or higher than critical to every user logged. But I am not getting any messages in serial port or ssh. I am sending the configurations and the debug log in attachment. Can you help me to understand what is happening? Thanks in advance, Alex

2 7

Requesting help with Grouping-by function
by Maciek Solnicki 04 Nov '20

04 Nov '20

Hi All, I would like to ask for help with usage of grouping-by function. Goal is to correlate logs from auditd. Logs produced by auditd are multiline, but events share some number in key with name *msg. *This in theory should allow aggregation of logs with the same *msg *key, but I cannot get it right. Here is my config: *log {* * source {* * file('/var/log/audit/audit.log' flags(no-parse));* * };* * parser {* * linux-audit-parser(prefix(".auditd."));* * };* * parser {* * grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );* * };* * destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};* After implementing above config, output logs look like this, prefix is added as expected and format is set o JSON, but no aggregation happens: *{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}* *{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}* Syslog-ng version I'm using: *syslog-ng 3 (3.29.1)* *Config version: 3.29* *Installer-Version: 3.29.1* *Revision: 3.29.1-1* *Compile-Date: Aug 28 2020 12:13:25* *Module-Directory: /usr/lib/syslog-ng/3.29* *Module-Path: /usr/lib/syslog-ng/3.29* *Include-Path: /usr/share/syslog-ng/include* *Available-Modules: pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis* *Enable-Debug: off* *Enable-GProf: off* *Enable-Memtrace: off* *Enable-IPv6: on* *Enable-Spoof-Source: on* *Enable-TCP-Wrapper: on* *Enable-Linux-Caps: on* *Enable-Systemd: on* Can anyone please tell what am I doing wrong ? Kind regards Maciej

4 8

Performance issue with python code in template
by Diego Billi 04 Nov '20

04 Nov '20

Hi, I have a performance problem. This is a skeleton of my syslog-ng configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> python { import ...mylibs... def t_my_python_function(msg): ... output_data = .... process msg object ... ... return output_data } source s_mysource { udp( .... ); } destination d_mydestination { file( "/tmp/mylogs.log" template("$(python t_my_python_function)") ); }; log { source(s_mysource); destination(d_mydestination); flags(flow-control); }; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I receive syslog messages via UDP with a very very high rate. Incoming messages are processed with a template written in python. Syslog-ng have performance issues with this flow due to the python code. I'm wondering how to speed up this solution. Note that the python function can be parallelized (no shared state/data between messages) I'm trying this solution but i don't know if it changes that much. ------------------------------------------------------------------------ python { import ...mylibs... def p_my_python_function(msg): ... ... output_data = .... process msg object ... ... ... msg['MY_OUTPUT_DATA'] = output_data } source s_mysource { ... } destination d_mydestination { file( "/tmp/mylogs.log" template("${MY_OUTPUT_DATA}") ); }; log { source(s_mysource); parser(p_my_python_function); <---- moved here (outside destination) destination(d_mydestination); flags(flow-control); }; ------------------------------------------------------------------------ Moving the "processing" outside the "destination" is really useful? I'm trying to understand if i can use threading and multi-core supporto of syslog-ng. Thank you for your time. Diego.

3 2

Recommended db for syslog?
by victorsugo 26 Oct '20

26 Oct '20

Hi, I'd like to store my logs in a database, instead of text files. Many years ago I thought a mysql or postgresql can be excellent for that purpose, but today... I'm a little bit confused. Should I use an RDBMS? Or should I turn to noSQL databases? (mongoDB?) The only thing I'm sure, I don't want to use any other log manager(??) software (like Graylog, elasticsearch etc.) I'd like to use only a syslog-ng server with a database. Could you help me? Victor :)

1 0

Insider 2020-10: Cisco; Signal Messenger; PCRE dupnames;
by Peter Czanik (pczanik) 08 Oct '20

08 Oct '20

Dear syslog-ng users, This is the 85th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Parsing Cisco logs in syslog-ng -------------------------------- Log messages generated by Cisco devices look like syslog messages at first glance, but on a closer inspection you will see that there are many smaller differences. By default, syslog-ng treats all incoming messages as syslog messages, however, Cisco logs do not conform. Log messages collected over the network from Cisco devices and saved to a file look broken. There are many Cisco log variants but luckily a good part of them are covered by the cisco-parser() of syslog-ng.From this blog you can learn how the Cisco parser in syslog-ng works and how you can check if it really works with your Cisco log messages. https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-sysl… Sending alerts to Signal Messenger from syslog-ng ------------------------------------------------- Signal Messenger is becoming the instant messaging platform of choice for privacy-minded individuals, including many sysadmins. No wonder that some of them would like to see alerts from syslog-ng in this IM platform. Developing a new destination for syslog-ng from scratch in the C programming language is a considerable effort. As a result, this first implementation is utilizing an already existing command line application. Below, you can learn about an initial implementation, and why it is not part of syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/sending-alerts-to-signal-m… Enabling PCRE dupnames in syslog-ng ----------------------------------- One of the major syslog-ng features is that it can parse log messages and create name-value pairs from them. Until now the PCRE parser could not handle duplicate names for named subpatterns. Version 3.29 of syslog-ng resolves this issue by adding the “dupnames” flag. From this blog you can learn why the dupnames flag is important and how you can enable and test it. https://www.syslog-ng.com/community/b/blog/posts/enabling-pcre-dupnames-in-… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

squid access.log to syslog-ng - how?
by victorsugo 06 Oct '20

06 Oct '20

Hi, I'd like to send all of my squid proxy's log to a syslog-ng server. But it doesn't work as I want, because when syslog-ng parses the lines from access.log, it interprets the first field (actually UTC timestamp) as a program. (I get the timestamp value in the PROGRAM field). But... if I put an rsyslog (installed locally on squid's container) between the squid and syslog-ng servers, I can get correct results. Could you help me, how can I omit the rsyslog from the logging and get a usable log? Is there a standard way to do it? Or do I need to create an own template in squid.conf for this log? regards, Victor --------------------- What's the vector Victor? :) ---------------------------------------------------

2 1

rewrite error
by operacoes - mls wireless 05 Oct '20

05 Oct '20

Hello there, I am runing syslog-ng 3.2.5 on a centos 6.10 server, the log server. Here we have 30 clients sending log via rsyslog to the log server. At the log server each one of them have its own log according to the host name. This operation works just fine. Now my manager ask me to remove some parts of the log, so the final log size will be smaller. At the log server I tried use rewrite and unset to suppress some parts of the log but it does not work, with the error : Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='rewrite' This is the config on log server for each one of the clients. filter f_client_serv01 { host( "client_serv01" ); }; destination d_client_serv01 { file("/var/log/client_serv01.log" perm(0644) owner(root) group(root) ); }; log { source(s_net); filter(f_client_serv01); destination(d_client_serv01); }; This is the log Oct 2 19:07:04 client_serv01 kernel: IN=rdc1 OUT= MAC=00:50:da:6f:0a:54:ec:08:6b:7f:d5:cb:08:00 SRC=10.1.16.35 DST=xxx.xxx.xxx.xxx LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=27903 DF PROTO=TCP SPT=40917 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 rewrite r_rewrite_unset{ unset(value("WINDOW")); }; rewrite r_rewrite_subst{ subst("WINDOW", " ", value("MESSAGE"), flags("global")); }; Does anyone have a tip to try to solve this? Thanks for help. Have a nice day! -- Para contato ou solicitações utilize nosso canal de atendimento via Suporte Técnico através de nossa central 24H via tel.:(21) 2538-8400 ou através do email: suporte(a)mls.com.br Atenciosamente, *João Tiago Braz*| Aux. Sistemas Operação 21 2538-8400 | 21 3923-5000 operacoes(a)mls.com.br

2 2

Rewrite JSON field without re-formating via custom template
by Nirgil 29 Sep '20

29 Sep '20

Hello all, sorry to bother you, but I'm bit lost ;] Trying to accomplish the following: Have a JSON inside syslog message and I need to rewrite, or better to remove, one selected particular field, it contains lot of data and I just want to drop this one field before forwarding to different destination. I have created a parser for JSON: parser p_json { channel { parser { json-parser (prefix(".json.")); }; }; }; Now I need remove or change a JSON field, lets say payload. So I created the following rewrite rule and tried multiple options to remove or change the content rewrite { unset(value("${.json.payload}")); unset(value(".json.payload")); groupunset(values("${.json.payload}")); set("TEST", value(".json.payload")); }; This is working on default fields ${HOST}, etc, but not for JSON fields. Message can be reformatted via template ie: template json_template { template("${DATE} ${HOST} $PROGRAM: {\"payload\"\:\""${.json.payload}"\"}") }; In that case, json.payload can be affected by appropriate rewrite rule. But I'd like to affect the json.payload field inside the original ${MSG} without re-formatting whole message via custom template, because the original message has lot of optional fields, other inner JSON's and it is not so simple to made an universal template for this. template json_template { template("${DATE} ${HOST} $PROGRAM: ${MSG}") }; Is there a way how to remove JSON field from original message (${MSG}) without reformatting the whole message via custom template ? Thanks for any thoughts ! Sample of Message: Sep 23 15:22:48 hostname myapplication: {"payload":"verylongpayload","field1":"value",.."lot of other fields, included inner JSON"...}

3 4

Syslog-NG within Kubernetes
by Allen Olivas 22 Sep '20

22 Sep '20

Hello all, I've been working on a containerized and orchestrated environment for a little while now (learning how it all works and slowly implementing it). One step I need to add is Syslog-NG. I"ve copied down the Syslog-NG Docker files and attempted adding these to a deployment in Kubernetes. Syslog-NG appears to be running successfully in Docker but in Kubernetes the logs I get say: syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' I'm not entirely sure where to turn too from this point. I've looked online for any ideas or resolution to this issue but so far non seem to work for my use case. I figured maybe someone on the mailing list might have an Idea? Specifically I'm trying to find out what that error means, and how best to orchestrate a containerized Syslog-NG. Thanks!

2 1