- syslog-ng - lists.balabit.hu

poll: web interface for syslog-ng
by Peter Czanik (pczanik) 27 Nov '20

27 Nov '20

Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 3

Re: [syslog-ng] poll: web interface for syslog-ng
by Clayton Dukes 25 Nov '20

25 Nov '20

Well naturally, my vote is for LogZilla 😉 Peter, I should probably get you an updated description of the product, it has grown a lot! Here are a few useful links: LogZilla in 4 minutes https://www.logzilla.net/logzilla-in-4-minutes.html Automation/Orchestration https://www.logzilla.net/tcat.html ELK Revealed https://www.logzilla.net/the-real-cost-of-elk.html [photo] [signature_543644359] Clayton Dukes CEO, LogZilla Corporation [signature_672330100]<http://www.linkedin.com/in/lzcdukes/> [signature_753363620]<http://twitter.com/logzilla> [signature_14708953] (855) LogZilla<tel:(855)%20LogZilla> [signature_1738705128] 919-600-3198<tel:919-600-3198> [signature_579604246] cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> [signature_1641853981] logzilla.net<http://logzilla.net/> [signature_1306807084] 4819 Emperor Blvd., Suite 400, Raleigh, NC, 27703 [signature_560762648]<https://calendly.com/teamneo/logzilla-neo-discovery> Schedule a 15 minute demo<https://calendly.com/teamneo/logzilla-neo-discovery> [signature_457621985]<https://youtu.be/yGQqkXegdoo> [signature_39202885]LZ in 4 Minutes<https://youtu.be/yGQqkXegdoo> LogZilla Corporation delivers a Centralized Log Management (CLM) platform, designed for IT Operations, Security, and Risk Management leaders who want to gain better incident investigation capabilities by capturing all network and security related logs into a single log collection platform. [Logo Description automatically generated] From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of "Peter Czanik (pczanik)" <Peter.Czanik(a)oneidentity.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Date: Wednesday, November 25, 2020 at 9:46 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] poll: web interface for syslog-ng Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslo…>) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsyslog-ng…> https://twitter.com/PCzanik<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Logzilla Corporation may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.

1 0

syslog-ng 3.30 Debian/Ubuntu packages (OBS)
by Laszlo Budai 20 Nov '20

20 Nov '20

Hi, syslog-ng 3.30.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 * Ubuntu 20.10 (build is still in progress in OBS) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-geat install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.30.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

issues with starting syslog-ng from systemd
by Russell Fulton 20 Nov '20

20 Nov '20

version 3.26.1 — ubuntu 14.04 I can start syslog-ng from the command line without issues (apart form a warning to stderr) but when I try and start it from systemd it starts up (it prints the warning message to the logs) and then exits silently. This version was compiled locally i.e. it was not installed from a package. The .service file was adapted from a sample in the contrib directory. I also had the same behaviour from and other .service file that I put together myself. It is presumably some to do with the environment. I did try starting it with -i (interactive) in the hope that it would spit something out on the console but instead it hung and I had to ^C it Any thoughts on how to figure out what is going on? ===================================== here are two different attempts (first commented) rful011@secmgrprd02:~$ cat /etc/systemd/system/syslog-ng.service #[Unit] #Description=syslog next generation #After=network.target auditd.service #[Service] #EnvironmentFile=/etc/default/syslog-ng #ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf #ExecReload=/usr/local/sbin/syslog-ng-ctl reload #Restart=on-failure #RestartPreventExitStatus=255 #Type=notify #[Install] #WantedBy=multi-user.target #Alias=syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) Conflicts=emergency.service emergency.target Wants=network.target network-online.target After=network.target network-online.target [Service] Type=notify EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf ExecReload=/usr/local/sbin/syslog-ng-ctl reload StandardOutput=journal StandardError=journal Restart=on-failure [Install] WantedBy=multi-user.target

3 2

syslog-ng-3.30.1
by Laszlo Budai 19 Nov '20

19 Nov '20

3.30.1 Highlights * filter template function * support proxy-protocol<https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> Features * kafka (C implementation): * Added template support to topic(). * Added fallback-topic() option, which will be used, if the templated topic() yields an invalid topic name. (#3372<https://github.com/syslog-ng/syslog-ng/pull/3372>) * transport: add proxy-protocol support http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt Example config, click to expand! ([#3437<https://github.com/syslog-ng/syslog-ng/pull/3437>](https://github.com//pull/3437<https://github.com/syslog-ng/syslog-ng/pull/3437>)) * filter: new template function The new introduced filter template function will allow filtering lists based on a filter expression. For example this snippet removes odd numbers log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * file, network, program destinations: : new truncate_size option introduced to truncate an output message to a specified max size. default value is -1 (disabled). network("127.0.0.1" truncate_size(100)); new stats counters: dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_count;1 dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_bytes;1 (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * network: add FreeBSD support for the so_reuseport(yes) the same as in linux SO_REUSEPORT (FreeBSD uses SO_REUSEPORT_LB flag). (#3438<https://github.com/syslog-ng/syslog-ng/pull/3438>) * date-parser: %z accepts local timezone std format as well (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * syslog-format: accepting longer sdata keys Triggered by #3197<https://github.com/syslog-ng/syslog-ng/issues/3197> At the end of the discussion in the mentioned issue, we decided to change the parser and accept longer than 32 character ID's. (#3244<https://github.com/syslog-ng/syslog-ng/pull/3244>) * systemd-journal: add namespace() option This option accepts a string which is identical to the --namespace option of journalctl. For systems defining this option with a systemd version older than v245 a warning is issued. (#3358<https://github.com/syslog-ng/syslog-ng/pull/3358>) Bugfixes * date-parse: %Z should parse the same timezones as %z not just local and gmt (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * python: printing the exception instead of None (if compiled with clang) (#3405<https://github.com/syslog-ng/syslog-ng/pull/3405>) * network/udp: message was lost (not sent) if it was too large, and a time reopen amount of time needed to expire to send the next message lowering the thoughtput. now it is truncated at 65507. (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * tlscontext: support IPv6 X509v3 Subject Alternative Name Fixes #3465<https://github.com/syslog-ng/syslog-ng/issues/3465> (#3466<https://github.com/syslog-ng/syslog-ng/pull/3466>) * map: pass $_ to if correctly. Prior this patchset, if did not receive $_ correctly. After this change, these configurations will work: log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(map $(if ('$(% $_ 2)' eq '0') 'even' 'odd') $INPUT)'\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * systemd-journal: add namespace to the persist name (#3407<https://github.com/syslog-ng/syslog-ng/pull/3407>) * syslog-ng: fixed numerous spelling mistakes in messages generated by syslog-ng (#3398<https://github.com/syslog-ng/syslog-ng/pull/3398>) * network: fix TLS certificate hostname verification when using failover() servers For TLS certificate hostname verification, the certificate's hostname needs to be compared to the configured hostname of the primary and each failover server. syslog-ng used always the primary server's name incorrectly. (#3447<https://github.com/syslog-ng/syslog-ng/pull/3447>) * afsocket: syslog-ng fails to bind() after config revert When having a program source or destination and a network destination in the config, if we reload with an invalid config, syslog-ng crashes, as it cannot init the old network source, because its address is in use. (#3416<https://github.com/syslog-ng/syslog-ng/pull/3416>) * syslog-ng-ctl: when syslog-ng gets stuck on executing a heavy stats-ctl command, should be able to do a graceful shutdown when it is requested. (#3349<https://github.com/syslog-ng/syslog-ng/pull/3349>) * json-parser: fix parsing 64 bit numbers (currently 32 bit was a limit) (#3403<https://github.com/syslog-ng/syslog-ng/pull/3403>) * usertty(): on each tty open error an error mesage and a 10 minutes long disabling of the usertty() destination has been added. Until now, the usertty() destination were only disabled for blocking write() calls. (#3473<https://github.com/syslog-ng/syslog-ng/pull/3473>) Notes to developers * Proxy protocol support added to loggen. Four new options added to loggen to suppport the proxy protocol: * --proxied : Generate PROXY protocol v1 header * --proxy-src-ip : Set the source IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-dst-ip : Set the destination IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-src-port : Set the source port for the PROXY protocol v1 header. If not specified a random port generated in the range 5000-10000. * --proxy-dst-port : Set the destination port for the PROXY protocol v1 header. If not specified the port number 514 will be used. (#3462<https://github.com/syslog-ng/syslog-ng/pull/3462>) * bison: Minimum required version is now 3.4.2. You still only need bison, if you are building from git source or changing the grammar in the released source tarball. (#2526<https://github.com/syslog-ng/syslog-ng/pull/2526>) * Template evaluation related function signatures changed. A new structure LogTemplateEvalOptions is introduced to group parameters together. (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) Other changes * json-parser: change every per message logs that was higher than debug to debug (#3401<https://github.com/syslog-ng/syslog-ng/pull/3401>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Boris Korzun, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Viktor Juhasz, Vivin Peris, Zoltan Pallagi, bjoe2k4

1 0

syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 13 Nov '20

13 Nov '20

Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe

2 1

Insider 2020-10: LaaS; PAN-OS; Security Onion; New Relic;
by Peter Czanik (pczanik) 12 Nov '20

12 Nov '20

Dear syslog-ng users, This is the 86th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS How to use syslog-ng with LaaS and why? --------------------------------------- The first Logging as a Service (LaaS) I learned about many years ago was created by Loggly. Of course there are many more LaaS providers now. While most services also provide their own clients for sending log messages, many of them also document sending log messages to them using syslog-ng. >From this blog, you can learn about the advantages of using syslog-ng with LaaS, or with any cloud based SIEM solution or a Managed Security Service Provider (MSSP), and also how easy it is to create a configuration block for a new provider. While creating a configuration block takes some time, the simplification it provides can make your life easier on the long term. https://www.syslog-ng.com/community/b/blog/posts/how-to-use-syslog-ng-with-… Parsing PAN-OS logs using syslog-ng ----------------------------------- Version 3.29 of syslog-ng was released recently including a user-contributed feature: the panos-parser(). It is parsing log messages from PAN-OS (Palo Alto Networks Operating System). Unlike some other networking devices, the message headers of PAN-OS syslog messages are standards-compliant. However, if you want to act on your messages (filtering, alerting), you still need to parse the message part. The panos-parser() helps you create name-value pairs from the message part of the logs. >From this blog you can learn why it is useful to parse PAN-OS log messages and how to use the panos-parser(). https://www.syslog-ng.com/community/b/blog/posts/parsing-pan-os-logs-using-… Syslog-ng and Security Onion ---------------------------- One of the most interesting projects utilizing syslog-ng is Security Onion, a free and open source Linux distribution for threat hunting, enterprise security monitoring, and log management. It is utilizing syslog-ng for log collection and log transfer and uses the Elastic stack to store and search log messages. Even if you do not use its advanced security features, you can still use it for centralized log collection and as a nice web interface for your logs. But it is also worth getting acquainted with its security monitoring features, as it can show you useful insights about your network. Best of all, Security Onion is completely free and open source, with commercial support available for it. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-security-oni… How to Use Syslog-ng with New Relic ----------------------------------- Learn how to use the http() destination to send logs to New Relic: https://blog.newrelic.com/product-news/how-to-use-syslog-ng-with-new-relic/ WEBINARS * Improve SIEM performance and cut SIEM costs with log management solutions: https://www.syslog-ng.com/event/improve-siem-performance-and-cut-siem-costs… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Using usertty
by Alexandre Santos 04 Nov '20

04 Nov '20

Hi, I am trying to use usertty(*) to send log all messages with severity equal or higher than critical to every user logged. But I am not getting any messages in serial port or ssh. I am sending the configurations and the debug log in attachment. Can you help me to understand what is happening? Thanks in advance, Alex

2 7

Requesting help with Grouping-by function
by Maciek Solnicki 04 Nov '20

04 Nov '20

Hi All, I would like to ask for help with usage of grouping-by function. Goal is to correlate logs from auditd. Logs produced by auditd are multiline, but events share some number in key with name *msg. *This in theory should allow aggregation of logs with the same *msg *key, but I cannot get it right. Here is my config: *log {* * source {* * file('/var/log/audit/audit.log' flags(no-parse));* * };* * parser {* * linux-audit-parser(prefix(".auditd."));* * };* * parser {* * grouping-by(* * key("${.auditd.msg}")* * timeout(10)* * aggregate(value("MESSAGE" "TEST"))* * );* * };* * destination {* * file('/tmp/test.json' template("$(format-json .auditd.*)\n"));* * };* *};* After implementing above config, output logs look like this, prefix is added as expected and format is set o JSON, but no aggregation happens: *{"_auditd":{"type":"EXECVE","msg":"audit(1604391099.830:195558):","argc":"2","a1":"-F","a0":"/usr/sbin/syslog-ng"}}* *{"_auditd":{"type":"CWD","msg":"audit(1604391099.830:195558):","cwd":"/"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/usr/sbin/syslog-ng","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"0","inode":"1078935","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PATH","rdev":"00:00","ouid":"0","ogid":"0","nametype":"NORMAL","name":"/lib64/ld-linux-x86-64.so.2","msg":"audit(1604391099.830:195558):","mode":"0100755","item":"1","inode":"1048592","dev":"08:01","cap_fver":"0","cap_fp":"0000000000000000","cap_fi":"0000000000000000","cap_fe":"0"}}* *{"_auditd":{"type":"PROCTITLE","proctitle":"/usr/sbin/syslog-ng\t-F","msg":"audit(1604391099.830:195558):"}}* Syslog-ng version I'm using: *syslog-ng 3 (3.29.1)* *Config version: 3.29* *Installer-Version: 3.29.1* *Revision: 3.29.1-1* *Compile-Date: Aug 28 2020 12:13:25* *Module-Directory: /usr/lib/syslog-ng/3.29* *Module-Path: /usr/lib/syslog-ng/3.29* *Include-Path: /usr/share/syslog-ng/include* *Available-Modules: pseudofile,cef,confgen,stardate,tfgetent,kvformat,afmongodb,afamqp,afsocket,system-source,csvparser,geoip2-plugin,syslogformat,graphite,affile,xml,dbparser,afprog,add-contextual-data,mod-python,sdjournal,map-value-pairs,timestamp,pacctformat,disk-buffer,afstomp,afsmtp,basicfuncs,afuser,tags-parser,linux-kmsg-format,hook-commands,examples,appmodel,afsql,cryptofuncs,json-plugin,riemann,redis* *Enable-Debug: off* *Enable-GProf: off* *Enable-Memtrace: off* *Enable-IPv6: on* *Enable-Spoof-Source: on* *Enable-TCP-Wrapper: on* *Enable-Linux-Caps: on* *Enable-Systemd: on* Can anyone please tell what am I doing wrong ? Kind regards Maciej

4 8

Performance issue with python code in template
by Diego Billi 04 Nov '20

04 Nov '20

Hi, I have a performance problem. This is a skeleton of my syslog-ng configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> python { import ...mylibs... def t_my_python_function(msg): ... output_data = .... process msg object ... ... return output_data } source s_mysource { udp( .... ); } destination d_mydestination { file( "/tmp/mylogs.log" template("$(python t_my_python_function)") ); }; log { source(s_mysource); destination(d_mydestination); flags(flow-control); }; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I receive syslog messages via UDP with a very very high rate. Incoming messages are processed with a template written in python. Syslog-ng have performance issues with this flow due to the python code. I'm wondering how to speed up this solution. Note that the python function can be parallelized (no shared state/data between messages) I'm trying this solution but i don't know if it changes that much. ------------------------------------------------------------------------ python { import ...mylibs... def p_my_python_function(msg): ... ... output_data = .... process msg object ... ... ... msg['MY_OUTPUT_DATA'] = output_data } source s_mysource { ... } destination d_mydestination { file( "/tmp/mylogs.log" template("${MY_OUTPUT_DATA}") ); }; log { source(s_mysource); parser(p_my_python_function); <---- moved here (outside destination) destination(d_mydestination); flags(flow-control); }; ------------------------------------------------------------------------ Moving the "processing" outside the "destination" is really useful? I'm trying to understand if i can use threading and multi-core supporto of syslog-ng. Thank you for your time. Diego.

3 2