- syslog-ng - lists.balabit.hu

visualizing firewall logs
by Czanik, Péter 15 Jan '19

15 Jan '19

Hi, It was almost two years ago, when I first published how to visualize firewall logs on a map and create a movie out of the results. By that time talking to many syslog-ng users the consensus was, that it has no practical value. It is an eye candy. It can only be used to attract C-level guys attentions towards security. Tow years later I see people asking me about log visualization a lot more often. Did anything change? Is it already more than an eye candy? I posted an Twitter poll about the topic: https://twitter.com/PCzanik/status/1084804811713929216 Let me know your opinion with a vote, and if you have the time also with an explanation here or on Twitter. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng newbie.
by Jason Long 30 Dec '18

30 Dec '18

Hello.I want to send Nginx log via syslog-ng to a server that Elasticsearch and Kibana installed on it. I'm a beginner and searched in Google but can't find a good tutorial. I want to know on Nginx server with syslog-ng installed is below config enough? options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$");};source s_myfile { file("/var/log/nginx/access.log" follow_freq(1) flags(no-parse));};log { source(s_myfile); }; I'm thankful if anyone share advice. Thank you.

2 3

Syslog-ng and NetSkope JSON logs
by Garcia, Julio (InfoSec) 21 Dec '18

21 Dec '18

Hi, I’m trying to get a valid json formatted log file. I’ve tried several options but none of them seem to work or parse out the data correctly. Here’s what the syslog-ng config looks like. @version:3.14 @include "scl.conf" # syslog-ng configuration file. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ ###################################### #Sources ###################################### source s_netskope { tcp(ip(0.0.0.0) port(51410)); }; ###################################### #Destinations ###################################### destination d_netskope { file("/data/log/syslog/netskope/$HOST/$YEAR-$MONTH-$DAY-netskope.log" create_dirs(yes)); }; ###################################### # Filters ###################################### ###################################### # Log ###################################### log { source( s_netskope); destination(d_netskope); }; options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("user01"); group("user01"); dir-owner("user01"); dir-group("user01"); dir-perm(0755); perm(0755); }; Any help is greatly appreciated. Thank you, Julio Garcia Pro, Information Security Engineer CoreLogic Direct (949) 214-1284 Mobile (714) 474-5254 julgarcia(a)corelogic.com<mailto:julgarcia@corelogic.com> corelogic.com<http://www.corelogic.com/> | Blog<http://www.corelogic.com/blog/default.aspx> LinkedIn<http://www.linkedin.com/company/corelogic> | Twitter<http://twitter.com/corelogicinc> | Facebook<http://www.facebook.com/CoreLogic> | Google+<https://plus.google.com/114618839782139347829> Our Vision: Deliver unique property-level insights that power the global real estate economy ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information that is legally privileged. If you are not the intended addressee, or the person responsible for delivering it to the intended addressee, you are hereby notified that reading, disseminating, distributing or copying this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete the original message and any copies immediately thereafter. Thank you. ****************************************************************************************** CLLD

3 4

Adding a background thread
by Naveen Revanna 21 Dec '18

21 Dec '18

Hello syslog-ng developers, I am planning to add a new thread of execution to syslog-ng. I am seeking inputs/expert advice for the same. Motivation: I had previously discussed a problem here where syslog-ng would hang if dns server is unreachable. The reason this was happening was that getaddrinfo() would take some time to timeout and all syslog-ng tasks would pile up resulting in delayed execution which would appear that syslog-ng has hung. I took the approach of using getaddrinfo_a() which would do the async resolution of the address. I have recently come to understand that getaddrinfo_a() finally calls getaddrinfo() in a separate pthread that it spawns. This is causing some issues resulting in a crash. The stacktrace is pointing to problem in glib allocators. Here is what I am thinking. I will call iv_thread() to create a new thread as part of the main process. All this thread does is execute a fixed task (which in my case is to just do address resolution). I am looking for inputs from you developers for pointers as to what would be a good place to spawn this thread. This thread would do very minimal stuff and hope not to have much complexity. Are there things that I should look out for if we have another thread of execution. Do any of you think it is bad idea and time consuming process to introduce a new thread to existing syslog-ng. I have some understanding of the code and have read a bit about ivykis and some of it's functions. All I'll be bothering you will be for your expertise and would be doing the development on my side. Sorry if I am bothering you in your time with family. Happy Holidays :) Thanks, Naveen.

1 0

Cyxtera json file and syslog-ng
by Garcia, Julio (InfoSec) 20 Dec '18

20 Dec '18

Hi, I’m currently sending Cyxtera json logs to our syslog-ng server via syslog-ng but for some reason I’m not able to get all of the data to parse out correctly. It’s being parsed out with _json. I was at some point able to get the message in json form but all the data was within a MESSAGE bracket. This is where I’m at now. Looks like it parses correctly but it’s all under _json. Is there a way to remove that and keep everything else? This is what my syslog-ng looks like. @version:3.14 @include "scl.conf" # syslog-ng configuration file. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ ###################################### #Sources ###################################### source s_cyxtera { tcp(port(51404) flags(no-parse)); }; ###################################### #Destinations ###################################### destination d_cyxtera { file("/data/log/syslog/cyxtera/$HOST/$YEAR-$MONTH-$DAY-cyxtera.log" template("$(format-json --scope dot-nv-pairs)\n")); }; ###################################### # Parser ###################################### parser p_json { json-parser (prefix(".json.")); }; ###################################### # Log ###################################### log { source( s_cyxtera); parser(p_json); destination(d_cyxtera); }; options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("usr01"); group("usr01"); dir-owner("user01"); dir-group("user01"); dir-perm(0755); perm(0755); }; This is how the data is showing up in splunk. I had to clear some info but these are the main keys showing up. I have to expand _json: to get everything inside the key. I want it where I don’t have the _json and everything else shows as their individual key. Hope this makes sense. I know I’m using some of the wrong terminology. [cid:image001.png@01D49851.505CD0E0] Thank you, Julio Garcia Pro, Information Security Engineer CoreLogic Direct (949) 214-1284 Mobile (714) 474-5254 julgarcia(a)corelogic.com<mailto:julgarcia@corelogic.com> corelogic.com<http://www.corelogic.com/> | Blog<http://www.corelogic.com/blog/default.aspx> LinkedIn<http://www.linkedin.com/company/corelogic> | Twitter<http://twitter.com/corelogicinc> | Facebook<http://www.facebook.com/CoreLogic> | Google+<https://plus.google.com/114618839782139347829> Our Vision: Deliver unique property-level insights that power the global real estate economy ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information that is legally privileged. If you are not the intended addressee, or the person responsible for delivering it to the intended addressee, you are hereby notified that reading, disseminating, distributing or copying this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete the original message and any copies immediately thereafter. Thank you. ****************************************************************************************** CLLD

2 2

ISODATE Macro not working when message contains non-ascii chars
by Jose Angel Santiago 20 Dec '18

20 Dec '18

Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my special use case. Unfortunately my syslog-ng agent reads some binary files which are piped through the log file I'm processing and sending it to elasticsearch. When this happens, the elastic bulk post fail (So I'am affected by this issue https://github.com/balabit/syslog-ng/issues/1993) I've debugged it and the problem was that message generated when applying the template is empty. This is my network destination: destination d_syslog_tcp { network( "syslog-relay.node" port(7601) template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n") template-escape(no) disk-buffer( mem-buf-size(200M) disk-buf-size(500M) reliable(yes) dir("/var/syslog-ng") ) ); }; I've opened a netcat server as syslog-relay.node and piped a small binary file through the stdout file which I'm using as source, and this what the relay receives: 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 � 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 $ 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4) As you can see, fourth message has no timestamp, hostname or ip address, it only contains "$"char, so when this message is sent to elastic, I've got a mapperException since it tries to validate the received timestamp. I know I can customize elastic so it doesn't fail in case a wrong timestamp is received, but the point is that syslog-ng seems to fail when templating the message, doesn't it? Also it would be great if syslog-ng had any mechanism to filter all those binary messages, I've already tried with filter expression, excluding all non-printable chars, but the issue still happens. Regards. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

2 4

3.19.1: help to spread the word
by Czanik, Péter 14 Dec '18

14 Dec '18

Hi, As you have seen here on the mailing list, syslog-ng 3.19.1 was released yesterday. Help us the spread the word about this new release where you can. Here are some pointers for you: - list of changes and source tgz: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1 - unofficial packages for: * Debian / Ubuntu: https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… * Fedora / RHEL / CentOS: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng319/ * openSUSE / SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng319 Tweets, LinkedIn posts, etc. based on your own experiences are of course the best, but here is also a list of posts you can easily share with a click of a button: - announcement: * https://twitter.com/PCzanik/status/1073233850581614592 * https://www.linkedin.com/feed/update/urn:li:activity:6479000458223587328 - packages: * deb: https://twitter.com/Budai_L/status/1073237929445781504 * suse: https://twitter.com/PCzanik/status/1073483538341740544 * fedora/red hat: https://twitter.com/PCzanik/status/1073506635195154432 Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 1

unofficial 3.19.1 rpm packages
by Czanik, Péter 14 Dec '18

14 Dec '18

Hi, Unofficial syslog-ng 3.19.1 rpm packages are now available: - openSUSE / SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng319 - Fedora / RHEL / CentOS: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng319/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Insider 2018-12: 3.19 release; optimizing Splunk; Python source; HTTP batch;
by Czanik, Péter 13 Dec '18

13 Dec '18

Dear syslog-ng users, This is the 71st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.19 of syslog-ng released ---------------------------------- Version 3.19 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to load-balancing to multiple servers. You can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. The new Slack destination allows you to send alerts to a Slack channel. Read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1. Optimize your Splunk infrastructure using new syslog-ng features ------------------------------------------------------- Learn how to use less resources for better performance in Splunk! Many people have been using syslog-ng for decades without knowing that it receives new features as well as bugfixes. While many Linux utilities are practically in maintenance mode, syslog-ng keeps evolving constantly. A strong focus in recent years has been on message parsing and destination drivers. After my talk at Suricon, Splunk users explained how they will change their syslog-ng configurations to optimize their Splunk infrastructure. https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infra… Python source in syslog-ng -------------------------- Using syslog-ng 3.18 and newer releases, you can write new source drivers for syslog-ng in Python. While performance is not as good as C, you gain flexibility and ease of implementation. There are quite a few log sources without a ready to use C API, but with a Python API. Using the Python source of syslog-ng you can leverage these. https://www.syslog-ng.com/community/b/blog/posts/python-source-in-syslog-ng Bulk mode message sending to Elasticsearch with syslog-ng http() destination --------------------------------------------------------- Learn how to send log messages in bulk mode to your Elasticsearch server with syslog-ng. Bulk mode offers better performance, because it sends multiple log messages in a single POST request. https://www.syslog-ng.com/community/b/blog/posts/bulk-mode-message-sending-… WEBINARS You can watch our past webinars: * Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 2

Unofficial syslog-ng-3.19 Debian/Ubuntu packages
by Laszlo Budai 13 Dec '18

13 Dec '18

Hi, syslog-ng 3.19.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [armv7l is still building] * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 * Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0