- syslog-ng - lists.balabit.hu

Latest syslog-ng RPM needed for RHEL7
by Harish Shetty 22 Feb '19

22 Feb '19

Hi I am planning to upgrade my syslog servers to RHEL7 , I am able to see syslog-ng-3.19.1.tar.gz. Can i get RPM for the same Regards Harish Shetty

2 1

Regex Irregularities
by N. Max Pierson 13 Feb '19

13 Feb '19

Hi List, I am having some weird issues with rewrite regex that I cannot explain. I am simply trying to filter out the first part of the message which has the date in this format. Feb 8 09:13:32 CST: (there is one space at the end) When I use the following syntax, it doesn't match as expected. ^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s I know this is the correct pattern because it works just fine on www.regexpal.com. I did some further testing and I have narrowed it down to the below ... ^\w+ 8 09:55:54 CST: (this seemed to also remove the space behind the month) ^\w+\s 8 09:59:37 CST: (notice this is the exact same as the above without the beginning space) ^\w+\s\d+ Feb 8 10:07:04 CST: (doesn't match anything as though the space between Feb and 8 isn't there) ^\w+\d+ Feb 8 10:11:54 CST: (again doesn't match anything as though there is a space between Feb and 8) So it seems to be something either with \w word class or the + quantifier and it somehow eats the space behind it possibly?? I am running 3.19.1 on Centos 7. Can anyone test this to confirm it isn't just local to my install for whatever reason? Regards, Max

3 3

MS-SQL Question
by Garridan 07 Feb '19

07 Feb '19

Hello! I'm a new syslog-ng user, so please be gentle with me. :) I'm attempting to log to an MS-SQL database and would like to send to different tables in the same DB based on the source IP - for example device A to its own table, device B to its own table, and so on. I thought I would simply need to create the same destination but define different table names in each, it works, however under load the syslog-ng service starts restarting over and over and eventually MS-SQL errors and alerts that the login packet is structurally invalid. Is it possible to send to different table names in this manner or is there another way to do it? Thanks!

6 8

syslog-ng 3.19 not processing /dev/log on RH#L6
by Evan Rempel 29 Jan '19

29 Jan '19

We just had a network even in our environment that resulted in many hosts being isolated on the network in some very odd ways. The hosts had some connections inbound, but no outbound and the network seemed to come and go very quickly. syslog-ng detected that it lost connection to its down-stream syslog collectors a did the normal connection reaping. We have this set for 5 seconds. At some point the syslog-ng process terminated and the supervisor process launched a new child. This worked well, but in this odd senario this happened quickly and at on point the supervisor process failed to reap one of its defunct children but still launched a new child process. The defunct process seemed to hold onto the /dev/log handle so the new child could not get access to it and the result is that we lost hours of logs that flow through the /dev/log socket. When we discovered the issue, there where two children of the supervisor process. One sas in a "defunct" state while the other was running, but only logging kernel messages (from /dev/kmsg) and from the internal syslog-ng source. I suspect that the child reaping process on SIGCHLD does not use a while loop and misses some children. Evan.

2 3

syslog-ng survey
by Scheidler, Balázs 29 Jan '19

29 Jan '19

Hi, If you have a bit of a time, please help us by filling this survey about syslog-ng: https://www.surveymonkey.com/r/ZHJ9MHP It is really appreciated. Cheers, Bazsi

1 0

Running multiple syslog-ng instances on the same host (RHEL6 CO6 SL6)
by Evan Rempel 20 Jan '19

20 Jan '19

In my environment we want to have isolation between the standard OS based syslog-ng service and the central syslog-ng collector service, which of course is running on a host that already has an OS based syslog-ng process running on it. Running muliple syslog-ng daemons under systemd is not so difficult. Copy the system service file and systemd automatically registers a .d folder for custom configurations, however, there is nothing automatic about the System V init mechanism used in the RHEL6 based systems. Don't worry because they syslog-ng init script from the copr yum repository (https://copr.fedorainfracloud.org/coprs/czanik) starting at 3.19 has already thought of this and made multiple instances as automatic as they can be. Quick steps to create a second syslog-ng instance: 1. Decide on the new name extension (shown here is yourName) 2. Copy the included init script cp /etc/init.d/syslog-ng /etc/init.d/syslog-ng-yourName 3. Change the SYSLOGNG_SUFFIX value in your new init script to the value "-yourName". You need the leading hyphen because your added -yourName to the init file name. 4. Copy and customize the included customization file cp /etc/sysconfig/syslog-ng /etc/sysconfig/syslog-ng-yourName By default all of the configuration and runtime files will default to file names including the -yourName suffix, so very little if any customization should be needed. 5. Create a new syslog-ng configuration file for your new instance vi /etc/syslog-ng/syslog-ng-yourName 6. Enable the auto-start of your new instance chkconfig --add syslog-ng-yourName 7. Start your instance /sbin/service syslog-ng-yourName start Thanks again to all of the people adding value to an already great product!

1 0

Issues with sql driver
by N. Max Pierson 18 Jan '19

18 Jan '19

Hi List, I have 2 questions about the sql driver. First, I am trying to get messages into sql using the sql driver but I get an error when I try and restart syslog-ng when I enable the log statement with the sql destination. The syslog-ng --syntax-only command runs without any issues but systemd throws and error that it cannot restart the service but doesn't give a clear reason. My config is below, doesn't anyone know where in a log I can see why it won't restart?? source s_network { udp(ip(0.0.0.0) port(514)); }; destination d_mysql { sql(type(mysql) host("127.0.0.1") username("syslog-ng") password("password") database("syslog") table("messages_${HOST}") columns("date", "host", "message") values("${R_DATE}", "${HOST}", "${MESSAGE}") indexes("date", "host") ); }; log { source(s_network); destination(d_mysql); }; My second question is can you use a template with the sql destination driver? I need to reformat some Cisco Nexus logs because of how it formats the date (looks to be non RFC compliant) and if so, can someone give me a sample of config with the template in the sql destination driver? I cannot seem to find in the docs if this is even possible much less and example of how to do it. TIA, Max

3 10

Insider 2019-01: central server; survey; SNMP; Slack; Conferences & Webinars
by Czanik, Péter 17 Jan '19

17 Jan '19

Dear syslog-ng users, This is the 72nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Creating a central syslog server -------------------------------- Your home network might already contain some devices or systems like a home server, a WiFi router, a media player, or home automation system. Creating a central syslog server to store system logs or syslogs of these systems is considered a best practice. Learn, how to get started: https://balagetech.hu/en/2019/01/14/creating-a-central-syslog-server/ Looking for your feedback about syslog-ng ----------------------------------------- Help us set the direction for syslog-ng by providing your valuable feedback through this short, 9-question survey. Your input will be of great value in determining which features and capabilities to focus our efforts on, going forward. While this survey can be completed anonymously, we would appreciate if you would share your email address, in case we would need to clarify your response and input. https://www.surveymonkey.com/r/ZHJ9MHP SNMP traps to Splunk -------------------- As we learned at the Splunk .conf18 in October, forwarding SNMP traps to Splunk can be a challenging task. Luckily, using syslog-ng can simplify it for us. All we need to make sure about is that snmptrapd logs traps to a file. The syslog-ng application can read and parse that file and forward the traps to Splunk HEC (HTTP Event Collector). https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-to-forward… Send your log messages to Slack ------------------------------- One of the many ways members of a team collaborate is to use Slack. >From syslog-ng version 3.19, you can send log messages to Slack. You can receive critical log messages in real-time in your Slack client on your mobile or desktop. Learn how to set up alerting to Slack in just a few minutes. https://www.syslog-ng.com/community/b/blog/posts/send-your-log-messages-to-… CONFERENCES * FOSDEM talk: Extending syslog-ng in Python ( https://fosdem.org/2019/schedule/track/python/ ) You can also discuss syslog-ng with me at the following events: * https://devconf.info/cz * https://wiki.centos.org/Events/Dojo/Brussels2019 * https://cfgmgmtcamp.eu/ WEBINARS Coming up: * Send logs using the Advanced Log Transport Protocol (ALTP): https://www.syslog-ng.com/event/send-logs-using-the-advanced-log-transport-… You can also watch our past webinars: * Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Rewrite regex not working entirely
by N. Max Pierson 17 Jan '19

17 Jan '19

Hi List, I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog). Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029] I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+). Here is my statement for the rewrite rule. rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); }; The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5?? Regards, Max

5 6

visualizing firewall logs
by Czanik, Péter 15 Jan '19

15 Jan '19

Hi, It was almost two years ago, when I first published how to visualize firewall logs on a map and create a movie out of the results. By that time talking to many syslog-ng users the consensus was, that it has no practical value. It is an eye candy. It can only be used to attract C-level guys attentions towards security. Tow years later I see people asking me about log visualization a lot more often. Did anything change? Is it already more than an eye candy? I posted an Twitter poll about the topic: https://twitter.com/PCzanik/status/1084804811713929216 Let me know your opinion with a vote, and if you have the time also with an explanation here or on Twitter. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng newbie.
by Jason Long 30 Dec '18

30 Dec '18

Hello.I want to send Nginx log via syslog-ng to a server that Elasticsearch and Kibana installed on it. I'm a beginner and searched in Google but can't find a good tutorial. I want to know on Nginx server with syslog-ng installed is below config enough? options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$");};source s_myfile { file("/var/log/nginx/access.log" follow_freq(1) flags(no-parse));};log { source(s_myfile); }; I'm thankful if anyone share advice. Thank you.

2 3

Syslog-ng and NetSkope JSON logs
by Garcia, Julio (InfoSec) 21 Dec '18

21 Dec '18

Hi, I’m trying to get a valid json formatted log file. I’ve tried several options but none of them seem to work or parse out the data correctly. Here’s what the syslog-ng config looks like. @version:3.14 @include "scl.conf" # syslog-ng configuration file. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ ###################################### #Sources ###################################### source s_netskope { tcp(ip(0.0.0.0) port(51410)); }; ###################################### #Destinations ###################################### destination d_netskope { file("/data/log/syslog/netskope/$HOST/$YEAR-$MONTH-$DAY-netskope.log" create_dirs(yes)); }; ###################################### # Filters ###################################### ###################################### # Log ###################################### log { source( s_netskope); destination(d_netskope); }; options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("user01"); group("user01"); dir-owner("user01"); dir-group("user01"); dir-perm(0755); perm(0755); }; Any help is greatly appreciated. Thank you, Julio Garcia Pro, Information Security Engineer CoreLogic Direct (949) 214-1284 Mobile (714) 474-5254 julgarcia(a)corelogic.com<mailto:julgarcia@corelogic.com> corelogic.com<http://www.corelogic.com/> | Blog<http://www.corelogic.com/blog/default.aspx> LinkedIn<http://www.linkedin.com/company/corelogic> | Twitter<http://twitter.com/corelogicinc> | Facebook<http://www.facebook.com/CoreLogic> | Google+<https://plus.google.com/114618839782139347829> Our Vision: Deliver unique property-level insights that power the global real estate economy ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information that is legally privileged. If you are not the intended addressee, or the person responsible for delivering it to the intended addressee, you are hereby notified that reading, disseminating, distributing or copying this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete the original message and any copies immediately thereafter. Thank you. ****************************************************************************************** CLLD

3 4

Adding a background thread
by Naveen Revanna 21 Dec '18

21 Dec '18

Hello syslog-ng developers, I am planning to add a new thread of execution to syslog-ng. I am seeking inputs/expert advice for the same. Motivation: I had previously discussed a problem here where syslog-ng would hang if dns server is unreachable. The reason this was happening was that getaddrinfo() would take some time to timeout and all syslog-ng tasks would pile up resulting in delayed execution which would appear that syslog-ng has hung. I took the approach of using getaddrinfo_a() which would do the async resolution of the address. I have recently come to understand that getaddrinfo_a() finally calls getaddrinfo() in a separate pthread that it spawns. This is causing some issues resulting in a crash. The stacktrace is pointing to problem in glib allocators. Here is what I am thinking. I will call iv_thread() to create a new thread as part of the main process. All this thread does is execute a fixed task (which in my case is to just do address resolution). I am looking for inputs from you developers for pointers as to what would be a good place to spawn this thread. This thread would do very minimal stuff and hope not to have much complexity. Are there things that I should look out for if we have another thread of execution. Do any of you think it is bad idea and time consuming process to introduce a new thread to existing syslog-ng. I have some understanding of the code and have read a bit about ivykis and some of it's functions. All I'll be bothering you will be for your expertise and would be doing the development on my side. Sorry if I am bothering you in your time with family. Happy Holidays :) Thanks, Naveen.

1 0

Cyxtera json file and syslog-ng
by Garcia, Julio (InfoSec) 20 Dec '18

20 Dec '18

Hi, I’m currently sending Cyxtera json logs to our syslog-ng server via syslog-ng but for some reason I’m not able to get all of the data to parse out correctly. It’s being parsed out with _json. I was at some point able to get the message in json form but all the data was within a MESSAGE bracket. This is where I’m at now. Looks like it parses correctly but it’s all under _json. Is there a way to remove that and keep everything else? This is what my syslog-ng looks like. @version:3.14 @include "scl.conf" # syslog-ng configuration file. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ ###################################### #Sources ###################################### source s_cyxtera { tcp(port(51404) flags(no-parse)); }; ###################################### #Destinations ###################################### destination d_cyxtera { file("/data/log/syslog/cyxtera/$HOST/$YEAR-$MONTH-$DAY-cyxtera.log" template("$(format-json --scope dot-nv-pairs)\n")); }; ###################################### # Parser ###################################### parser p_json { json-parser (prefix(".json.")); }; ###################################### # Log ###################################### log { source( s_cyxtera); parser(p_json); destination(d_cyxtera); }; options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("usr01"); group("usr01"); dir-owner("user01"); dir-group("user01"); dir-perm(0755); perm(0755); }; This is how the data is showing up in splunk. I had to clear some info but these are the main keys showing up. I have to expand _json: to get everything inside the key. I want it where I don’t have the _json and everything else shows as their individual key. Hope this makes sense. I know I’m using some of the wrong terminology. [cid:image001.png@01D49851.505CD0E0] Thank you, Julio Garcia Pro, Information Security Engineer CoreLogic Direct (949) 214-1284 Mobile (714) 474-5254 julgarcia(a)corelogic.com<mailto:julgarcia@corelogic.com> corelogic.com<http://www.corelogic.com/> | Blog<http://www.corelogic.com/blog/default.aspx> LinkedIn<http://www.linkedin.com/company/corelogic> | Twitter<http://twitter.com/corelogicinc> | Facebook<http://www.facebook.com/CoreLogic> | Google+<https://plus.google.com/114618839782139347829> Our Vision: Deliver unique property-level insights that power the global real estate economy ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information that is legally privileged. If you are not the intended addressee, or the person responsible for delivering it to the intended addressee, you are hereby notified that reading, disseminating, distributing or copying this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete the original message and any copies immediately thereafter. Thank you. ****************************************************************************************** CLLD

2 2

ISODATE Macro not working when message contains non-ascii chars
by Jose Angel Santiago 20 Dec '18

20 Dec '18

Hi, I'm using syslog-ng 3.18.1, and I'm having some issues due to my special use case. Unfortunately my syslog-ng agent reads some binary files which are piped through the log file I'm processing and sending it to elasticsearch. When this happens, the elastic bulk post fail (So I'am affected by this issue https://github.com/balabit/syslog-ng/issues/1993) I've debugged it and the problem was that message generated when applying the template is empty. This is my network destination: destination d_syslog_tcp { network( "syslog-relay.node" port(7601) template("${ISODATE} ${HOST} ${CONTAINER-IP} ${MESSAGE}\n") template-escape(no) disk-buffer( mem-buf-size(200M) disk-buf-size(500M) reliable(yes) dir("/var/syslog-ng") ) ); }; I've opened a netcat server as syslog-relay.node and piped a small binary file through the stdout file which I'm using as source, and this what the relay receives: 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 � 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 3F 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 $ 2018-12-17T15:35:57+00:00 agent-4 192.168.54.17 4) As you can see, fourth message has no timestamp, hostname or ip address, it only contains "$"char, so when this message is sent to elastic, I've got a mapperException since it tries to validate the received timestamp. I know I can customize elastic so it doesn't fail in case a wrong timestamp is received, but the point is that syslog-ng seems to fail when templating the message, doesn't it? Also it would be great if syslog-ng had any mechanism to filter all those binary messages, I've already tried with filter expression, excluding all non-printable chars, but the issue still happens. Regards. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

2 4

3.19.1: help to spread the word
by Czanik, Péter 14 Dec '18

14 Dec '18

Hi, As you have seen here on the mailing list, syslog-ng 3.19.1 was released yesterday. Help us the spread the word about this new release where you can. Here are some pointers for you: - list of changes and source tgz: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1 - unofficial packages for: * Debian / Ubuntu: https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… * Fedora / RHEL / CentOS: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng319/ * openSUSE / SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng319 Tweets, LinkedIn posts, etc. based on your own experiences are of course the best, but here is also a list of posts you can easily share with a click of a button: - announcement: * https://twitter.com/PCzanik/status/1073233850581614592 * https://www.linkedin.com/feed/update/urn:li:activity:6479000458223587328 - packages: * deb: https://twitter.com/Budai_L/status/1073237929445781504 * suse: https://twitter.com/PCzanik/status/1073483538341740544 * fedora/red hat: https://twitter.com/PCzanik/status/1073506635195154432 Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 1

unofficial 3.19.1 rpm packages
by Czanik, Péter 14 Dec '18

14 Dec '18

Hi, Unofficial syslog-ng 3.19.1 rpm packages are now available: - openSUSE / SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng319 - Fedora / RHEL / CentOS: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng319/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Insider 2018-12: 3.19 release; optimizing Splunk; Python source; HTTP batch;
by Czanik, Péter 13 Dec '18

13 Dec '18

Dear syslog-ng users, This is the 71st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.19 of syslog-ng released ---------------------------------- Version 3.19 of syslog-ng has been released with plenty of new features and bugfixes. Performance of the HTTP destination improved thanks to load-balancing to multiple servers. You can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. The new Slack destination allows you to send alerts to a Slack channel. Read the complete list of changes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1. Optimize your Splunk infrastructure using new syslog-ng features ------------------------------------------------------- Learn how to use less resources for better performance in Splunk! Many people have been using syslog-ng for decades without knowing that it receives new features as well as bugfixes. While many Linux utilities are practically in maintenance mode, syslog-ng keeps evolving constantly. A strong focus in recent years has been on message parsing and destination drivers. After my talk at Suricon, Splunk users explained how they will change their syslog-ng configurations to optimize their Splunk infrastructure. https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infra… Python source in syslog-ng -------------------------- Using syslog-ng 3.18 and newer releases, you can write new source drivers for syslog-ng in Python. While performance is not as good as C, you gain flexibility and ease of implementation. There are quite a few log sources without a ready to use C API, but with a Python API. Using the Python source of syslog-ng you can leverage these. https://www.syslog-ng.com/community/b/blog/posts/python-source-in-syslog-ng Bulk mode message sending to Elasticsearch with syslog-ng http() destination --------------------------------------------------------- Learn how to send log messages in bulk mode to your Elasticsearch server with syslog-ng. Bulk mode offers better performance, because it sends multiple log messages in a single POST request. https://www.syslog-ng.com/community/b/blog/posts/bulk-mode-message-sending-… WEBINARS You can watch our past webinars: * Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 2

Unofficial syslog-ng-3.19 Debian/Ubuntu packages
by Laszlo Budai 13 Dec '18

13 Dec '18

Hi, syslog-ng 3.19.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [armv7l is still building] * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 * Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

OSE 3.19.1
by Nemes, Antal 13 Dec '18

13 Dec '18

3.19.1 Features - HTTP load balancer (#2347 <https://github.com/balabit/syslog-ng/pull/2347>) - Slack destination (#2451 <https://github.com/balabit/syslog-ng/pull/2451>) - Add Cisco Catalyst formatted triplets support to cisco-parser() (#2394 <https://github.com/balabit/syslog-ng/pull/2394>) - Add RFC5424 syslog support to the system() source (FreeBSD 12.0 support) (#2430 <https://github.com/balabit/syslog-ng/pull/2430>) - Add network interface() option to network sources (#2389 <https://github.com/balabit/syslog-ng/pull/2389>) - Add so-reuseport() to network drivers (#2379 <https://github.com/balabit/syslog-ng/pull/2379>) - Enable supporting HTTP redirects (#2136 <https://github.com/balabit/syslog-ng/pull/2136>) Bugfixes - Fix frequent disconnects of syslog() with TLS (#2432 <https://github.com/balabit/syslog-ng/pull/2432>) - Fix possible refcount leak during reload/shutdown (#2434 <https://github.com/balabit/syslog-ng/pull/2434>) - Fix message storm on trace level (#2425 <https://github.com/balabit/syslog-ng/pull/2425>) - Fix use after free in file destinations (time-reap) (#2418 <https://github.com/balabit/syslog-ng/pull/2418>) - Fixing a few memleaks in the Java destination (#2417 <https://github.com/balabit/syslog-ng/pull/2417>) - Fix telegram dst default ca dir (#2416 <https://github.com/balabit/syslog-ng/pull/2416>) - Fix prefix handling in $(list-concat) and $(strip) (#2405 <https://github.com/balabit/syslog-ng/pull/2405>) - Fixing an eventfd leak with ivykis<=0.38 (threaded destinations) (#2404 <https://github.com/balabit/syslog-ng/pull/2404>) - Process flush result after worker thread exits (threaded destinations) (#2402 <https://github.com/balabit/syslog-ng/pull/2402>) - hdfs: do not try to write unopened file (#2391 <https://github.com/balabit/syslog-ng/pull/2391>) - Fix leaks in redis() destination (#2383 <https://github.com/balabit/syslog-ng/pull/2383>) - Block location tracking fixes (#2378 <https://github.com/balabit/syslog-ng/pull/2378>) - Fix $(basename) and $(dirname) in the presence of a prefix (#2371 <https://github.com/balabit/syslog-ng/pull/2371>) - Fixing a false positive corruption detection in non-reliable diskq ( #2356 <https://github.com/balabit/syslog-ng/pull/2356>) - Check if /proc/kmsg can be opened in system-source (#2408 <https://github.com/balabit/syslog-ng/pull/2408>) - Fix include guard in systemd-journal (#2445 <https://github.com/balabit/syslog-ng/pull/2445>) - Remove hexadecimal and octal number parsing from templates (#2401 <https://github.com/balabit/syslog-ng/pull/2401>) Other changes - Do not load certs from default CURL ca-dir by default (http() destination) (#2410 <https://github.com/balabit/syslog-ng/pull/2410>) - Disable SSL compression by default (#2372 <https://github.com/balabit/syslog-ng/pull/2372>) - Flush lines cleanup (#2386 <https://github.com/balabit/syslog-ng/pull/2386>, #2392 <https://github.com/balabit/syslog-ng/pull/2392>) - Refine json-parser() log messages to be less alarming (#2437 <https://github.com/balabit/syslog-ng/pull/2437>) - Move some messages to trace (#2358 <https://github.com/balabit/syslog-ng/pull/2358>) - Make include-path more discoverable (#2426 <https://github.com/balabit/syslog-ng/pull/2426>) - Adding build flag -Wmissing-format-attribute and -Wsuggest-attribute=noreturn (#2423 <https://github.com/balabit/syslog-ng/pull/2423>) - Rewrite filter unit tests based on criterion (#2422 <https://github.com/balabit/syslog-ng/pull/2422>) - PytestFramework in Travis (#2415 <https://github.com/balabit/syslog-ng/pull/2415>) - syslog-ng-mod-java debian pkg should depend on headless jre (#2388 <https://github.com/balabit/syslog-ng/pull/2388>) - Add contextual data error reporting improvements & csv-scanner refactor (#2373 <https://github.com/balabit/syslog-ng/pull/2373>) - Afsocket remove unused functions/bitfields (#2363 <https://github.com/balabit/syslog-ng/pull/2363>) - Afsocket minor cleanup/refactor (#2355 <https://github.com/balabit/syslog-ng/pull/2355>) - Riemann worker (#2313 <https://github.com/balabit/syslog-ng/pull/2313>) - Afsql threaded dest driver (#2097 <https://github.com/balabit/syslog-ng/pull/2097>) - dbld: do not mount .gitconfig if missing (#2419 <https://github.com/balabit/syslog-ng/pull/2419>) - dbld: Add missing docbook-xsl packages (#2398 <https://github.com/balabit/syslog-ng/pull/2398>) - dbld: update criterion to 2.3.3 (#2396 <https://github.com/balabit/syslog-ng/pull/2396>) - dbld: Remove "proposed" Ubuntu repository from enable_dbgsyms() (#2382 <https://github.com/balabit/syslog-ng/pull/2382>) - dbld: Add new target "list-builder-images" (#2381 <https://github.com/balabit/syslog-ng/pull/2381>) - dbld: Support Ubuntu Bionic and update existing images (#2318 <https://github.com/balabit/syslog-ng/pull/2318>) - dbld: release target should use the default image (#2464 <https://github.com/balabit/syslog-ng/pull/2464>) Notes to the developers - PytestFramework: Add initial test framework (#1940 <https://github.com/balabit/syslog-ng/pull/1940>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Abder Benbachir, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Gergely Tonté, JP Vossen, Juhasz Viktor, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1 <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> balabit/syslog-ng <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.19.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. github.com Regards, Antal Nemes

1 0

Re: [syslog-ng] syslog-ng not working with selinux in enforcing mode
by Mohan Lal 11 Dec '18

11 Dec '18

Hi, I am trying to configure syslog-ng to run with selinux in enforcing mode. I have tried to configure ports for syslog-ng in linux using semanage command using following link:- https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-with-selin… under section -> Using a different port a=After that I rebooted the server, I can see syslog-ng is listening in all ports configured in /etc/syslog-ng/conf.d/*.conf ( using netstat -nap|grep ports). I can see all ports in output of 'semanage port --list|grep syslog' also. but it's not processing any data. It start processing data the moment I disable selinux in /etc/selinux/config file. what might be the problem please, how do I go about troubleshooting this. syslog-ng --version syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: Compile-Date: Dec 30 2015 19:57:24 Available-Modules: affile,afprog,afsocket-notls,afsocket-tls,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,linux-kmsg-format,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.4 (Maipo) Regards Mohan This e-mail message (including any attachment) is intended only for the personal use of the recipient(s) named above. This message is confidential and may be legally privileged. If you are not an intended recipient, you may not review, copy or distribute this message. If you have received this communication in error, please notify us immediately by e-mail and delete the original message. Any views or opinions expressed in this message are those of the author only. Furthermore, this message (including any attachment) does not create any legally binding rights or obligations whatsoever, which may only be created by the exchange of hard copy documents signed by a duly authorised representative of Hutchison 3G UK Limited. Hutchison 3G UK Limited is a company registered in England and Wales with company number 3885486. Registered Office Star House, 20 Grenfell Road, Maidenhead, Berkshire SL6 1EH

1 0

balabit/syslog-ng:latest container and kafka output
by Mathias Dufresne 30 Nov '18

30 Nov '18

Hi all, I'm trying to push logs from files to Kafka using syslog-ng container provided there: https://hub.docker.com/r/balabit/syslog-ng/ I used also : https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… The ERROR: When running the container including Kafka configuration I get: [2018-11-30T12:20:51.324773] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; [2018-11-30T12:20:51.478631] Can't find class; class_name='org.syslog_ng.kafka.KafkaDestination' [2018-11-30T12:20:51.478667] If you have reloaded syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again; [2018-11-30T12:20:51.478680] Error initializing message pipeline; plugin_name='java', location='#buffer:4:3' Here are my configuration files: /etc/syslog-ng/syslog-ng.conf : @version: 3.18 @include "scl.conf" # Syslog-ng configuration file, compatible with default Debian syslogd # installation. # First, set some global options. options { chain_hostnames(off); flush_lines(0); #use_dns(yes); #use_fqdn(yes); use_dns(no); use_fqdn(no); #owner("root"); #group("adm"); perm(0640); # The default action of syslog-ng is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(0); stats_level(1); # The default action of syslog-ng is to log a MARK line # to the file every 20 minutes. That's seems high for most # people so turn it down to once an hour. Set it to zero # if you don't want the functionality at all. mark_freq(0); bad_hostname("^gconfd$"); threaded(yes); log_msg_size (16384); log_fifo_size (8000000); # # Time management: # # The goal is to have in logs the original date and time set by # the syslog which effectively receives the message # and that this timestamp is not changed during it's journey # to reach the database. # # To achieve that all clients are configured with keep_timestamp(no) # for syslog-ng marks when it receives the message. # This is done globally because system() and internal() sources are not # both supporting this keep_timestamp option when the syslog() source # used for networked syslog support that option. This options is forced to "yes" # in remote source(). keep_timestamp(no); # # Time precision is increased to microseconds frac_digits(6); # And format is set to "iso": YYYY-MM-DDThh:mm:ss.0123456+02.00 # where "T" is the separator between date and time # and +02.00 GMT decay (as an example) ts_format("iso"); }; ######################## # Sources ######################## # This is the default behavior of sysklogd package # Logs may come from unix stream, but not from another machine. # source s_src { system(); internal(); }; # Log from any hosts around # source s_net { network( transport("tcp") port(601) flags("syslog-protocol", "threaded") keep_timestamp(yes) max-connections(250) log_iw_size(25000) log_fetch_limit(100) flags("threaded") ); }; source s_app_logs { file(/opt/test.log,flags(no-parse),follow_freq(1),keep-timestamp(yes)); }; ################################################################################################# # Destinations ################################################################################################# destination d_messages { file("/var/log/messages"); }; destination d_clients { file("/var/log/syslog_clients/$HOST"); }; destination d_clients_all { file("/var/log/syslog_clients/all_logs"); }; @include "kafka_destination.conf" ################################################################################################# # Destinations ################################################################################################# log { source(s_net); # destination(d_clients); }; log { source(s_src); # destination(d_messages); }; ################################################################################################# # Filters ################################################################################################# filter f_no_debug { not level(debug); }; --------------------------------------------------------------------------------------------- kafka_destination.conf: @module mod-java @define java-module-dir "/usr/lib/syslog-ng/3.18/java-modules:/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/syslog-ng/3.18/java-modules/*.jar" destination d_kafka { kafka( client-lib-dir("/usr/lib/syslog-ng/3.18/java-modules:/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/syslog-ng/3.18/java-modules/*.jar:/etc/syslog-ng/kafka_libs/libs/:/etc/syslog-ng/kafka_libs/libs/*.jar:/etc/syslog-ng/kafka_libs/libs") #client-lib-dir("/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/x86_64-linux-gnu/") kafka-bootstrap-servers("10.11.12.129:9093,10.11.12.130:9093, 10.11.12.131:9093,10.11.12.132:9093,10.11.12.133:9093,10.11.12.134:9093") topic("roma") properties-file("/etc/syslog-ng/kafka_producer.properties") ); }; log { source(s_app_logs); destination(d_kafka); }; --------------------------------------------------------------------------------------------- Directory /etc/syslog-ng/kafka_libs/libs is in fact "libs" directory from kafka_2.11-2.1.0.tgz which comes from http://kafka.apache.org/downloads. Here is the content of this libs directory : ls kafka_libs/libs/ -1 activation-1.1.1.jar aopalliance-repackaged-2.5.0-b42.jar argparse4j-0.7.0.jar audience-annotations-0.5.0.jar commons-lang3-3.5.jar compileScala.mapping compileScala.mapping.asc connect-api-2.1.0.jar connect-basic-auth-extension-2.1.0.jar connect-file-2.1.0.jar connect-json-2.1.0.jar connect-runtime-2.1.0.jar connect-transforms-2.1.0.jar guava-20.0.jar hk2-api-2.5.0-b42.jar hk2-locator-2.5.0-b42.jar hk2-utils-2.5.0-b42.jar jackson-annotations-2.9.7.jar jackson-core-2.9.7.jar jackson-databind-2.9.7.jar jackson-jaxrs-base-2.9.7.jar jackson-jaxrs-json-provider-2.9.7.jar jackson-module-jaxb-annotations-2.9.7.jar javassist-3.22.0-CR2.jar javax.annotation-api-1.2.jar javax.inject-1.jar javax.inject-2.5.0-b42.jar javax.servlet-api-3.1.0.jar javax.ws.rs-api-2.1.1.jar javax.ws.rs-api-2.1.jar jaxb-api-2.3.0.jar jersey-client-2.27.jar jersey-common-2.27.jar jersey-container-servlet-2.27.jar jersey-container-servlet-core-2.27.jar jersey-hk2-2.27.jar jersey-media-jaxb-2.27.jar jersey-server-2.27.jar jetty-client-9.4.12.v20180830.jar jetty-continuation-9.4.12.v20180830.jar jetty-http-9.4.12.v20180830.jar jetty-io-9.4.12.v20180830.jar jetty-security-9.4.12.v20180830.jar jetty-server-9.4.12.v20180830.jar jetty-servlet-9.4.12.v20180830.jar jetty-servlets-9.4.12.v20180830.jar jetty-util-9.4.12.v20180830.jar jopt-simple-5.0.4.jar kafka_2.11-2.1.0.jar kafka_2.11-2.1.0.jar.asc kafka_2.11-2.1.0-javadoc.jar kafka_2.11-2.1.0-javadoc.jar.asc kafka_2.11-2.1.0-scaladoc.jar kafka_2.11-2.1.0-scaladoc.jar.asc kafka_2.11-2.1.0-sources.jar kafka_2.11-2.1.0-sources.jar.asc kafka_2.11-2.1.0-test.jar kafka_2.11-2.1.0-test.jar.asc kafka_2.11-2.1.0-test-sources.jar kafka_2.11-2.1.0-test-sources.jar.asc kafka-clients-2.1.0.jar kafka-log4j-appender-2.1.0.jar kafka-streams-2.1.0.jar kafka-streams-examples-2.1.0.jar kafka-streams-scala_2.11-2.1.0.jar kafka-streams-test-utils-2.1.0.jar kafka-tools-2.1.0.jar log4j-1.2.17.jar lz4-java-1.5.0.jar maven-artifact-3.5.4.jar metrics-core-2.2.0.jar osgi-resource-locator-1.0.1.jar plexus-utils-3.1.0.jar reflections-0.9.11.jar rocksdbjni-5.14.2.jar scala-library-2.11.12.jar scala-logging_2.11-3.9.0.jar scala-reflect-2.11.12.jar slf4j-api-1.7.25.jar slf4j-log4j12-1.7.25.jar snappy-java-1.1.7.2.jar validation-api-1.1.0.Final.jar zkclient-0.10.jar zookeeper-3.4.13.jar zstd-jni-1.3.5-4.jar ---------------------------------------------------------------------------------------------- Here is "syslog-ng --version" inside the conainer: root@e1ca5fdc292a:/# syslog-ng --version syslog-ng 3 (3.18.1) Config version: 3.18 Installer-Version: 3.18.1 Revision: 3.18.1-1 Compile-Date: Oct 24 2018 10:04:41 Module-Directory: /usr/lib/syslog-ng/3.18 Module-Path: /usr/lib/syslog-ng/3.18 Available-Modules: add-contextual-data,afamqp,affile,afmongodb,afprog,afsmtp,afsocket,afsql,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip-plugin,geoip2-plugin,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,mod-java,mod-python,pacctformat,pseudofile,redis,riemann,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on root@e1ca5fdc292a:/# mod-java is there but that's not enough and I can"t understand what I am missing. ------------------------------------------- Packages: root@e1ca5fdc292a:/# dpkg -l | grep kafka ii librdkafka-dev:amd64 0.8.5-2 amd64 library implementing the Apache Kafka protocol (development headers) ii librdkafka1:amd64 0.8.5-2 amd64 library implementing the Apache Kafka protocol ii syslog-ng-mod-kafka 3.18.1-1 amd64 Enhanced system logging daemon (Kafka destination) root@e1ca5fdc292a:/# If information are missing, please tell me. If you have some clue, that would be really appreciated. Best regards, mathias

1 0

Verifying local logs and remote logs are being sent to destination graylog
by Rodney Bizzell 23 Nov '18

23 Nov '18

I want to make sure that my local logs on my syslog box are being sent to my graylog host what config changes do I need to make to ensure that is configured correctly. Can I have an example for illustration. Thanks. @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (250000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; source s_net { udp(ip(0.0.0.0) port(514)); #tcp(ip(0.0.0.0) port(514) max-connections(256)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_graylog { tcp("graylog.host" port (12201) ); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_net); destination(d_graylog); }; log { source(s_sys); filter(f_default); destination(d_graylog);}; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"

4 20

IPV6 and IPV4 syslog-ng issue
by venkateswarlu vinjamuri 22 Nov '18

22 Nov '18

Hi, *Issue:* *when configured both IPV4 and IPV6 syslog server IP's, I could see syslog-ng messages going to only one syslog server, either IPV4 or IPV6 syslog-ng server and that too to the latest server IP configured only.* Is there any known issue ? Please help me on this. Thanks & Regards, Venkat.

2 3

Syslog-ng open source usage.
by vijay amruth 14 Nov '18

14 Nov '18

Hello team, does open source versions use is any different after syslog-ng acquisition? Vijay Amrut. -- Thanks, Vijay Amrut.

3 2