March 2017 - syslog-ng - lists.balabit.hu

Insider 2017-03: Docker; Central Log Management; Hadoop & Kafka performance; SCALE 2017;
by Czanik, Péter 16 Mar '17

16 Mar '17

Dear syslog-ng users, This is the 56th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Docker: 500K+ pulls ------------------- There are over 500K pulls for the syslog-ng image from the Docker hub at https://hub.docker.com/u/balabit/. You can use one of the -dev images if you want to compile syslog-ng yourself. If you use any of these images, share your use case and experiences with us. Gartner & RSA: Central Log Management ------------------------------------- With the increased reliance on analytics to drive security operations, IT security professionals are paying particular attention to one of the oldest tools in their toolbox, log management. A recently released report by Gartner focuses on the role of Central Log Management (CLM) in improving threat monitoring and detection: https://www.balabit.com/blog/renaissance-central-log-management/ The key takaway from the RSA conference was also CLM: https://www.balabit.com/blog/key-takeaways-about-central-log-management-fro… Upcoming webinar about CLM: https://pages2.balabit.com/central-log-management-webinar/ Hadoop and Kafka performance ---------------------------- The syslog-ng application has been supporting Hadoop and Kafka destinations for some time now, as it is an integral part of a versatile central log management solution. Seamless integration is great, but what about performance? Read our blogs to learn more: Hadoop: https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-n… Kafka: https://www.balabit.com/blog/testing-the-performance-of-log-streaming-to-ka… SCALE 2017 ---------- It is the third year that syslog-ng has participated at Southern California Linux Expo or, as better known to many, SCALE ‒ the largest Linux event in the USA. Balabit had a booth in the expo area and on Sunday I gave a presentation on how to deal with security logs using syslog-ng. Read more at https://www.balabit.com/blog/syslog-ng-scale-2017/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

hashing and verify
by Domenico Vitali 15 Mar '17

15 Mar '17

Dear all, I am writing a syslog-ng configuration able to rewrite (or templating) the MESSAGE macro of a syslog entry in such way that it includes its own hash. Consider for example the line: Mar 15 08:23:15 void systemd[1]: Started System Logger Daemon. i would like to rewrite the message obtaining this output: Mar 15 08:23:15 void systemd[1]: Started System Logger Daemon. hash: 87602bdc780a764ae26f30ddc3f09176 where "8ba7a67ce6ac67b9facb6a14b3095960" is the hash of the line (md5, sha1 or other hash funcition). I evaluate both the use of the template function as well as the use of the rewrite module. Unfortunatelly, once i got the line with the hash i can't understand how to verify if the hash is correct. In contrast, when I evalute the hash using the command line, the hash values mismatch. $ echo "Mar 15 08:23:15 void systemd[1]: Started System Logger Daemon." | md5sum - 8ba7a67ce6ac67b9facb6a14b3095960 - In order to perform my test I'm using this configuration: --------------------------------------------------------------------------- source sourcetest { file("/var/log/md5/input.log" flags(no-parse)); }; rewrite msghashing { set("$(md5 $MESSAGE)", value("MYMESS")); set ("-$MESSAGE- hash: $MYMESS", value("MESSAGE"), on-error("fallback-to-string")); }; destination testdestination { file("/var/log/md5/output.log"); }; log { source( sourcetest ); rewrite( msghashing ); # rewrite( r_rewrite_host ); destination( testdestination ); }; --------------------------------------------------------------------------- i can't find any hint in the documentation files, does anybody knows how can i correctly verify the hash ? Is the syslog-ng software using a salt by default ? thanks a lot, -- Domenico

2 1

syslog-ng in docker
by Czanik, Péter 13 Mar '17

13 Mar '17

Hi, Recently I figured out, that there are 500K+ pulls for syslog-ng from the docker hub: https://hub.docker.com/u/balabit/ On the other hand I don't know a single actual user of it yet. If you are one of them, let me know your use case and experiences! Also, if you are on Twitter, help spreading the word and re-tweet: https://twitter.com/PCzanik/status/841192391512059906 Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Increase file descriptors for syslog-ng.
by vijay amruth 11 Mar '17

11 Mar '17

Hello everyone, Hope you are all doing great. I am running into some issues with syslog-ng file descriptors, I have tried to set up ulimit -n value to 65535, *open files (-n) 65535* set the parameter to the follwoing in /etc/sysctl.conf, *fs.file-max = 512000* and tried raising the value with syslog-ng --fd-limit <number>, I get the following error ####syslog-ng --fd-limit 8192 WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.5 format at your time of convinience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.; WARNING: global: the default value of log_fifo_size() has changed to 10000 in syslog-ng 3.3 to reflect log_iw_size() changes for tcp()/udp() window size changes; Using a global log-fetch-limit() option was removed, please use a per-source log-fetch-limit(); WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='0', new_log_iw_size='100', min_log_fifo_size='1300000' and I am unable to stop the syslog-ng service using systemctl, it immediately re-starts, I am using centOS 7.2, I have noticed few other people on the internet had the same problem, please advice, any help is appreciated. *OS: CentOS 7.2* *Hardware : VM* *Problems : 1) Unable to increase the file descriptors for syslog-ng. 2) Unable to stop the service using systemctl stop command, it automatically restarts right after with a new pid.* Thanks, Vijay Amrut.

4 4

Can't get basic syslog to work for my firewall logs?
by Tim Tyler 10 Mar '17

10 Mar '17

Syslog-ng experts. I am very new to syslog-ng. I installed syslog-ng on a fresh Redhat 7.3 server. It defaults working with internal logging. So I configured my firewall to send syslog with facility set to log_user. I turned on Wireshark on the syslog-ng server and observed the firewall sending traffic to the server on udp 514. But the syslog server never created the directory structure and logs. I disabled the redhat firewall just to eliminate it as a possibility. Still no logging. So I don’t know what I am doing wrong at this point. I don’t know if this is a permission problem or some other configuration issue. I found someone that had posted a very basic syslog-ng configuration for firewalls. So I copied It into a firewall.conf I put in conf.d. Can anyone see what might be wrong with it? #################### options { create_dirs(yes); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); }; ################################################## source s_udp { udp(port(514)); }; #Template for a new firewall in the firewalls.conf file #Entries to be changed: NAMEOFTHEFIREWALL and IPOFTHEFIREWALL ################################################## filter f_NAMEOFTHEFIREWALL { host("192.168.30.1"); }; destination d_NAMEOFTHEFIREWALL { file("/var/log/firewalls/PA/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.PA.log"); }; log { source(s_udp); filter(f_NAMEOFTHEFIREWALL); destination(d_NAMEOFTHEFIREWALL); }; Tim Tyler Network Engineer Beloit College

6 8

syslog-ng lock ups
by Evan Rempel 07 Mar '17

07 Mar '17

I have been having a problem with syslog-ng where is just stops processing all input. There are destinations that receive and count all of the messages that come out of syslog-ng and they stop getting any messages. This is occurring on two syslog servers that have similar configurations. One is a superset of the simpler one. On these hosts we run two syslog-ng instances. One for the regular OS log messages from /dev/lo, /proc/kmsg, localhost:1514 and the second one which only listens on the network port(s) which we call our syslog server. The syslog server has its internal log messages going to localhost:1514 so any internal events such as new connections etc should be logged to the "OS syslog" instance. We had this problem on 3.7.x and after upgrading to 3.9.1 we still encounter this problem. The OS is now the latest Redhat 6 (6.8), but the same problem occurred on any Redhat 6.x system. Our host monitoring shows that the syslog-ng process did NOT increase its memory footprint which indicates that it stopped reading its source. All of the hosts that send logs to syslog server showed increased message queueing which confirms that the source stopped being read. The CPU consumption by the syslog-ng process is zero during this time period. Our network monitoring infrastructure opens a connection to the syslog server every 30 seconds. This gets logged via the localhost:1514 connection. These accepted and closed log lines are missing during our problem window even though the monitoring tool WAS able to connect through the entire problem window. We have configured the syslog stats log line to be sent to an external program which invokes syslog-ng-ctl stats and it processes this data. This process did not get any such stats line, so the exact syslog-ng stats counters are unavailable during this problem window. The syslog-ng-ctl program was NOT hung on the socket. Attempting a graceful shutdown of the syslog-ng process either with syslog-ng-ctl or with a kill -TERM appears to have no affect. I assume this is because syslog-ng is unable to flush its buffers so it does not terminate. Killing syslog-ng and restarting it starts processing again correctly. Has anyone else had this symptom? Has anyone found a solution to this? I realize that this is all anecdotal but was hoping it would trigger someone's memory. Thanks in advance. Evan.

2 1

mysql destinations - more options for table creation
by Nik Ambrosch 07 Mar '17

07 Mar '17

When creating a table in a mysql destination, I have not found a way to specify additional options - I’d like to add row_format in order to enable innodb compression. A regular mysql query would look like this: create table “my_table_yyyymmdd” (columns here) ROW_FORMAT=COMPRESSED; Is it possible to tell syslog-ng to make this happen with syslog-ng or will i need to cron an external job to create the tables daily? Thanks.

3 4

Syslog-ng filter for Snort or Barnyard2, if possible
by johnsc301＠gmail.com 02 Mar '17

02 Mar '17

I am able to send messages from Barnyard2 to syslog-ng to Redis. However, I believe that this option can be tweaked to make things easier: filter(f_syslog3); Is there a filter option for Snort such as: filter(snort) or something like filter(barnyard2)? Sent from Mail for Windows 10

1 0