- syslog-ng - lists.balabit.hu

[GSoC] Rotate log files based on size
by Devharsh Trivedi 13 Apr '22

13 Apr '22

Hi @all, I am a PhD student and would like to apply to the Google Summer of Code. The proposal "Rotate log files based on size” seems interesting to me. Please advise. Regards, Devharsh.

3 2

3.36.1 new log format of some internal messages
by Evan Rempel 06 Apr '22

06 Apr '22

syslog-ng version 3.36.1 has introduced an additional time stamp in some of its internally generated log messages. [2022-04-05T07:10:27.574110] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 3.36 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file; config-version='3.34' In the past this log message would have been formatted as WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 3.36 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file; config-version='3.34' Is this intentional? Why is it not consistent with other internal messages such as syslog-ng starting up; version='3.36.1' Another thing of note is that the configuration warning is actually logged via journald while all other internal messages are only available in the syslog-ng log streams sourcing "internal()" -- Evan

2 2

syslog-ng and ELK integration
by Park, Ethan 04 Apr '22

04 Apr '22

Greetings, I don't know this is the right place to ask about this question but I will give it a shot. We are running a syslog-ng server and we want to use Kibana as a visualization tool. I got stuck in at this point when configuring conf.d file. source s_tcp { tcp(ip("0.0.0.0") port("514")); udp(ip("0.0.0.0") port("514")); }; destination d_el { elasticsearch2( client-mode("http") index("syslog-ng") type("test") cluster-url(http://10.1.3.222:9200) ); #file("/var/log/networks/$HOST/$YEAR/$MONTH/$HOST-$YEAR-$MONTH-$DAY.log"); }; log { source(s_tcp); destination(d_el); flags(flow-control); }; If I start syslog-ng daemon, I got the below error. Apr 04 11:54:42 syslog-ng syslog-ng[22294]: [2022-04-04T11:54:42.443764] Plugin module not found in 'module-path'; module-path='/usr/lib/syslog-ng/3.13', module='mod-java' Apr 04 11:54:42 syslog-ng syslog-ng[22294]: Error parsing destination, destination plugin elasticsearch2 not found in /etc/syslog-ng/conf.d/firewals.conf at line 10, column Apr 04 11:54:42 syslog-ng syslog-ng[22294]: included from /etc/syslog-ng/syslog-ng.conf line 163, column 1 Apr 04 11:54:42 syslog-ng syslog-ng[22294]: elasticsearch2( Apr 04 11:54:42 syslog-ng syslog-ng[22294]: ^^^^^^^^^^^^^^ Apr 04 11:54:42 syslog-ng syslog-ng[22294]: syslog-ng documentation: https://www.balabit.com/support/documentation?product=syslog-ng-ose Apr 04 11:54:42 syslog-ng syslog-ng[22294]: contact: https://lists.balabit.hu/mailman/listinfo/syslog-ng Apr 04 11:54:42 syslog-ng systemd[1]: syslog-ng.service: Main process exited, code=exited, status=1/FAILURE Apr 04 11:54:42 syslog-ng systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Apr 04 11:54:42 syslog-ng systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed I am running 3.13 (syslog-ng) and what did I miss? PRIVILEGED AND CONFIDENTIAL COMMUNICATION This message is privileged and confidential under state and federal law. It is intended for the exclusive use of the named recipient. If the reader of this message is not the named or intended recipient, or the employee or agent responsible for delivering the message to its intended recipient, you are hereby notified that any use, copying, disclosure or dissemination of this message is strictly prohibited. If you are not the named or intended recipient and have received this communication in error, please notify us immediately by replying to the sender of this e-mail or at the address and phone number listed below, and take all steps necessary to delete this communication. Exactech, Inc. 2320 NW 66th Ct. Gainesville, FL. 32653 352-377-1140 An Equal Opportunity Employer

2 1

Multiple options blocks
by Faine, Mark R. (MSFC-IS40)[NICS] 02 Apr '22

02 Apr '22

Can I have an options block in syslog-ng.conf that is for default configuration such as for /var/log and another options block in an include file that is specific to those destinations?

2 1

Permissions are not set correctly on created directories
by Faine, Mark R. (MSFC-IS40)[NICS] 28 Mar '22

28 Mar '22

I have the following in my syslog-ng.conf for options: options { stats_freq(60); flush_lines(500); #flush_timeout(10000); time_reopen(10); time_reap(5); log_fifo_size(536870912); chain_hostnames(off); log-msg-size(65536); use_dns(no); keep_hostname(yes); use_fqdn(yes); dns_cache(off); dns-cache-size(2000); dns-cache-expire(87600); create_dirs(yes); perm(0640); owner(root); group(splunk); dir-perm(2750); dir-owner(root); dir-group(splunk); threaded(yes); }; However, instead of 2750 for the directory permissions, I'm seeing d-wSrwxrwT. 10 root splunk 4096 Mar 28 14:07 ise [root@molecule8-mfaine data]# stat -c '%a' ise 5276 I would expect: drwxr-s---. 2 root splunk 4096 Mar 28 14:11 ise Where are these permissions coming from? Thanks, Mark

2 2

Value is dropped or unset in resolved destination template
by Faine, Mark R. (MSFC-IS40)[NICS] 25 Mar '22

25 Mar '22

I have a destination template in my destination (d_default) that uses $location destination d_default { file("`BASEPATH`/$location/$(lowercase ${HOST})/$app/$(lowercase ${HOST})_$app.log" create_dirs(yes) flags("threaded", "no-multi-line")); }; That is set by a series of filters that look like this log { if { filter... rewrite { set("foo" value("location")); }; }; if { filter... rewrite { set("bar" value("location")); }; }; ... }; This appears to be working, I can see it in the logs: Setting value; name='location', value='unknown', msg='0x7f190c001e00' # Unknown is a default value that would be appropriate on this testing server Later additional log paths are applied like so: log { source(default); filter(f_asa); rewrite(r_asa); destination(d_default); flags(final ); }; ... log { source(default); filter(f_fw); rewrite(r_fw); destination(d_default); flags(final ); }; ... However, when I go to the destination on the filesystem the location is empty so the files are being written as though location was this: ("`BASEPATH`//$(lowercase ${HOST})/$app/$(lowercase ${HOST})_$app.log" Any ideas what could be causing this? Thanks, Mark

4 5

Help with embedded log paths
by Faine, Mark R. (MSFC-IS40)[NICS] 25 Mar '22

25 Mar '22

I need this configuration to do as the comments say but I can't figure out how to make it happen. Any pointers would be appreciated. Thanks, Mark log { source(default); # One of these first four should always match, if not $location=unknown log { filter(f_arc); rewrite(r_arc); }; log { filter(f_gsfc); rewrite(r_gsfc); }; log { filter(f_jsc); rewrite(r_jsc); }; log { filter(f_msfc); rewrite(r_msfc); }; ## The first one of these to match writes to disk and stops processing further log paths ## I don't know how this can be done since I can't add flags(final) inside of an embedded log path # Log Path for asa log { filter(f_asa); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase asa/${HOST}/${HOST}-asa.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fmg log { filter(f_fmg); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fmg/${HOST}/${HOST}-fmg.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for faz log { filter(f_faz); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase faz/${HOST}/${HOST}-faz.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fw log { filter(f_fw); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fw/${HOST}/${HOST}-fw.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fw_block log { filter(f_fw_block); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fw_block/${HOST}/${HOST}-fw_block.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fortigate log { filter(f_fortigate); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fortigate/${HOST}/${HOST}-fortigate.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for ips log { filter(f_ips); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase ips/${HOST}/${HOST}-ips.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for psa log { filter(f_psa); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase psa/${HOST}/${HOST}-psa.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for pubfw log { filter(f_pubfw); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase pubfw/${HOST}/${HOST}-pubfw.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for utm_other log { filter(f_utm_other); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase utm_other/${HOST}/${HOST}-utm_other.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for waf log { filter(f_waf); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase waf/${HOST}/${HOST}-waf.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for webfilter log { filter(f_webfilter); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase webfilter/${HOST}/${HOST}-webfilter.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; flags(final); };

3 4

Customizing syslog-ng snmp() destination option
by Maurya, Shivani 23 Mar '22

23 Mar '22

Hi All, I am using syslog-ng v3.30 and forwarding snmp traps from this server using snmp() destination option - [cid:image001.png@01D839F6.A0D8F220] I wanted to add a field "agent-addr" as part of the trap & not in the variable-binding. This field "agent-addr" will contain the IP address of the device which is sending the syslog message to the syslog server. How can I add a customized filed as part of the trap? Regards, Shivani Maurya

4 9

Gsoc preparation
by Kavinda Shehan Rajapakshe 22 Mar '22

22 Mar '22

Hello, I’m Kavinda Rajapakse, an undergraduate at the University of Moratuwa, Sri Lanka. Currently, I'm studying Computer Science and Engineering. This is my first time contributing to open source. I have some experience working in Rsyslog and Elastic stack. I would like to contribute to the "Support Elastic's beats protocol” project in GSOC 2022. I would be highly obliged if someone could provide me with an introduction as to how to get started. Thank you.

2 1

Insider 2022-03: syslog-ng 4; MQTT source; Zinc; Elastic Cloud; 3.36;
by Peter Czanik (pczanik) 17 Mar '22

17 Mar '22

Dear syslog-ng users, This is the 99th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng future: the path to syslog-ng 4 ----------------------------------------- Balázs Scheidler, founder of the syslog-ng project, describes the path to syslog-ng version 4.0: “It is very important to state that syslog-ng 4 is not the revolutionary, break-everything kind of release that we see too often in the software world. Rather it is an evolutionary change that will be produced similarly to previous releases” Read the rest of the blog at https://syslog-ng-future.blog/syslog-ng-future-the-path-to-syslog-ng-4/ MQTT source ----------- Version 3.35.1 of syslog-ng introduced an MQTT source. Just for some fun in the last syslog-ng blog post of the year, I created an endless loop using syslog-ng and the Mosquitto MQTT broker. Of course, it does not have much practical value other than possibly a bit of stress testing, but hopefully provides a fun introduction to MQTT-related technologies in syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/creating-an-endless-loop-u… Another use for the syslog-ng elasticsearch-http destination: Zinc --------------------------- There is a new drop-in replacement for Elasticsearch, at least if you don’t mind the limitations and the alpha status. However, it definitely lives up to the promise that it provides an Elasticsearch-compatible API for data ingestion. I tested it with the elasticsearch-http() destination of syslog-ng, and it worked perfectly after I modified the URL in the configuration example I found. https://www.syslog-ng.com/community/b/blog/posts/another-use-for-the-syslog… Sending logs to Elastic Cloud using syslog-ng --------------------------------------------- The Elastic Cloud is a service by Elastic providing Elasticsearch and related services in an easy-to-use package. Last year someone reported an issue that it does not work properly with syslog-ng. I did not have time to investigate at that time. Now I started a free trial and soon my log messages from syslog-ng started to appear in Kibana in Elastic Cloud. >From this blog you can learn how to configure syslog-ng for the Elastic Cloud. I go with the most basic settings: exploring Elastic Cloud and syslog-ng Elasticsearch features are both out of scope, as both are very well documented on their respective websites. NEW RELEASES * syslog-ng 3.36: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-36-news-better… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0