3.27.1

Highlights

DESTIP/DESTPORT/PROTO: new macros. (#2899)
set-facility(): add new rewrite operation to change the syslog facility associated with the message. (#3136)
network tls: Added ca-file() option. With this option the user can set a bundled CA-file to verify the peer. (#3145)
Forward integrity and confidentiality of logs (#3121): It is an experimental feature currently, we are still working on the final form with the authors.

Features

DESTIP/DESTPORT/PROTO: new macros

These new macros express the destination ip, destination port and used protocol on a source.

The use-case behind the PR is as follows:
- someone has an appliance which sends out log messages via both UDP and TCP
- the format of the two are different, and he wants to capture either with the simplest possible filter
- netmask() doesn't work because the IP addresses are the same
- host() doesn't work because the hostnames are the same
Example:
```
log {
  source { network(localip(10.12.15.215) port(5555) transport(udp)); };
  destination { file("/dev/stdout" template("destip=$DESTIP destport=$DESTPORT proto=$PROTO\n")); };
};
```
Output:
```
destip=10.12.15.215 destport=5555 proto=17
```
(#2899)

set-facility(): add new rewrite operation to change the syslog facility
associated with the message.

log {
    source { system(); };
    if (program("postfix")) {
      rewrite { set-facility("mail"); };
    };
    destination { file("/var/log/mail.log"); };
    flags(flow-control);
};

(#3136)

network tls: Added ca-file() option. With this option the user can set a bundled CA-file to verify the peer.
(#3145)
http: When a HTTP response is received, emit a signal with the HTTP response code.
(Later it can be extended to read the response and parse it in a slot...).

This PR also extends the Python HTTP header module with the possibility of writing custom HTTP response code handlers. When someone implements an auth header plugin in Python, it could be useful (for example invalidating a cache).

Example config, click to expand!

(#3159)
java/python: add support for the "arrow" syntax.
```
options("key" => "value")
```
(#3161) (#3247)
python: persist support for python

This feature enables users to persist data between reloads or restarts. The intended usage is to support bookmarking and acknowledgement in the future. It is not suitable for local database use cases.
(#3171)
rewrite: Added conditional set-tag() option. With this option the user can put condition statement inside set-tag option.
```
rewrite { set-tag("tag" condition(match("test" value("MSG")))); };
```
(#3190)
scl: add sumologic destinations: sumologic-syslog() and sumologic-http()
(#3194)
iterate: new template function

The iterate template function generates a series from an initial number and a next function.

For example you can generate a sequence of nonnegative numbers with
```
source {
  example-msg-generator(
    num(3)
    template("$(iterate $(+ 1 $_) 0)")
  );
};
```
(#3205)
telegram: new max-size option

Telegram message will be truncated for max-size size. Telegram does not accept message larger than 4096 utf8 characters. The default value is 4096.
(#3206)

example-message-generator : add support for values(name1 => value1, name2 => value2,..) syntax.

Example

@version: 3.27
log {
  source { example-msg-generator(template("message parameter")
                                 num(10)
                                 values("PROGRAM" => "program-name"
                                        "current-second" => "$C_SEC"
                                ));
         };
  destination { file(/dev/stdout template("$(format-json --scope all-nv-pairs)\n")); };
};

(#3237)

example-msg-generator: support freq(0) for fast message generation

log {
   source { example-msg-generator(freq(0) num(100)); };
   destination { file("/dev/stdout"); };
};

(#3245)

Bugfixes

file: changed time-reap() timer's schedule to respect the documentation (expires after last message)
(#3133)
dbld: fix building problems
- fix rpm package build on centos-7
- fix devshell image build
- fix ubuntu-trusty image build
- fix deb package build on ubuntu-trusty
- fix rpm package build on fedora-30
  (#3143)
tls (network): Properly log an error message, when key-file() or cert-file() is missing.
(#3145)
loggen: fix crash with invalid parameterization
(#3146)
format-json: fix printing of embedded zeros

Prior to 2.64.1, g_utf8_get_char_validated() in glib falsely identified embedded zeros as valid utf8 characters. As a result, format json printed the embedded zeroes as \u0000 instead of \x00. This change fixes this problem.
(#3175)
configure: fix --with-net-snmp configure option
(#3180)
python: fix Py_None reference counting in logger methods (trace, debug, info, warning, error)
(#3187)
afmongodb: do not build module when ENABLE_MONGODB=OFF
(#3188)
telegram: automatically truncate messages larger than 4096 utf8 characters to avoid telegram destination to get stuck
(#3206)
compat/glib: fix recursive call issue on CentOS-6/RHEL-6/platforms
(#3212)
timeutils: fix crash in %f conversion when non-numeric character is in usec field (e.g. ".asd123")
(#3270)

Packaging

macOS: add example startup configuration.
(#3172)
rpm: fix --without maxminddb option

If maxminddb development package was installed on the build system: rpmbuild fails if --without maxminddb was used.
(#3208)

Notes to developers

light: Support to relocate reports dir other than current base dir

For example

python -m pytest -lvs functional_tests/source_drivers/file_source/test_acceptance.py --installdir=/install --reports /tmp/

(#3157)

CONTRIBUTING.md: contribution guide updated
(#3174)
libtest: Now we install config_parse_lib.h, fake-time.h, mock-cfg-parser.h and queue_utils_lib.h
which help unit testing outside of core.
(#3179)
tests: Wait until snmptrapd process able to write traps into output file
(#3185)
mongodb: Replaced the deprecated get_server_status() API with command_simple().

This means, that syslog-ng can now be built with -Werror on systems with 1.15 libmongoc.
(#3199)
stats: add external stats counters

There are situations when someone wants to expose internal variables through stats-api.
Without this changeset, we need two distinct variables: one is for the internal state, other is for the
stats registration (internal state cannot depends on the life cycle of the StatsCounterGroup).
(#3225)

Other changes

afsnmp: merge two existing SNMP modules (trapd-parser and destination) into one.

Motivation: keep closely related modules together and decrease the number of small packages.

Packaging related changes:

libsnmptrapd-parser.so has been removed, both the snmp destination and the trapd parser are part of afsnmp.so .
- deb: we already had mod-snmp which shipped snmp-dest. From now, this packages installs the snmptrapd-parser syslog-ng module. The syslog-ng-mod-snmptrapd-parser package has been removed.
- rpm: snmpdest renamed to afsnmp and the snmptrapd-parser is installed by this package from now
  (#3142)
dbld: remove ubuntu-cosmic as it has reached EOL
(#3143)
afsocket-source: present the number of connections in stats

It helps in the debug process if we can see the number of source connections counted by syslog-ng internally.
(#3193)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Kokan, Laszlo Budai, Laszlo Szemere, László Várady, Mehul Prajapati, Roberto Meléndez, Stephan Marwedel, Steven Haigh, Peter Czanik, Thomas De Schampheleire, Vatsal Sisodiya, Vivin Peris