March 2022 - syslog-ng - lists.balabit.hu

a question about the elasticsearch-http driver
by Risto Vaarandi 12 Mar '22

12 Mar '22

hi all, I have a question about the elasticsearch-http driver. Is it possible to send events to a specific Elasticsearch ingest pipeline with this driver? Also, if sending events to ingest pipeline is possible, can the ingest pipeline name contain macros, so that the pipeline name can be set based on the data extracted from incoming events? Such functionality is implemented in rsyslog, and I was wondering if the same thing can be accomplished with syslog-ng. kind regards, risto

2 3

parsing cisco firepower logs problem with 3.33
by Stoffel, John (TAI) 09 Mar '22

09 Mar '22

Hi, I'm trying to parse some cisco logs from a Cisco firepower firewall, using syslog-ng v3.33 on a CentOS 7 system. After pounding my head against the wall a few times to realize that you can't just re-start syslog-ng and have it re-read a source file from scratch... that instead I need to just push the data using netcat, it's now in a state where I think I can try to debug things. My logs look like this: <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic UDP translation fr om TAI-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/33333 duration 0:00:00 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305012: Teardown dynamic TCP translation fr om FOO-WAN_IN:10.92.60.80/59877 to FOO-OUTSIDE:6.7.8.18/59877 duration 0:01:01 <166>2022-02-16T15:31:53Z na-zy-int-fp1140-p02 : %FTD-6-305011: Built dynamic UDP translation from FOO-INSIDE:1.2.3.110/51288 to FOO-OUTSIDE:6.7.8.18/5632 Looking at this log, vs the examples given in the /usr/share/syslog-ng/include/scl/cisco/plugin.conf file, I think the problem is that my logs shows the: sequence, date: origin, %MSG instead of sequence, origin, date: %MSG and it's not clear to me how I would hack the plugin.conf file to handle this issue. My end goal is to be able to parse the message enough by log level so I can forward only a subset of messages to another remote syslog system. Thanks, John Sr. Storage Architect TOSHIBA AMERICA, INC. 1251 6th, Ave 41st flr, New York, NY 10020 508-736-5499 (mobile) E-Mail: john.stoffel(a)toshiba.com<mailto:john.stoffel@toshiba.com> Website: Service Now Self Service Portal<https://nassc.service-now.com/ess/navpage.do>

2 7

syslog-ng-3.36.1
by Laszlo Varady (lvarady) 04 Mar '22

04 Mar '22

3.36.1 Highlights * system() source: added basic support for reading macOS system logs The current implementation processes the output of the original macOS syslogd: /var/log/system.log. (#3710<https://github.com/syslog-ng/syslog-ng/pull/3710>) * $(values) and $(names): these new template functions can be used to query a list of name-value pairs in the current message. The list of name value pairs queried are specified by a value-pairs expression, just like with $(format-json). Examples: This expression sets the JSON array values to contain the list of SDATA values, while the JSON array names would contain the associated names, in the same order. $(format-json values=list($(values .SDATA.*)) names=list($(names .SDATA.*))) The resulting name-value pairs are always sorted by their key, regardless of the argument order. (#3911<https://github.com/syslog-ng/syslog-ng/pull/3911>) * rename(): added a new rewrite rule, called rename() Example usage: rewrite { rename( "renamed-from" "renamed-to" ); }; (#3841<https://github.com/syslog-ng/syslog-ng/pull/3841>) Features * network() drivers: added TLS keylog support syslog-ng dumps TLS secrets for a given source/destination, which can be used for debugging purposes to decrypt data with, for example, Wireshark. This should be used for debugging purposes only! Example usage: source tls_source{ network( port(1234) transport("tls"), tls( key-file("/path/to/server_key.pem"), cert-file("/path/to/server_cert.pem"), ca-dir("/path/to/ca/") keylog-file("/path/to/keylog_file") ) ); }; (#3792<https://github.com/syslog-ng/syslog-ng/pull/3792>) * tls() block: added option for restricting TLS 1.3 ciphers The network(), syslog(), and the http() modules now support specifying TLS 1.3 cipher suites, Example usage: network( transport("tls") tls( pkcs12-file("test.p12") cipher-suite( tls12-and-older("ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"), tls13("TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384") ) ) ); tls12-and-older() can be used to specify TLS v1.2-and-older ciphers, tls13() can be used for TLS v1.3 ciphers only. Note: The old cipher-suite("list:of:ciphers") option restricts only the TLS v1.2-and-older cipher suite for backward compatibility. (#3907<https://github.com/syslog-ng/syslog-ng/pull/3907>) * file() destination: added a new option: symlink-as() This feature allows one to maintain a persistent symlink to a log file when a template is used (for example: /var/log/cron -> /var/log/cron.${YEAR}${MONTH}). Example usage: destination d_file_cron { file("/var/log/cron.${YEAR}${MONTH}" symlink-as("/var/log/cron")); }; >From a functional perspective, the symlink-as file inherits both create-dirs and file ownership from its file destination (permissions are not applicable to symlinks, at least on linux). The symlink is adjusted at the time a new destination file is opened (in the example above, if ${YEAR} or ${MONTH} changes). Although not specific to time macros, that's where the usefulness is. If the template contains something like ${PROGRAM} or ${HOST}, the configuration wouldn't necessarily be invalid, but you'd get an ever-changing symlink of dubious usefulness. (#3855<https://github.com/syslog-ng/syslog-ng/pull/3855>) * flags(no-rfc3164-fallback): added a new flag to sources that parse incoming syslog data and operate in RFC5424 mode (e.g. syslog-protocol is also set). With the new flag the automatic fallback to RFC3164 format is disabled. In this case if the parsing in RFC5424 fails, the syslog parser would result in an error message. In the case of syslog-parser(drop-invalid(yes)), the message would be dropped. (#3891<https://github.com/syslog-ng/syslog-ng/pull/3891>) * syslog-format: accept ISO timestamps that incorrectly use a space instead of a 'T' to delimit the date from the time portion. For example, a "2021-01-01T12:12:12" timestamp is well formed according to RFC5424 (which uses a subset of ISO8601, see https://datatracker.ietf.org/doc/html/rfc5424#section-6.2.3) Some systems simply use a space instead of a 'T'. The same format is accepted for both RFC3164 (e.g. udp(), tcp() and network() sources) and RFC5424 (e.g. syslog() source). (#3893<https://github.com/syslog-ng/syslog-ng/pull/3893>) * transport(text-with-nuls): added a new transport mechanism for the network() driver that allows NUL characters within the message. Note: syslog-ng does not support embedded NUL characters everywhere, so it is recommended that you also use flags(no-multi-line) that causes NUL characters to be replaced by space. (#3913<https://github.com/syslog-ng/syslog-ng/pull/3913>) Bugfixes * filter: fixed the not operator in filter expressions (regression in v3.35.1) Reusing a filter that contains the not operator more than once, or referencing a complex expression containing not might have caused invalid results in the previous syslog-ng version (v3.35.1). This has been fixed. (#3863<https://github.com/syslog-ng/syslog-ng/pull/3863>) * throttle() filter: support negation (#3863<https://github.com/syslog-ng/syslog-ng/pull/3863>) * disk-buffer(): fixed a crash which could happen in very rare cases, while a corrupted disk-buffer was getting replaced (#3845<https://github.com/syslog-ng/syslog-ng/pull/3845>) * disk-buffer(): fixed a memory leak issue and inconsistent buffer handling in rare cases (#3887<https://github.com/syslog-ng/syslog-ng/pull/3887>) * disk-buffer(): fixed underflowing queued stats counter (#3887<https://github.com/syslog-ng/syslog-ng/pull/3887>) * disk-buffer(): fixed queued stats were not adjusted when a disk-buffer became corrupt (#3851<https://github.com/syslog-ng/syslog-ng/pull/3851>) * disk-buffer(): fixed a disk-buffer corruption issue A completely filled and then emptied disk-buffer may have been recognised as corrupt. (#3874<https://github.com/syslog-ng/syslog-ng/pull/3874>) * amqp(): fixed a minor error reporting problem. (#3869<https://github.com/syslog-ng/syslog-ng/pull/3869>) * amqp(): syslog-ng now drops messages that are too large to send (#3869<https://github.com/syslog-ng/syslog-ng/pull/3869>) * amqp(): fixed a crash, which happened with librabbitmq v0.9.0 or v0.10.0, while using the tls() block. (#3929<https://github.com/syslog-ng/syslog-ng/pull/3929>) * file() source: fixed invalid buffer handling when encoding() is used A bug has been fixed that - under rare circumstances - could cause message duplication or partial message loss when non-fixed length or less known fixed-length encodings are used. (#3892<https://github.com/syslog-ng/syslog-ng/pull/3892>) * syslog-ng: fixed a SIGSEGV triggered by an incorrectly formatted "CONFIG" command, received on the syslog-ng control socket. The only known implementation of the control protocol is syslog-ng-ctl itself, which always sends a correct command, but anyone with access to the UNIX domain socket syslog-ng.ctl (root only by default) can trigger a crash. (#3900<https://github.com/syslog-ng/syslog-ng/pull/3900>) * credit-card-mask(): fixed visa, mastercard and jcb card regex pattern (#3853<https://github.com/syslog-ng/syslog-ng/pull/3853>) * cisco-parser(): allow a leading dot in the timestamp (not synced clocks) (#3843<https://github.com/syslog-ng/syslog-ng/pull/3843>) Notes to developers * plugins: we have made it easier to implement filter plugins An example can be found under modules/rate-limit-filter. (#3866<https://github.com/syslog-ng/syslog-ng/pull/3866>) * dev-utils: various fixes for the plugin skeleton generator script (#3866<https://github.com/syslog-ng/syslog-ng/pull/3866>) Other changes * The syslog-ng Docker image<https://hub.docker.com/r/balabit/syslog-ng/> is now automatically tagged and pushed to Docker Hub after each release (#3870<https://github.com/syslog-ng/syslog-ng/pull/3870>) * throttle() filter: renamed to rate-limit() (#3866<https://github.com/syslog-ng/syslog-ng/pull/3866>) * python: support Python 3.10 (#3865<https://github.com/syslog-ng/syslog-ng/pull/3865>) * java: upgraded from old log4j v1.x line to log4j v2.17.2 (#3861<https://github.com/syslog-ng/syslog-ng/pull/3861>) (#3927<https://github.com/syslog-ng/syslog-ng/pull/3927>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Andrea Biardi, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, László Várady, Mikel Olasagasti Uranga, Norbert Takacs, Parrag Szilárd, Peter Kokai, Szilárd Parrag, Zoltan Pallagi, Stanislav Osipov, Yash Mathne <https://github.com/syslog-ng/syslog-ng#installation-from-binaries>Installation from binaries https://github.com/syslog-ng/syslog-ng#installation-from-binaries

1 0

graylog2 plugin not found
by Daniel Estermann 03 Mar '22

03 Mar '22

Hi Fabien, that's it, thank you very much for the quick response! Kinda clairvoyant btw ;-) Best regards Daniel Am 03.03.22, 13:24 schrieb "syslog-ng im Auftrag von Fabien Wernli" <syslog-ng-bounces(a)lists.balabit.hu im Auftrag von wernli(a)in2p3.fr>: Hi Daniel, Could you show your full config ? I think you might be missing the following directive: @include "scl.conf" [http://www.ti8m.com/dam/jcr:59611d49-12ae-460f-b117-37384c0a05c7] <https://www.ti8m.com/insights/downloads/special-security>

1 0

graylog2 plugin not found
by Daniel Estermann 03 Mar '22

03 Mar '22

Hi everybody, I am running RHEL 8.5 ➜ ~ head -2 /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.5 (Ootpa)" ➜ ~ uname -a Linux zhm-lm00 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64 x86_64 GNU/Linux The installation of syslog-ng was performed via DNF using EPEL repository: ➜ ~ dnf install -y syslog-ng >8.......8< Installing: syslog-ng x86_64 3.23.1-2.el8 epel 842 k Installing dependencies: ivykis x86_64 0.42.4-2.el8 epel 47 k libnet x86_64 1.1.6-15.el8 rhel-8-for-x86_64-appstream-rpms 67 k >8.......8< Now, I was trying to configure transport of syslog to graylog using the following documentation: https://www.syslog-ng.com/community/b/blog/posts/graylog-destination-syslog… However, I've received the following error message, when i tried to start syslog-ng with the new configuration: Error parsing destination statement, destination plugin graylog2 not found in /etc/syslog-ng/syslog-ng.conf:9:5-9:13: 4 }; 5 6 filter f_gitlab_runner {"${.SDATA.journald._SYSTEMD_UNIT}" eq "gitlab-runner.service"}; 7 8 destination d_graylog { 9-----> graylog2( 9-----> ^^^^^^^^ 10 host("[MASKED]") 11 port(12201) 12 transport(tcp) 13 ); 14 }; syslog-ng documentation: https://www.balabit.com/support/documentation?product=syslog-ng-ose contact: https://lists.balabit.hu/mailman/listinfo/syslog-ng Do I have to install that plugin separately? How to do that? Kind Regards Daniel Estermann [http://www.ti8m.com/dam/jcr:59611d49-12ae-460f-b117-37384c0a05c7] <https://www.ti8m.com/insights/downloads/special-security>

2 1