- syslog-ng - lists.balabit.hu

syslog-ng-3.26.1
by Laszlo Budai 05 Mar '20

05 Mar '20

3.26.1 Highlights * file source: Added a new option to multi-line file sources: multi-line-timeout() After waiting multi-line-timeout() seconds without reading new data from the file, the last (potentially partial) message will be flushed and sent through the pipeline as a LogMessage. Since the multi-line file source detects the end of a message after finding the beginning of the subsequent message (indented or no-garbage/suffix mode), this option can be used to flush the last multi-line message in the file after a multi-line-timeout()-second timeout. There is no default value, i.e. this timeout needs to be explicitly configured. Example config: file("/some/folder/events" multi-line-mode("prefix-garbage") multi-line-prefix('^EVENT: ') multi-line-timeout(10) flags("no-parse") ); (#2963<https://github.com/syslog-ng/syslog-ng/pull/2963>) * python-http-header: Added this new plugin, which makes it possible for users to implement HTTP header plugins in Python language. It is built on top of signal-slot mechanism: currently HTTP module defines only one signal, that is signal_http_header_request and python-http-header plugin implements a python binding for this signal. This means that when the signal_http_header_request signal is emitted then the connected slot executes the Python code. The Python interface is: def get_headers(self, body, headers): It should return string List. The headers that will be appended to the request's header. When the plugin fails, http module won't try to send the http request without the header items by default. If you want http module to trying to send the request without these headers, just disable mark-errors-as-critical() function. Original code was written by Ferenc Sipos. Example config, click to expand! (#3123<https://github.com/syslog-ng/syslog-ng/pull/3123>) * azure-auth-header: Added this new plugin, which generates authorization header for applications connecting to Azure. It can be used as a building block in higher level SCLs. Implemented as a signal-slot plugin. Example config, click to expand! (#3078<https://github.com/syslog-ng/syslog-ng/pull/3078>) * python: From now on users can specify a persist name template from python code. @staticmethod def generate_persist_name(options): return options["file_name"] * Usage of this function is necessary, when one python destination is used multiple times in one config. * Persist name from config takes precedence over generate_persist_name. * Persist name is exposed through self.persist_name. (#3016<https://github.com/syslog-ng/syslog-ng/pull/3016>) Features * set-severity(): Added this new rewrite rule for changing message severity. It receives a template and sets message severity by evaluating the template. Numerical and textual severity levels are both supported. Examples: rewrite { set-severity("info"); set-severity("6"); set-severity("${.json.severity}"); }; (#3115<https://github.com/syslog-ng/syslog-ng/pull/3115>) * $(list-search): Added a new template function, which returns the first index of a pattern in a list. Starts the search at start_index. 0 based. If not found, returns empty string. Usage: $(list-search --mode MODE <pattern> ${list}) Where mode can be: literal (default), prefix, substring, glob, pcre. Add --start-index <index> to change the start index. (#2955<https://github.com/syslog-ng/syslog-ng/pull/2955>) * config version: Made the config version check of the configuration more liberal by accepting version numbers that had no changes relative to the current version. This means that if you are running 3.26 and the last semantic change in the configuration was 3.22, then anything between 3.22 and 3.26 (inclusive) is accepted by syslog-ng without a warning at startup. (#3074<https://github.com/syslog-ng/syslog-ng/pull/3074>) * $SEVERITY instead of $LEVEL: syslog-ng now follows the RFC3164 trend of using severity instead of level to refer to the severity of the message that is used in the template language ($SEVERITY), filter expressions (severity()) and so on. (#3128<https://github.com/syslog-ng/syslog-ng/pull/3128>) * http: Added ssl_version("tlsv1_3") and ssl_version("no-tlsv13") options to respectively force and disable TLSv1.3. (#3063<https://github.com/syslog-ng/syslog-ng/pull/3063>) * scl: Improved error message at init, when an scl is missing a dependency. (#3015<https://github.com/syslog-ng/syslog-ng/pull/3015>) * geoip2: Added template() option as an alias for the positional argument string, to match the grammar convention. (#3051<https://github.com/syslog-ng/syslog-ng/pull/3051>) * loggly: Added transport() option, so users can now use it with tls (or any network() supported transport). (#3149<https://github.com/syslog-ng/syslog-ng/pull/3149>) * config-option-database: Added support for parser, diskq and hook-commands blocks. (#3029<https://github.com/syslog-ng/syslog-ng/pull/3029>) Bugfixes * configure.ac: Fixed gethostbyname() function location detection (#3135<https://github.com/syslog-ng/syslog-ng/pull/3135>) * http: Fixed a crash, when workers() was set to 0. We do not allow nonnegative values anymore. (#3116<https://github.com/syslog-ng/syslog-ng/pull/3116>) * snmp-dest: engine-id() option now handles 5 to 32 characters, instead of the strict 10 before. (#3058<https://github.com/syslog-ng/syslog-ng/pull/3058>) * http: Fixed handling of ssl-version() option, which was ignored before. Prior this fix, these values of ssl-version in http destination were ignored by syslog-ng: tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3. (#3083<https://github.com/syslog-ng/syslog-ng/pull/3083>) * network sources: Added workaround for a TLS 1.3 bug to prevent data loss. Due to a bug in the OpenSSL TLS 1.3 implementation (openssl/openssl#10880<https://github.com/openssl/openssl/issues/10880>), it is possible to lose messages when one-way communication protocols are used, - such as the syslog protocol over TLS (RFC 5425<https://tools.ietf.org/html/rfc5425>, RFC 6587<https://tools.ietf.org/html/rfc6587>) - and the connection is closed by the client right after sending data. The bug is in the TLS 1.3 session ticket handling logic of OpenSSL. To prevent such data loss, we've disabled TLS 1.3 session tickets in all syslog-ng network sources. Tickets are used for session resumption, which is currently not supported by syslog-ng. The loggen testing tool also received some bugfixes (#3064<https://github.com/syslog-ng/syslog-ng/pull/3064>), which reduce the likelihood of data loss if the target of loggen has not turned off session tickets. If you're sending logs to third-party OpenSSL-based TLS 1.3 collectors, we recommend turning session tickets off in those applications as well until the OpenSSL bug is fixed. (#3082<https://github.com/syslog-ng/syslog-ng/pull/3082>) * cmake: Now we install loggen headers, as we do with autotools. (#3067<https://github.com/syslog-ng/syslog-ng/pull/3067>) * graylog2, format-gelf: Fixed sending empty message, when ${PID} is not set. Also added a default value "-" to empty short_message and host as they are mandatory fields. (#3112<https://github.com/syslog-ng/syslog-ng/pull/3112>) * loggen: fix dependency error with cmake + openssl from nonstandard location (#3062<https://github.com/syslog-ng/syslog-ng/pull/3062>) * config-option-database: Fixed reading 'grammar' and 'parser' files on 'POSIX' environment (#3125<https://github.com/syslog-ng/syslog-ng/pull/3125>) * file source: Fixed file source not able to process new message after log-msg-size() increase. (#3075<https://github.com/syslog-ng/syslog-ng/pull/3075>) * checkpoint parser: Fixed parsing ISO timestamp. (#3056<https://github.com/syslog-ng/syslog-ng/pull/3056>) * secret-storage: Fixed some cases, where diagnostical logs were truncated. (#3141<https://github.com/syslog-ng/syslog-ng/pull/3141>) * loggen, dqtool: Fixed a crash, when writing error/debug message or relocating qfile. (#3069<https://github.com/syslog-ng/syslog-ng/pull/3069>) * build: Fixed a compatibility related build error on Solaris 11. (#3070<https://github.com/syslog-ng/syslog-ng/pull/3070>) * loggen: Fixed address resolution when only loopback interface was configured. (#3048<https://github.com/syslog-ng/syslog-ng/pull/3048>) Packaging * scl: Moved scl files to the core package. (#2979<https://github.com/syslog-ng/syslog-ng/pull/2979>) * RHEL: Now we include the packaging/rhel/ folder in our release tarball. (#3071<https://github.com/syslog-ng/syslog-ng/pull/3071>) * RHEL 8 / CentOS 8: Added RHEL 8 / CentOS 8 support to syslog-ng.spec (#3034<https://github.com/syslog-ng/syslog-ng/pull/3034>) Notes to developers * signal-slot-connector: Introduced a generic event handler interface for syslog-ng modules. * The concept is simple: * There is a SignalSlotConnector which stores Signal - Slot connections * Signal : Slot = 1 : N, so multiple slots can be assigned to the same Signal. * When a Signal is emitted, the connected Slots are executed. * Signals are string literals and each module can define its own signals. * Interface/protocol between signals and slots: * A macro (SIGNAL) can be used for defining signals as string literals: SIGNAL(module_name, signal, signal_parameter_type) The parameter type is for expressing a kind of contract between signals and slots. * Usage: #define signal_cfg_reloaded SIGNAL(config, reloaded, GlobalConfig) the generated literal: "config::signal_reloaded(GlobalConfig *)" * emit passes the argument to the slots connected to the emitted signal. (#3043<https://github.com/syslog-ng/syslog-ng/pull/3043>) * http: Defined the Signal interface for HTTP - with one signal at this time. What's in the changeset? * List ADT (abstract data type for list implementations) added to lib. * Interface: * append * foreach * is_empty * remove_all * Implemented the List ADT in http module with struct curl_slist for storing the headers. * HTTP signal(s): * Currently only one signal is added, header_request. Note, that the license for http-signals.h is LGPL . (#3044<https://github.com/syslog-ng/syslog-ng/pull/3044>) * example-modules: Added example http slot plugin. This plugin is an example plugin that implements a slot for a HTTP signal (signal_http_header_request). When the plugin is attached, it CONNECT itself to the signal. When the signal is emitted by the http module, the slot is executed and append the header to the http headers. header is set in the config file. Example config, click to expand! (#3093<https://github.com/syslog-ng/syslog-ng/pull/3093>) * NEWS.md: From now on, for every PR that we want to include in the newsfile, we must create the news entry with the PR itself. See news/README.md. (#3066<https://github.com/syslog-ng/syslog-ng/pull/3066>) * snmp test in Light: Added snmp destination tests in the Light test framework. These tests requires snmptrapd as an external dependency. If you don't want to run this test, you can use the pytest's marker discovery feature: python -m pytest ... -m 'not snmp' The tests are run by syslog-ng's Travis workflow. (#3126<https://github.com/syslog-ng/syslog-ng/pull/3126>) * cmake: Added add_module function to cmake to normalize CMakeLists.txt files for modules. (#3106<https://github.com/syslog-ng/syslog-ng/pull/3106>) * dbld: Introduced syslog-ng-kira as a new CI image (#3125<https://github.com/syslog-ng/syslog-ng/pull/3125>) * FunctionalTests: Functional tests are now Python3 compatible (#3144<https://github.com/syslog-ng/syslog-ng/pull/3144>) * dbld: Added Ubuntu 19.10 and 20.04 (#3091<https://github.com/syslog-ng/syslog-ng/pull/3091>) * dbld: Added option to customize shell command. With this change, it is possible to override the option with rules.conf, while keeping the default behaviour. The simplest example: use existing images, start a new one if there is none. (use docker rm manually if you want to update) DOCKER_SHELL=$(DOCKER) inspect $* > /dev/null 2>&1; \ if [ $$? -eq 0 ]; then \ $(DOCKER) start -ia $*; \ else \ $(DOCKER) run $(DOCKER_RUN_ARGS) -ti --name $* balabit/syslog-ng-$* /source/dbld/shell; \ fi (#3038<https://github.com/syslog-ng/syslog-ng/pull/3038>) Other changes * python: Added --with-python3-home configure option to use a hard-coded PYTHONHOME for Python-based plugins. This can be useful when a Python interpreter is bundled with syslog-ng. Relocation is supported, for example: --with-python3-home='${exec_prefix}' (#3134<https://github.com/syslog-ng/syslog-ng/pull/3134>) * afmongodb: Removed the support of deprecated legacy configurations (#3092<https://github.com/syslog-ng/syslog-ng/pull/3092>) * http: use-system-cert-store() now autodetects the system provided cert-store (#3086<https://github.com/syslog-ng/syslog-ng/pull/3086>) * doc: Added manual page for persist-tool. (#3072<https://github.com/syslog-ng/syslog-ng/pull/3072>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Dylan Perry, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Czanik, Péter Kókai, Romain Tartière, Tomáš Mózes.

1 0

Spurious path, logfile not created; path=
by Pal, Laszlo 04 Mar '20

04 Mar '20

Hi, For one of my hosts, I can see lots of these messages *Spurious path, logfile not created; path=* What does it mean exactly? I'm creating files with this macro file("/var/log/netlog/unix/${HOST}/${YEAR}/${MONTH}/${HOST}-${YEAR}${MONTH}${DAY}.log" and even for this host, I have all the logs regardless of this message I also have messages for the same host like this Resource temporarily unavailable (11) Here is some more details may help to find out the reasons behind this - issue started 9th February (I have a total of 160K entries like this) - the filename/path was incorrect during the whole event 2020/02/servername-20200210.log - on 29th the server gone south by consuming lots of CPU and disappeared from the network, console was frozen, so we had to reset the vm The host running an old syslog-ng PE (syslog-ng-premium-edition 4 LTS (4.0.5a) Installer-Version: 4.0.5a Revision: ssh+git://ganesa@git.balabit //var/scm/git/syslog-ng/syslog-ng-pe--mainline--4.0#master#457ec2f494a46d62ecf8cd938f12f02cd0ae9e63) on RHEL5 Log sources are simple plain text files contains tomcat and other web server logs I have a twin-host with the exact same config and log sources, but I never seen messages like this from that one Do you have any idea? To me it looks very mysterious Thanks Laszlo

4 5

syslog driver: IP instead of hostname
by Alexandre Santos 27 Feb '20

27 Feb '20

Hello, I have the following a syslog-ng server and a syslog-ng client, which configurations I am sending in attachment. I am using the syslog driver in order to have full compatibility with RFC5424. I want to use the IP address and not the hostname, but I keep seeing the hostname in tcpdump: [root@tests tests]# tcpdump -A -i virbr0 port 60514 or 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on virbr0, link-type EN10MB (Ethernet), capture size 262144 bytes 18:30:09.810757 IP 192.168.122.11.34512 > tests.syslog: SYSLOG local0.info, length: 100 E.....@.@..K..z...z......l..<134>1 2020-02-26T18:30:09+00:00 localhost root 9519 - - This is a local0 info buffer filler string and in logfile: <134>1 2020-02-26T18:30:09+00:00 localhost root 9519 - - This is a local0 info buffer filler string Can you help me? Thanks in advance, Alex

2 2

system() vs systemd-journal()
by Alexandre Santos 27 Feb '20

27 Feb '20

Hi. I am using syslog-ng 3.19 in a Linux system. My sources configuration is: source s_src { system(); internal(); }; I would like to stack traces to end up in another facility than local0, which is what is happening with the configuration above. This happens because local0 is default facility when SYSLOG_FACILITY entry of journald is not defined. So I changed my sources configuration to: source s_src { internal(); systemd-journal(default-facility(local1)); }; And now the stacktraces are going to local1 facility. *Is there any difference between system() and systemd-journal() when using Linux? I am going to lose any logs?* Thanks in advance, Alex *BTW:* *source s_src { system(default-facility(local1)); internal();};* *Does not gives any error, but it does not have the desired effect.*

3 2

Re: [syslog-ng] network logging
by SZIGETVÁRI János 25 Feb '20

25 Feb '20

Hi Anatoly, I appreciate the trust you bestowed to me by sending me the bundle. >From syslog-ng perspective your setup looks all sane, and I couldn't spot any problems. Upon checking the debug logs, I saw, that syslog-ng did not report either of the incoming UDP packages as "Incoming messages" even though they were recorded in the pcap file. Given that SELinux is disabled, I and your firewall rules are all ALLOW, there are very few things that could prevent the reception of the messages. That led me to suspect a networking problem: a routing issue. You seem to have two interfaces active: A and B. A has a single IP configured, and B has two. Only two locally connected networks are present in the routing table, and a default route, that points to a host located in the subnet that interface A is connected to. The problem is that the packets arrive to interface B, and the IP address they arrive from should be in the direction of interface A, at least from routing perspective. Due to this the kernel will silently discard these packets. In order to fix this I would recommend you to review your routing table, or reconfigure your setup so that these packets arrive to interface A's configured IP address. If you want to verify this theory of mine, I can point you to the following article on how to make the kernel log these unroutable packets: https://www.cyberciti.biz/faq/linux-log-suspicious-martian-packets-un-routa… Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Anatoly Pugachev <matorola(a)gmail.com> ezt írta (időpont: 2020. febr. 25., K, 10:58): > On Wed, Feb 19, 2020 at 6:46 PM SZIGETVÁRI János <jszigetvari(a)gmail.com> > wrote: > > > > Hi Anatoly, > > > > Thank you for the debug log! > > It clearly shows that syslog-ng does not get the remote log at all. > > Could you please run the syslog-ng-debun script with the following > parameters, and send me the resulting tgz file? > > > > # syslog-ng-debun -r -P "port 3514" -D "-Fedv" -t 90 > > > > This will run syslog-ng in debug mode for 90 seconds, and will also > create a network traffic dump of the syslog traffic. Please try sending the > test logs from the other VM while it is running. > > > Janos, > > sorry for the delay, please see attached files. > > Thanks. > > tcpdump command was: > > localhost# tcpdump -n -w udp-3514.pcap -i ens224 udp and port 3514 > > this captured 2 packets coming from 2 different machines, which was run : > > vm1# logger -d -i -n 172.16.3.50 -P 3514 "test from vm1" > vm2# logger -d -i -n 172.16.3.50 -P 3514 "test from vm2" >

1 0

Re: [syslog-ng] Warnings and error while loading default.xml in syslog-ng-3.25.1
by Nitish Saboo 21 Feb '20

21 Feb '20

+Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> On Mon, Feb 17, 2020 at 2:37 PM Nitish Saboo <nitish.saboo55(a)gmail.com> wrote: > Hi Attila, > > 1,2) The issue is that you used "proxysg" in two different rulesets. I > believe syslog-ng could handle this, but we do not handle it right now. > >> But the same default.xml file is working fine in version > syslog-ng-3,6,2 and syslog-ng-3.7.1 where we have same program in different > rulesets.The behavior in version syslog-ng-3,6,2 and syslog-ng-3.7.1 is the > expected behavior or is the current behavior in syslog-ng-3.25.1 version is > having a Bug ? > > 3/4) You can merge the two rulesets, if it is okay for you: > >>Merging the rulesets is not possible at this moment. > Do we have any workaround for this scenario other than merging the > rulesets? > When can we expect a fix for this in near future ? > > Thanks, > Nitish > > On Mon, Feb 17, 2020 at 2:23 PM Attila Szakacs (aszakacs) < > Attila.Szakacs(a)oneidentity.com> wrote: > >> Hi! >> >> 1,2) The issue is that you used "proxysg" in two different rulesets. I >> believe syslog-ng could handle this, but we do not handle it right now. >> 3/4) You can merge the two rulesets, if it is okay for you: >> >> <?xml version='1.0' encoding='UTF-8'?> >> <patterndb version="4"> >> <ruleset id="f582419b3baa42d4a57e42b89704e38c" description=""> >> <pattern>proxysg</pattern> >> <pattern>ProxySG</pattern> >> <rules> >> <rule id="f1e2bfd7bb85402a88d0b732821a0f94"> >> <patterns> >> <pattern>foo</pattern> >> </patterns> >> </rule> >> <rule id="a681963842014480a83a2a2e38875439"> >> <patterns> >> <pattern>anything</pattern> >> </patterns> >> </rule> >> <rule id="bb169f917216467985cc16e28015f5fa"> >> <patterns> >> <pattern>bar</pattern> >> </patterns> >> </rule> >> <rule id="94d4a0c324c8-44a88cf3d4640477d35e"> >> <patterns> >> <pattern>something</pattern> >> </patterns> >> </rule> >> </rules> >> </ruleset> >> </patterndb> >> >> Regards, >> Attila >> >> ------------------------------ >> *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of >> Nitish Saboo <nitish.saboo55(a)gmail.com> >> *Sent:* Monday, February 17, 2020 6:30 AM >> *To:* Syslog-ng users' and developers' mailing list < >> syslog-ng(a)lists.balabit.hu> >> *Subject:* Re: [syslog-ng] Warnings and error while loading default.xml >> in syslog-ng-3.25.1 >> >> CAUTION: This email originated from outside of the organization. Do not >> follow guidance, click links, or open attachments unless you recognize the >> sender and know the content is safe. >> >> Hi Evan, >> >> Apologies for the confusion but I did close the rule tag in my >> default.xml , missed adding it here. >> The error does not seem to be related to improper closing of tags. >> >> This is how it looks : >> >> <ruleset id="f582419b3baa42d4a57e42b89704e38c" description=""> >> <pattern>proxysg</pattern> >> <rules> >> <rule id="f1e2bfd7bb85402a88d0b732821a0f94"> >> <patterns> >> <pattern>foo</pattern> >> </patterns> >> </rule> >> <rule id="a681963842014480a83a2a2e38875439"> >> <patterns> >> <pattern>anything</pattern> >> </patterns> >> </rule> >> </rules> >> </ruleset> >> <ruleset id="17fae6edff32a53f9f294ab21240fc2641e7a4db" description=""> >> <pattern>ProxySG</pattern> >> <pattern>proxysg</pattern> >> <rules> >> <rule id="bb169f917216467985cc16e28015f5fa"> >> <patterns> >> <pattern>bar</pattern> >> </patterns> >> </rule> >> <rule id="94d4a0c324c8-44a88cf3d4640477d35e"> >> <patterns> >> <pattern>something</pattern> >> </patterns> >> </rule> >> </rules> >> </ruleset> >> >> >> I am getting the following error message: >> >> 2020-02-13T10:47:29.631090] Error parsing pattern database file; >> filename='/home/nsaboo/abc/default.xml', >> error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with >> mismatching program name sets, program=proxysg'. >> >> My hunch is the issue is related to merging of two rulesets but I am not >> able to understand why is there mismatching of program names. >> >> 1) Can someone please help me understand the issue here ? >> >> 2) Is the issue seen because a ruleset has multiple programs in it or is >> it because the same program 'proxysg' is being used in different rulesets ? >> >> 3) From the above snippet of default.xml, what changes can I make into >> default.xml to avoid the error ? >> >> 4) Is there a workaround for this issue ? >> >> Thanks, >> Nitish >> >> >> On Sun, Feb 16, 2020 at 12:40 AM Evan Rempel <erempel(a)uvic.ca> wrote: >> >> I'm not exactly sure what is or is not permitted in the pattern database >> but I two comments. >> >> 1. you need end your rule tag and your rules tag before you start a new >> ruleset tag. >> >> 2. What I do in my pattern database is of the form. >> >> <ruleset id="f582419b3baa42d4a57e42b89704e38c" description=""> >> <pattern>proxysg</pattern> >> <rules> >> <rule id="f582419b3baa42d4a57e42b89704e38c"> >> <patterns> >> <pattern>foo</pattern> >> </patterns> >> </rule> >> <rule id="bb169f917216467985cc16e28015f5fa"> >> <patterns> >> <pattern>bar</pattern> >> </patterns> >> </rules> >> </ruleset> >> >> >> Note: >> 1. the closing tag of </rule> before a new starting tag of <rule> >> 2. Multiple "rule" entries inside the the "rules" entry. >> 3. the closing tag of </rule> before the closing tag of </rules> >> 4. the closing tag of </rules> before the closing tag of </ruleset> >> >> I hope that helps. >> >> Evan. >> >> On 2/15/20 12:43 AM, Nitish Saboo wrote: >> >> Hi, >> >> After debugging further into the issue looks like there was a fix for >> patterndb rule clash in syslog-ng-3.8 and this is the commit-id >> '12cd960c8f47260b0b0d4154b096994d66fe345' >> for the fix. And for this reason I am getting the following error for >> same default.xml in syslog-ng-3.25.1 version and not in syslog-ng3.6.2 and >> syslog-ng3.7.1. >> >> 2020-02-13T10:47:29.631090] Error parsing pattern database file; >> filename='/home/nsaboo/abc/default.xml', >> error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with >> mismatching program name sets, program=proxysg'. >> >> Snippet from default.xml >> ========================== >> >> <ruleset id="f582419b3baa42d4a57e42b89704e38c" description=""> >> <pattern>proxysg</pattern> >> <rules> >> <rule id="f582419b3baa42d4a57e42b89704e38c"> >> <patterns> >> <pattern>foo</pattern> >> </patterns> >> >> <ruleset id="8d633c824e844a559088d803464e507a" description=""> >> <pattern>ProxySG</pattern> >> <pattern>proxysg</pattern> >> <rules> >> <rule id="bb169f917216467985cc16e28015f5fa"> >> <patterns> >> <pattern>bar</pattern> >> </patterns> >> >> I am not able to understand the error message clearly. >> >> 1) Can someone please help me understand the issue here ? >> >> 2) Is the issue seen because a ruleset has multiple programs in it or is >> it because the same program 'proxysg' is being used in different rulesets ? >> >> 3) From the above snippet of default.xml, what changes can I make into >> default.xml to avoid the error ? >> >> 4) Is there a workaround for this issue ? >> >> Thanks, >> Nitish >> >> On Fri, Feb 14, 2020 at 2:40 PM Nitish Saboo <nitish.saboo55(a)gmail.com> >> wrote: >> >> Hi Attila, >> >> Thanks for your response. >> >> And what about the following error: >> >> 2020-02-13T10:47:29.631090] Error parsing pattern database file; >> filename='/home/nsaboo/abc/default.xml', >> error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with >> mismatching program name sets, program=proxysg'. >> >> The same default.xml file was getting loaded correctly in syslog-ng-3.6.2 >> and syslog-ng-3.7.1 but getting following error while loading same >> default.xml in syslog-ng3.25.1 >> >> I came across a similar issue on githib ' >> https://github.com/syslog-ng/syslog-ng/issues/2763 >> <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co…>' >> .I see the issue is still in open state.Is there a workaround for >> this issue? >> >> Thanks, >> Nitish >> >> On Fri, Feb 14, 2020 at 1:12 PM Attila Szakacs (aszakacs) < >> Attila.Szakacs(a)oneidentity.com> wrote: >> >> Hi! >> >> WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison >> operators like '!=' in filter expressions were evaluated as string >> operators. This is fixed in syslog-ng 3.8. As we are operating in >> compatibility mode, syslog-ng will exhibit the buggy behaviour as previous >> versions until you bump the @version value in your configuration file; >> >> ^^^ This refers to the syslog-ng.conf file version. >> >> The correct way to resolve it, and fix the buggy behavior of != and ==, >> should be to change the != operators between strings to neq in your filters. >> >> Regards, >> Attila >> ------------------------------ >> *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of >> Nitish Saboo <nitish.saboo55(a)gmail.com> >> *Sent:* Thursday, February 13, 2020 12:17 PM >> *To:* Syslog-ng users' and developers' mailing list < >> syslog-ng(a)lists.balabit.hu> >> *Subject:* [syslog-ng] Warnings and error while loading default.xml in >> syslog-ng-3.25.1 >> >> Hi, >> >> I am using syslog-ng version 3.25.1.Getting following Warnings and error >> while initialising syslog-ng engine: >> >> [2020-02-13T10:47:29.627899] WARNING: due to a bug in versions before >> syslog-ng 3.8numeric comparison operators like '!=' in filter expressions >> were evaluated as string operators. This is fixed in syslog-ng 3.8. As we >> are operating in compatibility mode, syslog-ng will exhibit the buggy >> behaviour as previous versions until you bump the @version value in your >> configuration file; >> [2020-02-13T10:47:29.627968] WARNING: due to a bug in versions before >> syslog-ng 3.8numeric comparison operators like '!=' in filter expressions >> were evaluated as string operators. This is fixed in syslog-ng 3.8. As we >> are operating in compatibility mode, syslog-ng will exhibit the buggy >> behaviour as previous versions until you bump the @version value in your >> configuration file; >> [2020-02-13T10:47:29.628059] WARNING: due to a bug in versions before >> syslog-ng 3.8numeric comparison operators like '!=' in filter expressions >> were evaluated as string operators. This is fixed in syslog-ng 3.8. As we >> are operating in compatibility mode, syslog-ng will exhibit the buggy >> behaviour as previous versions until you bump the @version value in your >> configuration file; >> [2020-02-13T10:47:29.631090] Error parsing pattern database file; >> filename='/opt/tap-parsing/patterns/default.xml', >> error='/opt/tap-parsing/patterns/default.xml:17274:22: Joining rulesets >> with mismatching program name sets, program=proxysg' >> >> >> 1)For the following warnings, to which version I have to bump up the >> configuration file ? >> >> 2020-02-13T10:47:29.627899] WARNING: due to a bug in versions before >> syslog-ng 3.8numeric comparison operators like '!=' in filter expressions >> were evaluated as string operators. This is fixed in syslog-ng 3.8. As we >> are operating in compatibility mode, syslog-ng will exhibit the buggy >> behaviour as previous versions until you bump the @version value in your >> configuration file; >> [2020-02-13T10:47:29.627968] WARNING: due to a bug in versions before >> syslog-ng 3.8numeric comparison operators like '!=' in filter expressions >> were evaluated as string operators. This is fixed in syslog-ng 3.8. As we >> are operating in compatibility mode, syslog-ng will exhibit the buggy >> behaviour as previous versions until you bump the @version value in your >> configuration file; >> [2020-02-13T10:47:29.628059] WARNING: due to a bug in versions before >> syslog-ng 3.8numeric comparison operators like '!=' in filter expressions >> were evaluated as string operators. This is fixed in syslog-ng 3.8. As we >> are operating in compatibility mode, syslog-ng will exhibit the buggy >> behaviour as previous versions until you bump the @version value in your >> configuration file; >> >> Currrently the configuration version is the following: >> >> configuration = cfg_new(0x0302) >> >> Do I have to change it to '0x0319' as defined in 'lib/versioning.h' ? >> >> 2)The same default.xml file was getting loaded correctly in >> syslog-ng-3.6.2 and syslog-ng-3.7.1 but getting following error while >> loading same default.xml in syslog-ng3.25.1 >> >> >> 2020-02-13T10:47:29.631090] Error parsing pattern database file; >> filename='/home/nsaboo/abc/default.xml', >> error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with >> mismatching program name sets, program=proxysg'. >> >> What can be the reason for this error ? >> >> >> Thanks, >> Nitish >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.bal…> >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabi…> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabi…> >> >>

4 7

Java Destination Method Documentation
by Jamal Habash 20 Feb '20

20 Feb '20

Hey All, I am interested in developing a Java destination for syslog-ng. I have read the documentation which states "To interface with syslog-ng, you will need to extend the TextLogDestination or StructuredLogDestination". There is no documentation or code comments for either of these classes. 1) What is the intended use-case for TextLogDestination? 2) What is the intended use-case for StructuredLogDestination? Additionally, as there are no comments in the code, it is unclear what the abstract methods in these classes are meant to do. For example, each class has the same abstract methods called open(), close(), isOpened(), getNameUniqOptions(). 3) What is the purpose of these methods? 4) When will they be invoked by syslog-ng? It is difficult to write a valid implementation without knowing the purpose of each method. Is there any documentation for this? Thanks in advance! J

2 2

network logging
by Anatoly Pugachev 19 Feb '20

19 Feb '20

Hello! Can someone help me with network logging with syslog-ng : # rpm -q syslog-ng syslog-ng-3.25.1.239.g0535e8a-1.el7.x86_64 # rpm -qf /etc/os-release centos-release-7-7.1908.0.el7.centos.x86_64 server is centos 7 x86_64 vm /etc/syslog-ng/syslog-ng.conf is in the default configuration (no changes made from rpm package installed), have additional configuration in : # cat conf.d/mtcaptive.conf source s_mtcaptive { network( ip("0.0.0.0") port(3514) transport("udp")); }; destination d_mtcaptive { file("/var/log/mt-captive-remote"); }; log { source(s_mtcaptive); destination(d_mtcaptive); }; If i try to log a test message with logger on a localhost (where syslog-ng runs): [root@localhost syslog-ng]# logger -d -i -n 172.16.3.50 -P 3514 "test from localhost" it does work and appears in /var/log/mt-captive-remote : $ tail -1 /var/log/mt-captive-remote Feb 19 06:15:01 172.16.3.50 root[11326]: test from localhost But if i try the same command on a nearby host (vm2, same L2 network, but different IP network), it does not logged, but udp packet with test messages is received by host: [root@vm2 ~]# logger -d -i -n 172.16.3.50 -P 3514 "test from vm2" back to syslog-ng host: # ngrep -d ens224 -n 10 "test from" "udp and port 3514" interface: ens224 (172.16.3.0/255.255.255.0) filter: ( udp and port 3514 ) and ((ip || ip6) || (vlan && (ip || ip6))) match: test from ########################################################################################################################################################################################################################## U 100.64.128.5:24046 -> 172.16.3.50:3514 #218 <5>Feb 19 06:20:27 root[31663]: test from vm2. #####################################################################################################^Cexit 319 received, 1 matched looking at socket status on syslog-ng host: # ss -unlp | grep 3514 UNCONN 0 0 *:3514 *:* users:(("syslog-ng",pid=11198,fd=23)) no firewall or iptables rules (default all ACCEPT). Can someone suggest why syslog-ng does not log remote messages to my destination file? Thanks.

2 3

fedora/rhel & logrotate
by Peter Czanik (pczanik) 18 Feb '20

18 Feb '20

Hi, Recently I received a bug report about logrotate and RHEL 8: https://bugzilla.redhat.com/show_bug.cgi?id=1802165 In RHEL 8 /etc/logroate.d/syslog is now rsyslog specific and thus conflicts with the same file coming from syslog-ng. I changed my test repo https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ in a way that RHEL 7 keeps using /etc/logroate/syslog, but all the Fedora and RHEL 8 packages change the logroate file name to /etc/logroate/syslog-ng. Of course this means, that there is no conflict with rsyslog and there are two logrotate scripts for the same set of logs. I am trying to test it on my own as well on RHEL 8, but any additional testing and feedback are very welcome! Thanks, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

syslog-ng doesn't log if system date is older (in the past) than date on last shutdown
by Abhi Arora 18 Feb '20

18 Feb '20

Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every halt. *uname -a:* *Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux* I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown. My source was "systemd_journal()" and I changed it to "unix-dgram("/dev/log");". However, after changing it to unix-dgram, I don't see any logs even if my system date and time are correct. *"ls -l /dev/log":> lrwxrwxrwx 1 root root 28 Sep 30 13:28 /dev/log -> /run/systemd/journal/dev-log * *This is my old /etc/syslog-ng/scl.conf* : *@include 'scl/*/*.conf'@define java-module-dir "`module-install-dir`/java-modules"* ******************** This by* *cat /etc/syslog-ng/syslog-ng.conf* *@version: 3.15## Syslog-ng configuration file, compatible with default Debian syslogd# installation. Originally written by anonymous (I can't find his name)# Revised, and rewrited by me (SZALAY Attila <sasa(a)debian.org <sasa(a)debian.org>>)# First, set some global options.options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); keep_hostname(yes); keep_timestamp(yes);};######################### Sources######################### This is the default behavior of sysklogd package# Logs may come from unix stream, but not from another machine.#source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel"));};# If you wish to get logs from remote machine you should uncomment# this and comment the above source line.##source s_net { tcp(ip(127.0.0.1) port(1000) authentication(required) encrypt(allow)); };######################### Destinations######################### First some standard logfile#destination d_auth { file("/var/log/auth.log"); };destination d_cron { file("/var/log/cron.log"); };destination d_daemon { file("/var/log/daemon.log"); };destination d_kern { file("/var/log/kern.log"); };destination d_lpr { file("/var/log/lpr.log"); };destination d_mail { file("/var/log/mail.log"); };destination d_syslog { file("/var/log/syslog"); };destination d_user { file("/var/log/user.log"); };destination d_uucp { file("/var/log/uucp.log"); };# This files are the log come from the mail subsystem.#destination d_mailinfo { file("/var/log/mail/mail.info <http://mail.info/>"); };destination d_mailwarn { file("/var/log/mail/mail.warn"); };destination d_mailerr { file("/var/log/mail/mail.err"); };# Logging for INN news system#destination d_newscrit { file("/var/log/news/news.crit"); };destination d_newserr { file("/var/log/news/news.err"); };destination d_newsnotice { file("/var/log/news/news.notice"); };# Some 'catch-all' logfiles.#destination d_debug { file("/var/log/debug"); };destination d_error { file("/var/log/error"); };destination d_messages { file("/var/log/messages"); };# The root's console.#destination d_console { usertty("root"); };# Virtual console.#destination d_console_all { file("/dev/tty10"); };# The named pipe /dev/xconsole is for the nsole' utility. To use it,# you must invoke nsole' with the -file' option:## $ xconsole -file /dev/xconsole [...]#destination d_xconsole { pipe("/dev/xconsole"); };# Send the messages to an other host##destination d_net { tcp("127.0.0.1" port(1000) authentication(on) encrypt(on) log_fifo_size(1000)); };# Debian onlydestination d_ppp { file("/var/log/ppp.log"); };######################### Filters######################### Here's come the filter options. With this rules, we can set which# message go where.filter f_dbg { level(debug); };filter f_info { level(info); };filter f_notice { level(notice); };filter f_warn { level(warn); };filter f_err { level(err); };filter f_crit { level(crit .. emerg); };filter f_alllevel { level(debug, info, notice, warn,err,crit,emerg) and not filter(f_daemon);};filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };filter f_error { level(err .. emerg) ; };filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); };filter f_auth { facility(auth, authpriv) and not filter(f_debug); };filter f_cron { facility(cron) and not filter(f_debug); };filter f_daemon { facility(daemon) and not filter(f_debug); };filter f_kern { facility(kern) and not filter(f_debug); };filter f_lpr { facility(lpr) and not filter(f_debug); };filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); };filter f_mail { facility(mail) and not filter(f_debug); };filter f_news { facility(news) and not filter(f_debug); };filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };filter f_user { facility(user) and not filter(f_debug); };filter f_uucp { facility(uucp) and not filter(f_debug); };filter f_cnews { level(notice, err, crit) and facility(news); };filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); };filter f_ppp { facility(local2) and not filter(f_debug); };filter f_console { level(warn .. emerg); };######################### Log paths########################log { source(s_src); filter(f_auth); destination(d_auth); };log { source(s_src); filter(f_cron); destination(d_cron); };log { source(s_src); filter(f_daemon); destination(d_daemon); };log { source(s_src); filter(f_kern); destination(d_Gateway); };log { source(s_src); filter(f_lpr); destination(d_lpr); };log { source(s_src); filter(f_syslog3); destination(d_syslog); };log { source(s_src); filter(f_user); destination(d_user); };log { source(s_src); filter(f_uucp); destination(d_uucp); };log { source(s_src); filter(f_mail); destination(d_mail); };#log { source(s_src); filter(f_mail); filter(f_info); destination(d_mailinfo); };#log { source(s_src); filter(f_mail); filter(f_warn); destination(d_mailwarn); };#log { source(s_src); filter(f_mail); filter(f_err); destination(d_mailerr); };log { source(s_src); filter(f_news); filter(f_crit); destination(d_newscrit); };log { source(s_src); filter(f_news); filter(f_err); destination(d_newserr); };log { source(s_src); filter(f_news); filter(f_notice); destination(d_newsnotice); };#log { source(s_src); filter(f_cnews); destination(d_console_all); };#log { source(s_src); filter(f_cother); destination(d_console_all); };#log { source(s_src); filter(f_ppp); destination(d_ppp); };log { source(s_src); filter(f_debug); destination(d_debug); };log { source(s_src); filter(f_error); destination(d_error); };log { source(s_src); filter(f_messages); destination(d_messages); };log { source(s_src); filter(f_console); destination(d_console_all); destination(d_xconsole); };log { source(s_src); filter(f_crit); destination(d_console); };# All messages send to a remote site##log { source(s_src); destination(d_net); };# GentApp/GatewayAppdestination d_Gateway { file("/var/log/Gateway.log"template("$FULLDATE $FULLHOST $PROGRAM $PID [$FACILITY.$LEVEL] $MESSAGE\n") ); };filter f_Gateway { match("Gate*") or match("Gent*") or match("Notifier*") or match("Alarm*"); };* *log { source(s_src); filter(f_Gateway);filter(f_alllevel); destination(d_Gateway);} ;* Please help me in debugging the issue and fixing it.

2 2

Set log directory based on the source
by Frank DiGennaro 18 Feb '20

18 Feb '20

Hello; I'm somewhat of a newbie when it comes to syslog-ng so forgive me if the question is somewhat basic. I am running syslog-ng 3.25 on both server and clients and would like to set a variable based on the source. This is what I have: @define server-logdir "/var/log" @define client-logdir "/var/log/CLIENTS/$YEAR/$MONTH_ABBREV/$DAY/$HOST" @define logdir "" source s_local { logdir `server-logdir`; *No! ... ... }; source s_network { logdir `client-logdir`; *No! ... ... }; destination d_local0 { file( "`logdir`/local/local0.log" ); }; The goal is to set 'logdir' to either 'server-logdir' or 'client-logdir' and use it in the destination. I know there are probably several ways that this can be accomplished but I think this was will cut down on the number of lines in the configuration. So my question Is this. How do I set 'logdir' to either 'server-logdir' or 'client-logdir' in the source definition? Thanks; DiGennaro Frank DiGennaro 301-676-8193 (C) <mailto:frank@digennaro.com> frank(a)digennaro.com

3 5

[Question]: Initialisation step in syslog-ng-3.25.1
by Nitish Saboo 18 Feb '20

18 Feb '20

I am using syslog-ng-3.25.1.I am initialising syslog-ng in the following manner: int initialize_engine(const gchar* _module_path) { resolvedConfigurablePaths.initial_module_path = _module_path; app_startup(); msg_init(TRUE); [*]//resolved_configurable_paths_init(&resolvedConfigurablePaths); configuration = cfg_new(0x0302); cfg_set_global_paths(configuration); cfg_load_module(configuration, "basicfuncs"); cfg_load_module(configuration, "syslogformat"); pattern_db_global_init(); return 0; } I have assigned value to 'resolvedConfigurablePaths.initial_module_path' to provide module-path location. Do I have to add this step [*]'resolved_configurable_paths_init(&resolvedConfigurablePaths)' that is commented currently in initialize_engine method ? Thanks, Nitish

2 1

Warnings and error while loading default.xml in syslog-ng-3.25.1
by Nitish Saboo 17 Feb '20

17 Feb '20

Hi, I am using syslog-ng version 3.25.1.Getting following Warnings and error while initialising syslog-ng engine: [2020-02-13T10:47:29.627899] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file; [2020-02-13T10:47:29.627968] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file; [2020-02-13T10:47:29.628059] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file; [2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/opt/tap-parsing/patterns/default.xml', error='/opt/tap-parsing/patterns/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg' 1)For the following warnings, to which version I have to bump up the configuration file ? 2020-02-13T10:47:29.627899] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file; [2020-02-13T10:47:29.627968] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file; [2020-02-13T10:47:29.628059] WARNING: due to a bug in versions before syslog-ng 3.8numeric comparison operators like '!=' in filter expressions were evaluated as string operators. This is fixed in syslog-ng 3.8. As we are operating in compatibility mode, syslog-ng will exhibit the buggy behaviour as previous versions until you bump the @version value in your configuration file; Currrently the configuration version is the following: configuration = cfg_new(0x0302) Do I have to change it to '0x0319' as defined in 'lib/versioning.h' ? 2)The same default.xml file was getting loaded correctly in syslog-ng-3.6.2 and syslog-ng-3.7.1 but getting following error while loading same default.xml in syslog-ng3.25.1 2020-02-13T10:47:29.631090] Error parsing pattern database file; filename='/home/nsaboo/abc/default.xml', error='/home/nsaboo/abc/default.xml:17274:22: Joining rulesets with mismatching program name sets, program=proxysg'. What can be the reason for this error ? Thanks, Nitish

3 5

Insider 2020-02: Portability; secure logging; Mac support; RPM;
by Peter Czanik (pczanik) 13 Feb '20

13 Feb '20

Dear syslog-ng users, This is the 78th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Keeping syslog-ng portable -------------------------- I define syslog-ng as an “Enhanced logging daemon with a focus on portability and high-performance central log collection”. One of the original goals of syslog-ng was portability: running the same application on a wide variety of architectures and operating systems. After one of my talks mentioning syslog-ng, I was asked how we ensure that syslog-ng stays portable when all the CI infrastructure focus on 64bit x86 architecture and Linux. You can learn my answer from: https://www.syslog-ng.com/community/b/blog/posts/keeping-syslog-ng-portable Why choose syslog-ng over rsyslog -------------------------------- A question I often receive is ‘what are the differences between rsyslog and syslog-ng?’ It’s a little tricky to answer. First, because my experience is mostly with syslog-ng, and because there are many similarities between the two projects. This is where the syslog-ng users help me. They can clearly explain from firsthand experience why they chose syslog-ng. The following blog post includes some of the most common reasons why they choose syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/why-chose-syslog-ng-over-r… Secure logging with syslog-ng ----------------------------- In his presentation at FOSDEM, Stephan Marwedel mentioned a new syslog-ng feature he is working on, namely “(...) the design, implementation, and configuration of the secure logging service. Its aim is to provide tamper evident logging, i.e., to adequately protect log records of an information system against tampering and to provide a sensor indicating attack attempts.” (Marwedel, S. Secure logging with syslog-ng. (n.d.). Retrieved from https://fosdem.org/2020/schedule/event/security_secure_logging_with_syslog_… more about the presentation, and access the slides at https://fosdem.org/2020/schedule/event/security_secure_logging_with_syslog_… A pull request implementing this feature was already sent to syslog-ng on GitHub. You can follow its progress at https://github.com/syslog-ng/syslog-ng/pull/3121 State of syslog-ng on Mac ------------------------- Mac support is a returning question among syslog-ng users, especially when I talk to users in the US. For recent releases, each commit is automatically tested on macOS. However, there is not much information available on Mac support. Recently, I bought a MacBook to be able to test and document syslog-ng on Mac. Here are my first experiences and some future plans. https://www.syslog-ng.com/community/b/blog/posts/state-of-syslog-ng-on-mac Installing the latest syslog-ng on openSUSE, RHEL and other RPM distributions -------------------------------------- The syslog-ng application is included in all major Linux distributions, and you can usually install syslog-ng from the official repositories. If the core functionality of syslog-ng meets your needs, use the package in your distribution repository (yum install syslog-ng), and you can stop reading here. However, if you want to use the features of newer syslog-ng versions, for example, sending log messages to Elasticsearch or Apache Kafka, you have to either compile syslog-ng from source, or install it from unofficial repositories. This post explains how to do that. https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… NEW WHITE PAPERS * Upgrade to State-of-the-Art Log Management: https://www.syslog-ng.com/whitepaper/log-management-essentials8137754/ WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Getting error : .../libsyslog-ng.so: file not recognized: File truncated
by Nitish Saboo 11 Feb '20

11 Feb '20

Hi, I compiled the syslog-ng-3.7.1.tar.gz on a Ubuntu 16.04 machine .The syslog-ng got compiled successfully on this machine. ---------------------------------------------------------------------- Libraries have been installed in: /home/anassa/nitish/syslog-ng-3.7.1/install/lib If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the `-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the `LD_LIBRARY_PATH' environment variable during execution - add LIBDIR to the `LD_RUN_PATH' environment variable during linking - use the `-Wl,-rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to `/etc/ld.so.conf' See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages. ---------------------------------------------------------------------- --------------------------------------------------------------------Libraries have been installed in: /home/anassa/nitish/syslog-ng-3.7.1/install/lib/syslog-ng If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the `-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the `LD_LIBRARY_PATH' environment variable during execution - add LIBDIR to the `LD_RUN_PATH' environment variable during linking - use the `-Wl,-rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to `/etc/ld.so.conf' See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages. ---------------------------------------------------------------------- -Ubuntu:~/nitish/syslog-ng-3.7.1/install/lib$ gcc --version gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. My project got compiled in 'syslog-ng-3.7.1' folder on machine Ubuntu 16.04. I have a different machine with Ubuntu 14.04.I copied 'syslog-ng-3.7.1' folder on this machine (Ubuntu 14.04). In this machine I tried compiling my CGO project (Go and C) where my Go code is calling the C wrapper code that internally call syslog-ng api's. I have defined all the CFLAGS and LDFLAGS in my Go code for the run-timne dependencies. At the time of compilation it is failing with the following error: Error ===== lib/../deps/syslog/syslog-ng-3.7.1/install/lib/libsyslog-ng.so: file not recognized: File truncated [11:36:44][shellscript] collect2: error: ld returned 1 exit status [11:36:44][shellscript] make: *** [main] Error 2 The file 'libsyslog-ng.so' is present at the location 'lib/../deps/syslog/syslog-ng-3.7.1/install/lib/': total 3152 drwxrwxr-x 2 anassa anassa 4096 Feb 5 16:47 pkgconfig -rwxr-xr-x 1 anassa anassa 3208176 Feb 5 16:47 libsyslog-ng-3.7.so.0.0.0 lrwxrwxrwx 1 anassa anassa 25 Feb 5 16:47 libsyslog-ng.so -> libsyslog-ng-3.7.so.0.0.0 -rwxr-xr-x 1 anassa anassa 1155 Feb 5 16:47 libsyslog-ng.la lrwxrwxrwx 1 anassa anassa 25 Feb 5 16:47 libsyslog-ng-3.7.so.0 -> libsyslog-ng-3.7.so.0.0.0 drwxrwxr-x 3 anassa anassa 4096 Feb 5 16:47 python2.7 drwxrwxr-x 3 anassa anassa 4096 Feb 5 16:47 syslog-ng Can someone please guide me what could be the possible reasons to get this error ? What am I missing here ? Thanks, Nitish

2 5

syslog doesn't log if system date is older than date on last reboot
by Abhi Arora 07 Feb '20

07 Feb '20

Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every reboot. uname -a returns: Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown. Please help me in debugging the issue and fixing it.

4 16

Problem creating boot.log file
by Frank DiGennaro 04 Feb '20

04 Feb '20

All; I am running syslog-ng 3.25 and am trying to log the boot messages to boot.log. This is my configuration: @define logdir "/var/log/CLIENTS/$YEAR/$MONTH_ABBREV/$DAY/$HOST" filter f_boot { facility( local7 ) }; destination d_boot { file( "`logdir`/boot.log" ); }; log { source( s_local ); source( s_network ); filter( f_boot ); destination( d_boot ); }; source s_local { system( ); internal( ); }; The problem that I have is that the file is being created in /var/log, not in "logdir". Perhaps the reason for that could be that the syslog-ng service is started later in the boot process. Any insight at all would be greatly appreciated. Thanks; Frank

1 0

C_* macros with UTC time?
by Matus UHLAR - fantomas 04 Feb '20

04 Feb '20

Hello, I have syslog-ng OSE 3.8 on debian 9 server (soon to be upgraded). when I receive syslog lines using the syslog protocol, I found out that C_* macros seem to contain date in UTC format (one hour than our current zone). I have tried to use following template: "${C_DATE} ${R_DATE} HOST ${HOST} FULLHOST ${FULLHOST} HOST_FROM ${HOST_FROM} FULLHOST_FROM ${FULLHOST_FROM} ${MESSAGE}\n" with following result: Jan 31 10:42:06 Jan 31 11:42:07 HOST fantomas FULLHOST fantomas HOST_FROM 195.80.174.185 FULLHOST_FROM 195.80.174.185 Configuration reload request received, reloading configuration; Is there any reason why C_DATE should be in different time zone than R_DATE? timezones are not configured on either of servers, both servers are in CET (DST+1) thank you -- Matus UHLAR - fantomas, uhlar(a)fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Fighting for peace is like fucking for virginity...

2 4

unsubscribe
by Zhang, Husen 03 Feb '20

03 Feb '20

-----Original Message----- From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> On Behalf Of Peter Kokai (pkokai) Sent: Monday, February 3, 2020 7:33 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: EXTERNAL: Re: [syslog-ng] Not finding header files in syslog-ng-3.25.1 as compared to syslog-ng-3.6.2 Hello, There is no dev doc about it, as you might noticed. Probably no replace that with this answer exists. Luckily git remembers: git log --follow -p -- <path> git log --follow -p -- modules/dbparser/patterndb-int.h commit 1931dabf07b79a9b10d5902b5ad0b7f0509e7594 Author: Balazs Scheidler <bazsi77(a)gmail.com> Date: Wed Jul 29 15:10:06 2015 +0200 dbparser: get rid off patterndb-int.h As we have the broken-down header files, this is not needed anymore. Signed-off-by: Balazs Scheidler <balazs.scheidler(a)balabit.com> This commit might gave you a hint, which headers it was replaced with. Alternative idea: simply check which header provides the function you need. git log --follow -p -- src/tags.h commit ee1d0d7f0720c3ded0d2bfaf22ca6e42d669b7a6 Author: Balazs Scheidler <bazsi77(a)gmail.com> Date: Thu Sep 9 13:48:42 2010 +0200 separate directory hierarchy into lib/, modules/ and syslog-ng/ It was more and more cumbersome to compile syslog-ng during development, since a core change always recompiled (or just relinked) the modules as well even when that was not strictly necessary. (make install in src/) This patch reorganizes the directory hierarchy, lib is where the core lives modules where plugins are and syslog-ng is where the main program resides. The change here more simple, just moving the actual file. (Note tags.h was moved twice since your version, and now it is under lib/logmsg/) -- Kokan On Mon, Feb 03, 2020 at 05:34:21PM +0530, Nitish Saboo wrote: > CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. > > Hi, > > I was using syslog version: syslog-ng-3.6.2. > In that particular version of my c wrapper code I was including the following header files: > > #include <modules/dbparser/patterndb-int.h> > #include <syslog-ng/tags.h> > > When I upgraded the syslog-ng version to 3.25.1,I don't see these > header file in the modules and include folder respectively post > installation > > Can someone please help me clarify, what these files have been replaced with? > > Thanks, > Nitish > ______________________________________________________________________ > ________ Member info: > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter > .kokai%40oneidentity.com%7C0616c4f0e4cc4cd1d26f08d7a8a13d03%7C91c369b5 > 1c9e439c989c1867ec606603%7C0%7C0%7C637163282794906164&sdata=mn27bw > mBoTLzPsp%2FeL648GZ81vo%2FfZMrfAeNdfwX5bk%3D&reserved=0 > Documentation: > https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b > alabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data > =02%7C01%7Cpeter.kokai%40oneidentity.com%7C0616c4f0e4cc4cd1d26f08d7a8a > 13d03%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637163282794916099& > amp;sdata=W5ip5K3TXAoux%2BBRNPoJpw19tQoxaSHF0nkLMc37%2BFQ%3D&reser > ved=0 > FAQ: > https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.b > alabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40one > identity.com%7C0616c4f0e4cc4cd1d26f08d7a8a13d03%7C91c369b51c9e439c989c > 1867ec606603%7C0%7C0%7C637163282794916099&sdata=UZs%2FG%2Fg%2BoUES > RozPNJOfyUwDKYpB5gzQ9jKc4XysbSY%3D&reserved=0 > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Not finding header files in syslog-ng-3.25.1 as compared to syslog-ng-3.6.2
by Nitish Saboo 03 Feb '20

03 Feb '20

Hi, I was using syslog version: syslog-ng-3.6.2. In that particular version of my c wrapper code I was including the following header files: #include <modules/dbparser/patterndb-int.h> #include <syslog-ng/tags.h> When I upgraded the syslog-ng version to 3.25.1,I don't see these header file in the modules and include folder respectively post installation Can someone please help me clarify, what these files have been replaced with? Thanks, Nitish

3 2

Memory not freeing up in syslog-ng-3.6.2 during reload
by Nitish Saboo 29 Jan '20

29 Jan '20

Hi, I am using syslog-ng version 3.6.2: syslog-ng --version syslog-ng 3.6.2 Installer-Version: 3.6.2 Revision: Compile-Date: May 2 2019 01:20:32 Available-Modules: afmongodb,afstomp,afuser,csvparser,afprog,afamqp,cryptofuncs,afsocket,graphite,dbparser,affile,syslogformat,afsocket-notls,system-source,confgen,pseudofile,linux-kmsg-format,basicfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off I initialised syslog-ng with the following configuration: int initialize_engine(const gchar* filename, const gchar* _module_path) { module_path = _module_path; app_startup(); msg_init(TRUE); configuration = cfg_new(0x0302); plugin_load_module("basicfuncs", configuration, NULL); plugin_load_module("syslogformat", configuration, NULL); pattern_db_global_init(); return 0; } int reload_pattern_db(const gchar* filename, key_value_cb cb) { if(patterndb!= NULL){ pattern_db_free(patterndb); } patterndb = pattern_db_new(); pattern_db_reload_ruleset(patterndb, configuration, filename); pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb); return 0; } When I reload the syslog-ng, the memory keeps on increasing.Looks like syslog-ng is not freeing up the memory.It continues to occupy the memory unless the process is restarted. Just want to understand if it is a sign of memory leak or default behavior of syslog-ng to not free-up memory during reload ? Thanks, Nitish

2 15

which platform do you use for syslog-ng?
by Peter Czanik (pczanik) 28 Jan '20

28 Jan '20

Hi, I am often asked on which platforms people are running syslog-ng. Based on mailing list questions, github issues and so on I have some guesses, but every once in a while I run a Twitter poll: https://twitter.com/PCzanik/status/1222089870782484480 Please answer it and share it, so more people can answer it! Thanks, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

facility override
by Alexandre Santos 20 Jan '20

20 Jan '20

Hi, Is there a way of overriding the facility of the messages on the destination, besides using a template like: { template("$(if ('${LEVEL_NUM}' == '2') '<186>1' $(if ('${LEVEL_NUM}' == '3') '<187>1' $(if ('${LEVEL_NUM}' == '4') '<188>1' ${HOST}\\@`HOSTIPV4` ${PROGRAM} ${PID} ${SDATA} ${MESSAGE}\n"); }; Thanks in advance, Alex

2 1

Rewrite set var value with tag?
by Faine, Mark R. (MSFC-IS40)[NICS] 20 Jan '20

20 Jan '20

Is it possible to do something like: rewrite { set-tag("bar"); }; And then use a rewrite to set the value of a variable to the value of the tag rewrite { set("${TAGS}", value("foo")); }; I need to set a value that I've determined in a log path at the top of the file in several log paths that occur after. It looks the scope for variables is at the log path level so that doesn't help me. I'm also trying to avoid nesting all of the other log paths inside of the top log path where I determine the value. I'm trying to avoid: log{ ... filter{ ... tags("bar"); rewrite{ set("bar", "foo"); }; }; }; log{... log{... Thanks, -Mark

3 3

Go away
by johns spam 16 Jan '20

16 Jan '20

Hello, I am wondering if there is any facility to tell a client to go away? i.e. https://http2.github.io/http2-spec/#GOAWAY I know syslog-ng is not http2 but is there any side-channel mechanism a central server can use to make a client go away? The use case I am thinking is doing a rolling restart of a pool of syslog-ng central server. I think this could also be useful to cycle connections periodically, i.e. to rebalance connections to a central server. Thanks

2 1

librdkafka and messages dropped
by John Skopis 15 Jan '20

15 Jan '20

Hello, We have a syslog-ng aggregator that accepts messages in a few different formats and logs the message format to kafka. The mapping looks something like: format1 -> topic1 format2 -> topic2 ... Each agent will emit messages in ~4 different formats and send to the aggregator. The aggregator is configured with a kafka_c destination with 8 threads/destination. We were previously using the java kafka client without issue. We recently upgraded to 3.24.1 + librdkafka 1.22 After upgrading we are seeing an issue where (seemingly randomly) syslog-ng stops producing messages into kafka. Checking syslog-ng-ctl stats I can see d_kafka_format1 has dropped message counter increasing d_kafka_format1 queued messages counter increases to 80000 (10000*8 threads iw-size) d_kafka_format1 processed messages does not increase There are messages printed to the log that the destination queue is full for all of the rdkafka threads, There are spurious failures to a kafka broker but it seems rdkafka reconnects. Seems like once the destination queue becomes full syslog-ng never recovers and the instance must be restarted. It also seems like rdkafka stops trying to reconnect to kafka (possibly?). Is this expected behavior or is there some bug around reconnecting to kafka after a spurious timeout? Just speculating here but does rdkafka stop trying to reconnect after N times? Are there any examples of destroy/create destination client after failure thresholds? Thanks

2 5

3.25 and my RPM stable repos
by Peter Czanik (pczanik) 13 Jan '20

13 Jan '20

Hi, Version 3.25 of syslog-ng was released a week ago, and it seems to be rock solid. Still, as today is my last working day this year, I feel more comfortable to push it into my stable repos only on the first week of January, when I am back. If you still want to experiment with 3.25 on openSUSE/SLES or fedora/RHEL/CentOS you can do that in the versioned repositories. Note that RHEL 8 / CentOS 8 support is now merged with the rest, it's not in a separate repository any more. suse: https://build.opensuse.org/project/show/home:czanik:syslog-ng325 red hat: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng325/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Lines per file + Syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 13 Jan '20

13 Jan '20

Hello everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

Load Balancing question
by Janczuk, George 13 Jan '20

13 Jan '20

I'm new to syslog-ng, so please excuse my ignorance if this a newbie or dumb question (I did do some googling first though - regrettably without success). Where I work we have deployed a syslog-ng OSE cluster (five members) fronted by an F5 LTM. One of our up-stream syslog event sources is a service provider's syslog-ng server acting as a relay and passing on all ingested syslog events to our cluster (via the F5 LTM VIP). The up-steam syslog-ng server aggregates events from hundreds of devices, and therefore the stream is very busy (i.e. never idle/quiet). Therefore, it turns out that the connection from the up-stream relay is NEVER dropped and remains in place almost permanently (certainly days at a time). Consequently, without any cycling of the connection the connection is bound to a single cluster member, and ALL events from the up-stream relay are delivered to that single cluster member and are NOT actually load balanced (with the remaining four members essentially being idle as the volume from the relay greatly eclipses the event volume from other even sources). I suppose the question is whether: a) Can we in some way configure the up-steam syslog-ng relay to cycle the outbound connection after some sort of threshold (be that time based - say every 15 seconds; or volume based - say every 100 events; or even both... - whichever occurs first)?, or b) Can the up-stream syslog-ng keep multiple connections open concurrently and can it then use round-robin or stochastic logic to determine which of the connections to send each relayed event to? I would very much like to hear from people about how to actually load balance an event stream from a syslog-ng relay. NOTE: People might ask the question: "If the up-stream relay is a single node, then why do even need a down-stream cluster, if it's not required upstream"? Two reasons really: 1. The up-steam service provider has deployed dedicated chunky tin (i.e. they've used a vertical scale model), whilst for our syslog-ng cluster we're using a private-cloud with much smaller VMs combined with a horizontal scale model, and 2. The relay is doing a fairly minimal collect and forward, whilst our syslog-ng cluster will be doing more event transformation and enrichment, which is ultimately more CPU intensive. ************** IMPORTANT MESSAGE ***************************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. **************************************************************

3 2

Disk-buffer in kafka-destinations
by Ronald Fenner 10 Jan '20

10 Jan '20

I can't see to me get syslog-ng to start if I add the disk-buffer to the destination. From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block. Is this tied to a specific destination type? We are trying to make out messages going to kafka more reliable when the cluster is unavailable. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 4

Increase number of records per file
by William Luiz Ribeiro Vasconcelos Da Silva 10 Jan '20

10 Jan '20

Good morning everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

customized rewrite to mysql in syslog-ng
by JAHANZAIB SYED 09 Jan '20

09 Jan '20

I have syslog-ng version 3.25.1 on Ubuntu 18.04 server. All logs from remote mikrotik router device is stored in local file and in mysql DB as well. I want to know if there is any method in which we can customize the incoming message and write it to mysql table in different columns. Incoming message example: Jan 8 13:50:24 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49466->172.217.19.10:443, NAT (172.16.0.2:49466->101.11.11.252:2223)->172.217.19.10:443, len 1023 Jan 8 13:53:09 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49482->58.27.130.12:443, NAT (172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443, len 1152 The relevant part that I want to store is HOSTNAME, DATETIME, AND message part 172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443 * into following tables hostname datatime username userip userport wanip wanport dst-ip dst-port 101.11.11.252 xxxx pppoe-zaib 172.16.0.2 49466 101.11.11.252 2223 172.217.19.10 443 [cid:48adc02d-e37b-47af-aa2d-67e59b25f8d6] Regards, SYED JAHANZAIB <http:///>

2 6

Syslog-NG mysql log with auto increment not working
by JAHANZAIB SYED 08 Jan '20

08 Jan '20

I have Syslog-ng (ver 3.25.1) on Ubuntu 18.4 server. I am using following code to dynamically create date wise table and insert records accordingly. ************************ destination d_mysql { sql(type(mysql) host("localhost") username("root") password("XXXXXXXX") database("syslog") table("${R_YEAR}_${R_MONTH}_${R_DAY}") columns( "id int(20) unsigned not null auto_increment primary key", "host varchar(40) not null", "facility varchar(20)", "priority varchar(10)", "level varchar(10)", "program text", "date date not null", "time time not null", "message text not null") values("", "$FULLHOST", "$FACILITY", "$PRIORITY", "$LEVEL", "$PROGRAM", "$R_YEAR-$R_MONTH-$R_DAY", "$R_HOUR:$R_MIN:$R_SEC", "$MSG",) indexes("host","priority")); }; ************************ Above code auto creates table like this ************************ | Field | Type | Null | Key | Default | Extra | +----------+------------------+------+-----+---------+----------------+ | id | int(11) unsigned | NO | PRI | NULL | auto_increment | | host | varchar(40) | NO | MUL | NULL | | | facility | varchar(20) | YES | | NULL | | | priority | varchar(10) | YES | MUL | NULL | | | level | varchar(10) | YES | | NULL | | | program | text | YES | | NULL | | | date | date | NO | | NULL | | | time | time | NO | | NULL | | | message | text | NO | | NULL | | +----------+------------------+------+-----+---------+----------------+ 9 rows in set (0.01 sec) ************************ Problem is that its not inserting record in the table. I enabled mysql log and I can see only one entry ************************ 2020-01-08T05:26:29.016553Z 167 Query INSERT INTO 2020_01_08 (id, host, facility, priority, level, program, date, time, message) VALUES ('NULL', '101.11.11.252', 'user', 'notice', 'notice', 'script,warning', '2020-01-08', '10:23:30', '10:23:30 jan/08/2020') ************************ but this entry is not recorded in the mysql table, When I remove the ID columns/values from the syslog code, DROP the table, & restart syslog-ng service, then table again recates, and all records inserts fine without problem. When I insert record using phpmyadmin, the record enter fine with this code INSERT INTO `2020_01_08` (`id`, `host`, `facility`, `priority`, `level`, `program`, `date`, `time`, `message`) VALUES (NULL, '2', '2', '2', '2', '2', '2020-01-01', '42:00:00', '2'); I tried to add NULL in the values for ID in the syslog-ng code , but no use, as soon as i remove ID DROP the table, & restart syslog-ng service, all goes fine. How can I configure the auto increment column ? Regards, SYED JAHANZAIB Web : http://aacable.wordpress.com LinkedIn: http://pk.linkedin.com/pub/syed-jahanzaib/24/3b/407 <http:///>

2 2

State of syslog-ng on Mac
by Peter Czanik (pczanik) 20 Dec '19

20 Dec '19

Hi, I just published a blog about the state of syslog-ng on Mac: https://www.syslog-ng.com/community/b/blog/posts/state-of-syslog-ng-on-mac As I write about future plans at the end of the blog, you can see that I'm an almost complete newbie on the Mac (the last time I used it actively was on the Motorola 68040) and it's mostly a free time project. Still, any questions, suggestion or help are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 3

Add prefix Or suffix inside of FileName
by William Luiz Ribeiro Vasconcelos Da Silva 17 Dec '19

17 Dec '19

Hello everyone, It is possible during syslog-ng processing to insert a suffix, as we have a system that will collect these files. And if we keep the same suffix, we have a big problem inside of prodction enviroment. Here's an example of how the file generation part: destination d_model { file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") owner(syslog) group(syslog) perm (0755) ); }; Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 2

json destination config help
by Zhang, Husen 12 Dec '19

12 Dec '19

Hi community, I'm trying to have syslog-ng to write logs to json. My d_json_syslog-ng.conf: [cid:image001.png@01D5B023.F072D8A0] ├── conf.d │ ├── d_json_syslog-ng.conf │ └── es.conf.bak ├── patterndb.d ├── scl.conf └── syslog-ng.conf

3 5

syslog-ng 3.25 packages for Debian/Ubuntu
by Laszlo Budai 12 Dec '19

12 Dec '19

Hi, syslog-ng 3.25.1[1] packages are available in OBS repositories[2, 3]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 Note that Debian 10 and Ubuntu 19.04 are available from a separate repository[3] and there is no java support on these distributions. The reason is that these OBS builders don't have the required openjdk packages. Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.25.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.25.1
by Laszlo Budai 12 Dec '19

12 Dec '19

3.25.1 Highlights * http-destination: Users now can specify the action for any HTTP result code. Use with response-action(response_code => action) in your http block. Available actions are: success, retry, drop and disconnect. (#3007<https://github.com/syslog-ng/syslog-ng/pull/3007>) * syslog-ng-cfg-db: Added a new script, which can provide the options of sources and destinations queried by the user. This tool can make the configuration of syslog-ng a lot easier. Use with ./syslog-ng-cfg-db.py from the contrib/config_database dir.(#2997<https://github.com/syslog-ng/syslog-ng/pull/2997>) * redis-destination: Improved the performance by 2 orders of magnitude. In our labor environment, now it operates at 25k EPS. (#2972<https://github.com/syslog-ng/syslog-ng/pull/2972>) Features * create-dirs(): Added to pipe() source/destination, and standardize the behavior. (#3018<https://github.com/syslog-ng/syslog-ng/pull/3018>, #2635<https://github.com/syslog-ng/syslog-ng/issues/2635>) * default-network-drivers: Added max-connections() option, to change the limit from 10. (#2961<https://github.com/syslog-ng/syslog-ng/pull/2961>) * checkpoint: Added support for timezone value at the end of timestamps. (#3033<https://github.com/syslog-ng/syslog-ng/pull/3033>) * filter/rewrite: Added disable-jit flag to disable JIT PCRE compilation. (#2992<https://github.com/syslog-ng/syslog-ng/pull/2992>, #2986<https://github.com/syslog-ng/syslog-ng/issues/2986>) * syslog-ng-ctl: Added export-config-graph option to visualize config graph. (#2990<https://github.com/syslog-ng/syslog-ng/pull/2990>) * build/travis: Added ARM64 arch support. (#2967<https://github.com/syslog-ng/syslog-ng/pull/2967>) * build/dbld: Readded CentOS 6 support. (#2860<https://github.com/syslog-ng/syslog-ng/pull/2860>, #2971<https://github.com/syslog-ng/syslog-ng/pull/2971>, #3028<https://github.com/syslog-ng/syslog-ng/pull/3028>) * python: Added Python 3.8 support. (#3017<https://github.com/syslog-ng/syslog-ng/pull/3017>) Bugfixes * tls: Fixed an infinite loop which occured, when a TLS connection broke. (#3026<https://github.com/syslog-ng/syslog-ng/pull/3026>, #3009<https://github.com/syslog-ng/syslog-ng/issues/3009>) * log-block: Fixed an issue, where inline network destinations disjointed the rest of the config. (#2989<https://github.com/syslog-ng/syslog-ng/pull/2989>, #2820<https://github.com/syslog-ng/syslog-ng/issues/2820>) * kafka/network-load-balancer: Fixed a crash when an argument was set to empty. (#3002<https://github.com/syslog-ng/syslog-ng/pull/3002>) * python-source: Fixed a memory corruption during reload. (#3014<https://github.com/syslog-ng/syslog-ng/pull/3014>) * python-destination: Actually use return value of open method. (#2998<https://github.com/syslog-ng/syslog-ng/pull/2998>, #2513<https://github.com/syslog-ng/syslog-ng/issues/2513>) * python-fetcher: Fixed FETCH_NO_DATA and FETCH_TRY_AGAIN constants. (#3012<https://github.com/syslog-ng/syslog-ng/pull/3012>) * python: Fixed python Exception reporting when no Exception happened. (#2995<https://github.com/syslog-ng/syslog-ng/pull/2995>) * telegram: Fixed the syntax error of the use-system-cert-store() option. (#2977<https://github.com/syslog-ng/syslog-ng/pull/2977>) * config: Throw error to single dots, which were ignored before. (#3000<https://github.com/syslog-ng/syslog-ng/pull/3000>) * file-destination: Delay ACKs until messages are written to disk. This fixes message drop on I/O error and message lost in the LogProtoFileWriter in case of a crash, by retrying to send the message. (#2985<https://github.com/syslog-ng/syslog-ng/pull/2985>) * http-destination: Handle global template options values. (#3020<https://github.com/syslog-ng/syslog-ng/pull/3020>) * timeutils: Fixed month and day name parsing, when only the first 2 characters matched. (#3035<https://github.com/syslog-ng/syslog-ng/pull/3035>) * logmsg: Added default PRI value (LOG_USER | LOG_NOTICE) to log messages created without initial parsing. (#2974<https://github.com/syslog-ng/syslog-ng/pull/2974>) * packaging: Added ordering dependencies network.target and network-online.target to the service files. (#2994<https://github.com/syslog-ng/syslog-ng/pull/2994>, #2667<https://github.com/syslog-ng/syslog-ng/issues/2667>) * amqp: Support older (0.7.1) version (#2999<https://github.com/syslog-ng/syslog-ng/pull/2999>) * loggen: Set plugin path in installation time. (#3019<https://github.com/syslog-ng/syslog-ng/pull/3019>) * timeutils/patterndb: Fixed some undefined behaviours. (#2969<https://github.com/syslog-ng/syslog-ng/pull/2969>) * stomp: Fixed a buffer over-read on connection. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) * pseudofile: Fixed a crash, when template() option is not set. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) * wildcard-source: Fixed a crash, when max-files() was set to 0. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) Other changes * syslog-ng-debun: Various maintenance updates and small fixes. (#2993<https://github.com/syslog-ng/syslog-ng/pull/2993>) * scl: Avoid @requires loading the plugins themselves. (#2887<https://github.com/syslog-ng/syslog-ng/pull/2887>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Clément Besnier, Gabor Nagy, jadhavsumit98, Janos Szigetvari, Laszlo Budai, Laszlo Szemere, László Várady, MikeLim, Nikita Uvarov, Norbert Takacs, pabloli, Péter Kókai, Zoltan Pallagi.

1 0

Insider 2019-12: Kibana 7 & GeoIP; PE 6 to 7 upgrade; RHEL 8; Elastic stack;
by Peter Czanik (pczanik) 12 Dec '19

12 Dec '19

Dear syslog-ng users, This is the 77th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Set up Kibana 7 for syslog-ng & GeoIP ------------------------------------- Version 7 of the Elastic stack was released a few months ago, and brought several breaking changes that affect syslog-ng. In my previous blog post, I gave details about how it affects sending GeoIP information to Elasticsearch. From this blog post you can learn about the Kibana side, which has also changed considerably compared to previous releases. Configuration files for syslog-ng are included, but not explained in depth, as that was already done in previous posts. https://www.syslog-ng.com/community/b/blog/posts/set-up-kibana-7-for-syslog… Upgrading a syslog-ng PE 6 configuration to 7 --------------------------------------------- Upgrading syslog-ng PE 6 to 7 is no more difficult than any major version upgrade. The key here is to use the latest version of PE 7 (at least version 7.0.17), as it has the necessary compatibility fixes to make the upgrade experience smoother. From this blog, you can learn how to upgrade a simple configuration containing a wildcard file source and an RLTP source to work correctly with syslog-ng PE 7. https://www.syslog-ng.com/community/b/blog/posts/upgrading-a-syslog-ng-pe-6… State of syslog-ng on RHEL 8 / CentOS 8 --------------------------------------- Version 8, a new major version of Red Hat Enterprise Linux was released this spring. Now that CentOS 8 is also available, there is a rapidly growing interest in syslog-ng running on these platforms. From this blog, you can learn about the availability of the current syslog-ng package and future plans. https://www.syslog-ng.com/community/b/blog/posts/state-of-syslog-ng-on-rhel… Using syslog-ng with the Elastic stack -------------------------------------- One of the most popular destinations of syslog-ng is Elasticsearch. Any time a new language binding was introduced to syslog-ng, someone implemented an Elasticsearch destination for it. For many years, the official Elasticsearch destination for syslog-ng was implemented in Java. With the recent enhancements to the http() destination of syslog-ng, a new, native C-based implementation called the elasticsearch-http() destination is available. https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-with-the-e… WEBINARS * You can achieve 1m+ UDP messages per second ingestion challenge: https://www.syslog-ng.com/webcast-ondemand/you-can-achieve-1m-udp-messages-… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

How to syslog-ng execute shell script
by William Luiz Ribeiro Vasconcelos Da Silva 10 Dec '19

10 Dec '19

Hello People, How to configure the syslog-ng.conf file, to call a shell script, for example to do zip file. It is possible to do this? Tks, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 5

Support for netflow logs
by Raghunath Adhyapak 04 Dec '19

04 Dec '19

Hi, I was trying to receive Netflow logs from firewall devices in syslog-ng and then forward to a central server. Does syslog-ng support netflow such that I can validate and filter out all non-netflow log lines? I also dumped some netflow logs to a file and found it to be binary. Therefore I haven't been able to ascertain the format and filtering mechanism. Any pointers on this topic would be helpful. Thanks Raghu

4 7

Its possible compress file using syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 03 Dec '19

03 Dec '19

How to configure the syslog-ng.conf to do compress file? Its possible to use this feature inside of file structure: file("/<dir>/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}${SEC}_GMT${TZ}_${UNIQID}.log" template("${PROGRAM};${MESSAGE};\n") owner(user) group(user) perm (0755) Tks, Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

Structure data set to "-"
by Debjyoti Mukherjee 03 Dec '19

03 Dec '19

Hello Trying to send logs to remote syslog server in RFC 5424 format. The STRUCTURE_DATA should be set to "-". What is the way to the this value to "-" Currently it is coming as [meta sequenceId="21"]. I am using Openwrt and the syslog version is 3.24 Thank you

2 5

Uniqid
by William Luiz Ribeiro Vasconcelos Da Silva 03 Dec '19

03 Dec '19

Hello People, I'm using the uniqid in my syslog-ng.conf , but when using it in the file name, the output was presented in hexadecimal, example: "8e99d7b2(a)000000000000040b.log" How to convert the output to decimal because we have a balancer layer to collect files ending in even numbers and another layer for odd numbers. This is part of syslog-ng.conf: "${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}${SEC}_GMT${TZ}_${UNIQID}.log" Tks Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

SYSLOGHOST is being replaced with IP
by Raghunath Adhyapak 29 Nov '19

29 Nov '19

Hi, I'm observing that syslog-ng is modifying the SYSLOGHOST in the incoming log line and outputting an IP instead. I would like to retain the incoming hostname in incoming syslog and forward the same information. Here is my incoming log line: <13>Nov 29 04:07:40 BVRM-DC04 AgentDevice=WindowsLog\tAgentLogFile=Security\tPluginVersion=7.2.8.91\tSource=Microsoft-Windows-Security-Auditing\tComputer= BVRM-DC04.xxxxxxxx.com\tOriginatingComputer=172.26.1.60\tUser=\tDomain=\tEventID=4634\tEventIDCode=4634\tEventType=8\tEventCategory=12545\tRecordNumber=166757582\tTimeGenerated=1575029259\tTimeWritten=1575029259\tLevel=Log Always\tKeywords=Audit Success\tTask=SE_ADT_LOGON_LOGOFF\tOpcode=Info\tMessage=An account was logged off. Outgoing log line: <13>Nov 29 04:07:40 172.22.2.55 AgentDevice=WindowsLog\tAgentLogFile=Security\tPluginVersion=7.2.8.91\tSource=Microsoft-Windows-Security-Auditing\tComputer= BVRM-DC04.xxxxxxxx.com\tOriginatingComputer=172.26.1.60\tUser=\tDomain=\tEventID=4634\tEventIDCode=4634\tEventType=8\tEventCategory=12545\tRecordNumber=166757582\tTimeGenerated=1575029259\tTimeWritten=1575029259\tLevel=Log Always\tKeywords=Audit Success\tTask=SE_ADT_LOGON_LOGOFF\tOpcode=Info\tMessage=An account was logged off. FYI, this is log from Windows, but same is happening for syslog from other firewalls as well. My syslog-ng.conf: @version: 3.24 @include "scl.conf" ######################## # Sources ######################## source s_test_net { syslog(transport(udp) port(2514) ); }; ######################## # Destinations ######################## destination d_test { file("/tmp/test.log"); }; ######################## # Log paths ######################## log { source(s_test_net); destination(d_test); }; Thanks Raghu

2 2

syslog-ng packages for RHEL 8 / CentOS 8
by Peter Czanik (pczanik) 27 Nov '19

27 Nov '19

Hi, Today I reached a new milestone with syslog-ng packaging for RHEL 8 / CentOS 8, so here is a quick update: * syslog-ng 3.23 is available in EPEL 8 ( https://fedoraproject.org/wiki/EPEL ). It is missing the C-based Kafka driver (librdkafka in RHEL is too old) and the Java-based destination drivers (Kafka, Elastic, HDFS) due to missing dependencies in RHEL/EPEL. * highly experimental syslog-ng package missing most of the features needing extra dependencies. Core features, Elasticsearch (HTTP) destination are available in the package. You need to first enable the EPEL 8 repository and then add: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng324el8/ * Today I successfully added RHEL 8 support to my git snapshot repository: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ Some dependencies, like recent version of Kafka are available from this repository, the rest from EPEL. As far as I am aware, all OSE features are now supported, except for tcp-wrapper, which was removed from Fedora & RHEL 8. This repository is regularly updated from git. * Once syslog-ng 3.25 is available, the upcoming syslog-ng325 repo will already support RHEL 8 / CentOS 8 and soon after my syslog-ng-stable repo will also receive RHEL 8 support. As usual, any feedback is very welcome and will be handled with best effort. I'd especially welcome feedback on my syslog-ng-githead repo -- even if that's not intended for production environments -- as that's where I prepare for the next syslog-ng release. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syntax error when using syslog-ng.conf from debian packaging
by Alexandre Santos 21 Nov '19

21 Nov '19

Hi, I am running into the following error when running the syslog-ng with the syslog-ng.conf from debian packaging: *Error parsing affile, syntax error, unexpected ')', expecting LL_IDENTIFIER or LL_STRING in /usr/local/etc/syslog-ng.conf:68:34-68:35:63 #64 destination d_console { usertty("root"); };6566 # Virtual console.67 #68----> destination d_console_all { file(`tty10`); };68----> ^* Do you have any idea about this? Thanks in advance, Alex

3 4

Re: [syslog-ng] Abort message processing - multiple includes
by Matthias Gruber 19 Nov '19

19 Nov '19

Hi! Good Lord, I think I just found the answer to my question in the Documentation :-/ :-) "This default behavior can be changed using the flags() parameter. Flags apply to individual log paths, they are not global options. " Therefore I think the final breaks out of the log-processing of the actual log-directive, but others "matching" will processed Argh... think I have to cat some of my config-files together... Sorry for "spamming" :-) cheers Matthias P.S. If I am wrong, please correct me (hope dies last ;->) ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Von: Matthias Gruber/METZLER An: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Datum: 19.11.2019 08:54 Betreff: Abort message processing - multiple includes Hi! Perhaps I misunderstood it, or got a bit crossed up, because I dig into syslog-ng-Configs since weeks, but I am unsure about the "flags(final)" statement I have several includes with several log-statements and several sources e.g. (no runnable syntax, only for demonstration-purpose) 0001-A.conf log { source(s_tcp_A); filter(f_filter_A); destination(d_somewhere_A); flags(final); }; 0002-B.conf log { source(s_tcp_A); destination(d_somewhere_BBB); flags(final); }; As I understand it, first of all, its included in "alphabetical order", so in my case first 0001-A.conf, then 0002-B.conf and so on.... But now my "Problem" and question (Yes I would normally do a "if"-Construct to solve that, problem above, but I used that example to understand the flags(final),) If I have a message received by s_tcp_A, filter f_filter_A hits, the rule sends it to d_somewhere_A and do a "Escape" cause of flags(final), does the message still got processes by the include of 0002-B.conf?? Does the flags(final) is only relevant for one "log-rule", or does it ends the whole processing? To be honest, I habe several includes, and a "last" one, which is imported last, with a "hit"-all, in my childish thought, if the flags(final) stopps processing of the message compleetly this hit-all only gets "the junk" where nothing else hists. Why do I do that, and not write it in one file with a large if-elif-else-construct, I do that for the readability since this file would be extremly large. I am thankfull for any help cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Persönlich haftende Gesellschafter: Harald Illy, Michael Klaus, Friedrich von Metzler, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 27 515 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

1 0

Abort message processing - multiple includes
by Matthias Gruber 19 Nov '19

19 Nov '19

Hi! Perhaps I misunderstood it, or got a bit crossed up, because I dig into syslog-ng-Configs since weeks, but I am unsure about the "flags(final)" statement I have several includes with several log-statements and several sources e.g. (no runnable syntax, only for demonstration-purpose) 0001-A.conf log { source(s_tcp_A); filter(f_filter_A); destination(d_somewhere_A); flags(final); }; 0002-B.conf log { source(s_tcp_A); destination(d_somewhere_BBB); flags(final); }; As I understand it, first of all, its included in "alphabetical order", so in my case first 0001-A.conf, then 0002-B.conf and so on.... But now my "Problem" and question (Yes I would normally do a "if"-Construct to solve that, problem above, but I used that example to understand the flags(final),) If I have a message received by s_tcp_A, filter f_filter_A hits, the rule sends it to d_somewhere_A and do a "Escape" cause of flags(final), does the message still got processes by the include of 0002-B.conf?? Does the flags(final) is only relevant for one "log-rule", or does it ends the whole processing? To be honest, I habe several includes, and a "last" one, which is imported last, with a "hit"-all, in my childish thought, if the flags(final) stopps processing of the message compleetly this hit-all only gets "the junk" where nothing else hists. Why do I do that, and not write it in one file with a large if-elif-else-construct, I do that for the readability since this file would be extremly large. I am thankfull for any help cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Persönlich haftende Gesellschafter: Harald Illy, Michael Klaus, Friedrich von Metzler, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 27 515 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

1 0

log server duplication
by freebsd＠tango.lu 15 Nov '19

15 Nov '19

Hello, I have a syslogNG based siem setup with customized rules like: options { use_dns(no); use_fqdn(no); check_hostname(no); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); create_dirs(yes); normalize_hostnames(yes); keep_hostname(yes); # disable stats stats_freq(0); }; destination d_net_auth { file("/var/log/corporate/$HOST_FROM/auth.log"); }; ... These settings will not do dns resolution will result that when hosts sending their logs into this SIEM directories will be created by their IP addresses where the logs go. I would like to replicate this server on a second location without using brute methods like rsyncing the whole directory structure daily. I have configured syslogng to keep forwarding the logs to a remote destination which works fine however I can't select the messages based on the same criteria on the new log server because if I use the same config everything will originate from the IP for logserver 1. I need IP based directories on the second loghost as well, everything to be identical. I'm using syslogng 3.12. Is there a workaround for this? Thanks

3 4