- syslog-ng - lists.balabit.hu

Insider 2020-06: edge; log management layer; WSL;
by Peter Czanik (pczanik) 11 Jun '20

11 Jun '20

Dear syslog-ng users, This is the 82nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Syslog-ng on the edge --------------------- After many years of pushing all computing from on-site to the cloud or huge data centers, there is a new trend: edge computing. There can be many reasons, legal or practical, why data should be processed locally instead of being sent to a central location as soon as it is created. Edge computing was a central theme of the recently held Red Hat Summit. Luckily syslog-ng is well prepared for this use case right from the beginning. While most people only know that syslog-ng can act as a client or a server, it can also collect, process and forward log messages. In syslog-ng terminology it is called a relay, but on the edge you might want to combine server and a relay functionality into one. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-on-the-edge Creating a dedicated log management layer ----------------------------------------- Event logging is a central source of information both for IT security and operations, but different teams use different tools to collect and analyze log messages. The same log message is often collected by multiple applications. Having each team using different tools is complex, inefficient and makes systems less secure. Using a single application to create a dedicated log management layer independent of analytics instead, however, has multiple benefits. Using syslog-ng is a lot more flexible than most log aggregation tools provided by log analytics vendors. This is one of the reasons why my talks and blogs focused on how to make your life easier using its technical advantages. Of course, I am aware of the direct financial benefits as well. If you are interested in that part, talk to my colleagues on the business side. They can help you to calculate how much you can save on your SIEM licenses when syslog-ng collects log messages and ensures that only relevant messages reach your SIEM and only at a predicatively low message rate. You can learn more about this use case on our Optimizing SIEM page. In this blog, I will focus on a third aspect: simplifying complexity. This was the focus of many of my conference discussions before the COVID-19 pandemic. If we think a bit more about it, we can see that this is not really a third aspect, but a combination of the previous two instead. Using the flexibility of syslog-ng, we create a dedicated log management layer in front of different log analytics solutions. By reducing complexity, we can save in many ways: on computing and human resources, and on licensing when using commercial tools for log analysis as well. https://www.syslog-ng.com/community/b/blog/posts/creating-a-dedicated-log-m… I will cover this topic more in depth in my upcoming talk at the Pass the SALT conference: https://pass-the-salt.org/ Using syslog-ng in WSL ---------------------- Windows Subsystem for Linux (WSL) is an optional feature of Windows 10 for developers who want the power of Linux (especially the Linux shell) on their Windows desktops. Of course, it is more than just a shell: you can easily install and run any command line applications (but not GUI). As a Linux desktop user, I do not need WSL to access a Linux shell, but as I am often asked how syslog-ng runs in WSL, I finally gave it a try. The recurring questions are if syslog-ng runs at all in WSL and what the performance compared to syslog-ng installed on Linux is. As I run openSUSE Leap 15.1 as my main operating system on my laptop, I used that in WSL as well. I tested not just WSL 1, which has been generally available for years, but also the upcoming WSL 2, which brings tons of performance improvements. As WSL 2 involves virtualization, I also tested syslog-ng in Vmware Workstation running on Windows. In all cases, I used the latest syslog-ng 3.26 from my unofficial syslog-ng repository for openSUSE and a minimally modified syslog-ng.conf to enable the network source. Benchmarking was done both from localhost and from a small Xeon server on the local network, attached through Gigabit Ethernet. https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-in-wsl CONFERENCES * ”Creating a dedicated log management layer” talk at the virtual Pass the SALT conference: https://pass-the-salt.org/ WEBINARS * Learn how to do custom HTTP REST API Authentication with syslog-ng: https://www.syslog-ng.com/event/customize-http-rest-apis-authentication-wit… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Poll: Do you use any of the Java-based destination drivers of syslog-ng?
by Peter Czanik (pczanik) 10 Jun '20

10 Jun '20

Hi, Just posted a poll on Twitter, but answers are also very welcome here. Do you use any of the Java-based destination drivers of syslog-ng? HDFS, Kafka (not the C implementation) or Elasticsearch (not the elasticsearch-http() version). Or do you use the syslog-ng Java bindings with your own custom Java module? Let as know here or on Twitter at https://twitter.com/PCzanik/status/1270700394059104263 Also, please RT the above tweet to reach more syslog-ng users! Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Fwd: Memleak in syslog-ng v3.19.1
by Vidya Balakrishnan 09 Jun '20

09 Jun '20

Greetings, This mail is regarding Memory leak in Syslog-ng v3.19.1. The process gets launched at 26MB and then grows to 37MB or even upto 45MB. Pasting the configurations used below: @version: 3.19 @include "scl.conf" source s_local { #system(); #internal(); syslog(port(514) transport("udp")); }; rewrite tag_SN { set("ABC-XXXXXXX", value("HOST")); }; destination d_Log_Server { network("abcd.com" port(514) transport(tls) tls( ca-dir("/etc/ssl") peer-verify("required-trusted") ) time-zone("+00:00") template("${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}.${R_MSEC}${TZOFFSET} ${HOST} ${MSGHDR}${MESSAGE}\n") ); }; log { source(s_local); rewrite(tag_SN); destination(d_Log_Server); }; Also, tried debugging using valgrind but did not resolve the issue. ==19671== Memcheck, a memory error detector ==19671== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==19671== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==19671== Command: /sbin/syslog-ng --cfgfile=/var/syslog-ng.conf ==19671== Parent PID: 2190 ==19671== ==19671== Conditional jump or move depends on uninitialised value(s) ==19671== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19671== by 0x4007F33: expand_dynamic_string_token (in /lib/ld-linux.so.3) ==19671== ==19671== Conditional jump or move depends on uninitialised value(s) ==19671== at 0x401A8D8: index (in /lib/ld-linux.so.3) ==19671== by 0x4007F33: expand_dynamic_string_token (in /lib/ld-linux.so.3) ==19671== ==19671== Conditional jump or move depends on uninitialised value(s) ==19671== at 0x4007F38: expand_dynamic_string_token (in /lib/ld-linux.so.3) ==19671== disInstr(arm): unhandled instruction: 0xEC510F1E cond=14(0xE) 27:20=197(0xC5) 4:4=1 3:0=14(0xE) ==19671== valgrind: Unrecognised instruction at address 0x4cd6e68. ==19671== at 0x4CD6E68: _armv7_tick (in /lib/libcrypto.so.1.1) ==19671== Your program just tried to execute an instruction that Valgrind ==19671== did not recognise. There are two possible reasons for this. ==19671== 1. Your program has a bug and erroneously jumped to a non-code ==19671== location. If you are running Memcheck and you just saw a ==19671== warning about a bad jump, it's probably your program's fault. ==19671== 2. The instruction is legitimate but Valgrind doesn't handle it, ==19671== i.e. it's Valgrind's fault. If you think this is the case or ==19671== you are not sure, please let us know and we'll try to fix it. ==19671== Either way, Valgrind will now raise a SIGILL signal which will ==19671== probably kill your program. ==19700== Invalid read of size 4 ==19700== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19700== by 0x40145BB: dl_open_worker (in /lib/ld-linux.so.3) ==19700== Address 0x4f86bb4 is 3 bytes after a block of size 41 alloc'd ==19700== at 0x4847654: malloc (vg_replace_malloc.c:299) ==19700== by 0x49B3BC3: g_malloc (gmem.c:94) ==19700== by 0x49CEA17: g_strdup (gstrfuncs.c:363) ==19700== by 0x49417BF: g_module_open (gmodule.c:543) ==19700== by 0x489AFDB: plugin_dlopen_module_as_filename (plugin.c:252) ==19700== by 0x489B8B3: plugin_dlopen_module_as_dir_and_filename (plugin.c:271) ==19700== by 0x489B8B3: plugin_load_candidate_modules (plugin.c:457) ==19700== by 0x4887017: cfg_lexer_preprocess (cfg-lexer.c:1068) ==19700== by 0x4887017: cfg_lexer_lex (cfg-lexer.c:1110) ==19700== by 0x48A876F: main_parse (cfg-grammar.c:3143) ==19700== by 0x4887D8F: cfg_parser_parse (cfg-parser.c:369) ==19700== by 0x48845BF: cfg_run_parser (cfg.c:504) ==19700== by 0x48846B3: cfg_read_config (cfg.c:556) ==19700== by 0x489766B: main_loop_read_and_init_config (mainloop.c:552) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19700== by 0x40145BB: dl_open_worker (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D8: index (in /lib/ld-linux.so.3) ==19700== by 0x40145BB: dl_open_worker (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x40145C0: dl_open_worker (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401AD9C: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Invalid read of size 4 ==19700== at 0x401AC60: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== Address 0x5185254 is 0 bytes after a block of size 20 alloc'd ==19700== at 0x4849EB4: realloc (vg_replace_malloc.c:785) ==19700== by 0x49B3C9B: g_realloc (gmem.c:159) ==19700== by 0x49FCAF3: _g_gnulib_vasnprintf (vasnprintf.c:5533) ==19700== by 0x49FD263: _g_gnulib_vasprintf (printf.c:142) ==19700== by 0x49F6B0B: g_vasprintf (gprintf.c:310) ==19700== by 0x49CEB7F: g_strdup_vprintf (gstrfuncs.c:514) ==19700== by 0x49CEBAB: g_strdup_printf (gstrfuncs.c:540) ==19700== by 0x489AF7F: plugin_get_module_init_name (plugin.c:234) ==19700== by 0x489B363: plugin_load_module (plugin.c:364) ==19700== by 0x48ABCAB: pragma_parse (pragma-grammar.y:443) ==19700== by 0x4887D8F: cfg_parser_parse (cfg-parser.c:369) ==19700== by 0x4886E9B: cfg_lexer_parse_pragma (cfg-lexer.c:996) ==19700== by 0x4886E9B: cfg_lexer_preprocess (cfg-lexer.c:1040) ==19700== by 0x4886E9B: cfg_lexer_lex (cfg-lexer.c:1110) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401AD80: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Use of uninitialised value of size 4 ==19700== at 0x401ACB8: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Invalid read of size 4 ==19700== at 0x401ADAC: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== Address 0x533bdac is 3 bytes after a block of size 25 alloc'd ==19700== at 0x4849EB4: realloc (vg_replace_malloc.c:785) ==19700== by 0x49B3C9B: g_realloc (gmem.c:159) ==19700== by 0x49FCAF3: _g_gnulib_vasnprintf (vasnprintf.c:5533) ==19700== by 0x49FD263: _g_gnulib_vasprintf (printf.c:142) ==19700== by 0x49F6B0B: g_vasprintf (gprintf.c:310) ==19700== by 0x49CEB7F: g_strdup_vprintf (gstrfuncs.c:514) ==19700== by 0x49CEBAB: g_strdup_printf (gstrfuncs.c:540) ==19700== by 0x489AF7F: plugin_get_module_init_name (plugin.c:234) ==19700== by 0x489B363: plugin_load_module (plugin.c:364) ==19700== by 0x489B6BB: plugin_find (plugin.c:199) ==19700== by 0x489921B: msg_format_options_init (msg-format.c:81) ==19700== by 0x4891A9B: log_reader_options_init (logreader.c:728) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401ADC4: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19700== by 0x4008BFB: _dl_map_object (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D8: index (in /lib/ld-linux.so.3) ==19700== by 0x4008BFB: _dl_map_object (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x4008C00: _dl_map_object (in /lib/ld-linux.so.3) ==19700== ==19700== Syscall param epoll_pwait(sigmask) points to unaddressable byte(s) ==19700== at 0x4BCC73C: epoll_pwait (in /lib/libc.so.6) ==19700== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==19700== ==19671== ==19671== HEAP SUMMARY: ==19671== in use at exit: 11,980 bytes in 55 blocks ==19671== total heap usage: 179 allocs, 124 frees, 49,157 bytes allocated ==19671== ==19671== 419 (44 direct, 375 indirect) bytes in 1 blocks are definitely lost in loss record 44 of 46 ==19671== at 0x4847654: malloc (vg_replace_malloc.c:299) ==19671== by 0x49B3BC3: g_malloc (gmem.c:94) ==19671== by 0x488BD07: g_process_set_argv_space (gprocess.c:569) ==19671== by 0x1127B: main (main.c:216) ==19671== ==19671== LEAK SUMMARY: ==19671== definitely lost: 44 bytes in 1 blocks ==19671== indirectly lost: 375 bytes in 10 blocks ==19671== possibly lost: 0 bytes in 0 blocks ==19671== still reachable: 11,561 bytes in 44 blocks ==19671== suppressed: 0 bytes in 0 blocks ==19671== Reachable blocks (those to which a pointer was found) are not shown. ==19671== To see them, rerun with: --leak-check=full --show-leak-kinds=all The memory usage keeps increasing. There are no local disks used to store the logs messages. It is directly sent to the Graylog server. Could you please take a look at this issue and help resolving the same. Regards, Vidya Balakrishnan

2 4

my syslog-ng-stable repos updated to 3.27
by Peter Czanik (pczanik) 08 Jun '20

08 Jun '20

Hi, My unofficial syslog-ng-stable repos for openSUSE/SLES and Fedora/RHEL/CentOS are now updated to syslog-ng 3.27. As you might recall, version 3.26 was skipped, as it arrived at a time when nobody was touching systems (beginning of WFH). 3.27 was relesaed a while ago. It seems to run fine everywhere, so I found it as a safe choice for the stable repo. (side note: thanks to those who reminded me to update the repos. The one drawback that I don't update immediately after release, that sometimes I forget it completely... 🙂 ). As usual: let me know, if you run into any problems with the packages. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] Message and Header are being split incorrectly
by Sass, Fabian 05 Jun '20

05 Jun '20

Hi Peter, thanks a lot, this resolved the issue without having to use hacky templates. Still it bugs me out that rewriting $PROGRAM didn't work. Any idea why? According to the other list posting it should've. Cheers, Fabian Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Peter Czanik (pczanik) Gesendet: Freitag, 5. Juni 2020 08:36 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Message and Header are being split incorrectly Hi, FreeBSD 12.1 changed from the legacy syslog protocol to RFC 5424 format. When you use the system() source for local logs, this is handled automagically. But you can also fix it by hand: https://github.com/syslog-ng/syslog-ng/issues/2428 Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com<mailto:peter.czanik@oneidentity.com>> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Sass, Fabian <Fabian.Sass(a)f-i-ts.de<mailto:Fabian.Sass@f-i-ts.de>> Sent: Friday, June 5, 2020 08:28 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Message and Header are being split incorrectly CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers. Jun 5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root) Jun 5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro. $MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line. However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update. The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost. The behavior seems to be analog to this bug: https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.bal…> As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test): @version: 3.5 # # options # options { mark_freq(3600); flush_lines(0); dir_perm(0640); chain_hostnames(off); keep_hostname(yes); create_dirs(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(3600); }; source s_all { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); }; # # rewrite since syslog message splitting is broken since update to freebsd12... # rewrite r_msg { set( "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*")) ); subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE")); }; # # destinations # destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); }; destination d_test2 { file("/var/log/fabian_messages2"); }; destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); }; destination d_loghost { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); }; # # # logging # log { source(s_all); rewrite(r_msg); destination(d_messages); }; log { source(s_all); rewrite(r_msg); destination(d_loghost); }; log { source(s_all); rewrite(r_msg); destination(d_test); }; log { source(s_all); destination(d_test2); }; Am I missing something here, or is syslog-ng somehow behaving unintended? Fabian

1 0

Message and Header are being split incorrectly
by Sass, Fabian 05 Jun '20

05 Jun '20

Hi, since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers. Jun 5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root) Jun 5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro. $MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line. However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update. The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost. The behavior seems to be analog to this bug: https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test): @version: 3.5 # # options # options { mark_freq(3600); flush_lines(0); dir_perm(0640); chain_hostnames(off); keep_hostname(yes); create_dirs(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(3600); }; source s_all { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); }; # # rewrite since syslog message splitting is broken since update to freebsd12... # rewrite r_msg { set( "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*")) ); subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE")); }; # # destinations # destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); }; destination d_test2 { file("/var/log/fabian_messages2"); }; destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); }; destination d_loghost { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); }; # # # logging # log { source(s_all); rewrite(r_msg); destination(d_messages); }; log { source(s_all); rewrite(r_msg); destination(d_loghost); }; log { source(s_all); rewrite(r_msg); destination(d_test); }; log { source(s_all); destination(d_test2); }; Am I missing something here, or is syslog-ng somehow behaving unintended? Fabian

2 1

build on MacOS
by Pal, Laszlo 01 Jun '20

01 Jun '20

Hi, For quite some time I'm trying to build syslog-ng on my MacBook and I'm always getting the following error. Is there any Mac user who can help me with this? Is there a plan to make syslog-ng available again in homebrew? ld: warning: ignoring file > ./modules/dbparser/.libs/libsyslog-ng-patterndb.a, building for > macOS-x86_64 but attempting to link with file built for unknown-unsupported > file format ( 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A 0x2F 0x20 0x20 0x20 > 0x20 0x20 0x20 0x20 ) > Undefined symbols for architecture x86_64: > "_pattern_db_debug_ruleset", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pattern_db_expire_state", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pattern_db_free", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_get_ruleset", referenced from: > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_global_init", referenced from: > _main in pdbtool-pdbtool.o > "_pattern_db_new", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_process", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_reload_ruleset", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > "_pattern_db_set_emit_func", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pdb_example_free", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_file_validate", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_get_filenames", referenced from: > _pdbtool_merge in pdbtool-pdbtool.o > "_pdb_rule_get_name", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_rule_set_load", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > _pdbtool_dictionary in pdbtool-pdbtool.o > "_pdb_sort_filenames", referenced from: > _pdbtool_merge in pdbtool-pdbtool.o > "_ptz_find_clusters", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_free", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_load_file", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_new", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_print_patterndb", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_r_find_all_applicable_nodes", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_r_find_node", referenced from: > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > _pdbtool_dictionary in pdbtool-pdbtool.o > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [modules/dbparser/pdbtool/pdbtool] Error 1 > make[2]: *** Waiting for unfinished jobs.... > ld: warning: ignoring file > ./modules/diskq/.libs/libsyslog-ng-disk-buffer.a, building for macOS-x86_64 > but attempting to link with file built for macOS-x86_64 > Undefined symbols for architecture x86_64: > "_log_queue_disk_load_queue", referenced from: > _open_queue in dqtool.o > "_log_queue_disk_non_reliable_new", referenced from: > _open_queue in dqtool.o > "_log_queue_disk_reliable_new", referenced from: > _open_queue in dqtool.o > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) Thanks L

1 0

How to ensure that only my hosts log to my syslog server?
by Evan Rempel 29 May '20

29 May '20

We are starting to explore laptop logging which means that I have to open up firewalls to public networks as the laptops are moved around. Is there a way to ensure that only computers configured by my organization are able to connect to or send logs to my log server? I looked at "Mutual authentication using TLS" but if I understand that correctly the client is required to have a IP/hostname that matches the CN of the certificate. I couldn't find other information but perhaps I am searching for the wrong terms. -- Evan

2 2

Logs not making it to Splunk, high scratch buffers count
by Faine, Mark R. (MSFC-IS40)[NICS] 28 May '20

28 May '20

We've recently been seeing interruptions or delays to logs going into Splunk. I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing. Here are the stats: SourceName;SourceId;SourceInstance;State;Type;Number destination;d_spol;;a;processed;0 src.internal;s_sys#2;;a;processed;384 src.internal;s_sys#2;;a;stamp;1590690126 center;;received;a;processed;1925133 destination;d_mesg;;a;processed;384 destination;d_mail;;a;processed;2 destination;d_auth;;a;processed;3 destination;d_mlal;;a;processed;0 center;;queued;a;processed;1962257 source;pan_splunk;;a;processed;1924743 destination;d_file;;a;processed;1561759 destination;d_cron;;a;processed;1 global;scratch_buffers_count;;a;queued;150323855375 destination;#anon-destination0;;a;processed;362983 global;payload_reallocs;;a;processed;5 global;sdata_updates;;a;processed;24 global;scratch_buffers_bytes;;a;queued;1536 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;0 source;s_sys;;a;processed;390 global;msg_clones;;a;processed;4662233 destination;d_file_no_location;;a;processed;37125 global;internal_queue_length;;a;processed;0 Here is the source configuration: source pan_splunk { network(port(514) transport("tcp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(port(514) transport("udp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(transport("tls") port(10514) max-connections(100) flags(syslog-protocol) log-iw-size(100000) log-fetch-limit(100) log-msg-size(10485760) tls(peer-verify(optional-untrusted) key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; Here is the destination configuration destination d_file { file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log" create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750) owner("splunk") group("splunk") perm(0640) log-fifo-size(1000) disk-buffer( mem-buf-size(10000) disk-buf-size(2000000000) reliable(yes) dir("/var/log/remote/disk-buffer_dfile") ) flags("threaded", "no-multi-line") ); }; This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference. I would welcome any help you could provide. Thanks, -Mark

1 0

[SOLVED] Strange issue when forwarding using LEEF template
by Pal, Laszlo 28 May '20

28 May '20

Hi, I have a strange issue with templates... Maybe some stupid issue on my side, maybe a bug I don't know *<spoiler> Solutions at the end of this </spoiler>* Here is the scenario Logs coming from Windows using Syslog-ng Agent (syslog destination) Logs received using latest OSE (but same issues at the customer with PE6) Logs must be forwarded to Qradar using a special LEEF template like this (there are also tabs in the template). I've tried both network and syslog destination template t_leefwin { template("<${PRI}>${BSDDATE} ${HOST} LEEF:1.0|Microsoft|Windows|2k8r2|${EVENT_ID}| devTime=${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}GMT${TZOFFSET} devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat=${EVENT_TYPE} sev=${EVENT_LEVEL} resource=${HOST} usrName=${EVENT_USERNAME} application=${EVENT_SOURCE} message=${EVENT_MSG}\n"); }; The forwarded results like this <46> IP-address LEEF:1.0|Microsoft|Windows|2k8r2||devTime=2020-05-28T09:10:02GMT+02:00 devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat= sev= resource=172.17.24.174 usrName= application= message= So, the $macro-s are empty except the one which is not from .sdata (resource) I've created a local json and welf destination and I can see the macros there SDATA="[win(a)18372.4 EVENT_CATEGORY=\"None\" EVENT_FACILITY=\"16\" EVENT_ID=\"4098\" EVENT_LEVEL=\"3\" EVENT_NAME=\"Application\" EVENT_REC_NUM=\"73006\" EVENT_SID=\"S-1-5-18\" EVENT_SID_TYPE=\"User\" EVENT_SOURCE=\"Group Policy Services\" EVENT_TYPE=\"Figyelmeztetés\" EVENT_HOST=\"hostname\" EVENT_USERNAME=\"NT AUTHORITY\\\\SYSTEM\" I remember I used this method years ago to forward logs from syslog-ng agent to Qradar with proper LEEF format and it was ok... but now, some small thing is missing Versions syslog-ng 3 (3.25.1) Config version: 3.25 Installer-Version: 3.25.1 But, we have a similar issue with PE6 at the customer Any idea? --- so after playing with the welf output a bit, I figured out the short version of the macro names are not longer working somehow, but the long ones are ok like this template t_leefwin { template("<${PRI}>${BSDDATE} ${HOST} LEEF:1.0|Microsoft|Windows|2k8r2|${.SDATA.win@18372.4.EVENT_ID }|devTime=${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}GMT${TZOFFSET} devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat=${.SDATA.win(a)18372.4.EVENT_TYPE} sev=${.SDATA.win(a)18372.4.EVENT_LEVEL} resource=${SDATA.win(a)18372.4.EVENT_HOST} usrName=${.SDATA.win(a)18372.4.EVENT_USERNAME} application=${.SDATA.win(a)18372.4.EVENT_SOURCE} message=${MESSAGE}\n"); }; Thanks L:

2 2