- syslog-ng - lists.balabit.hu

Fwd: Memleak in syslog-ng v3.19.1
by Vidya Balakrishnan 09 Jun '20

09 Jun '20

Greetings, This mail is regarding Memory leak in Syslog-ng v3.19.1. The process gets launched at 26MB and then grows to 37MB or even upto 45MB. Pasting the configurations used below: @version: 3.19 @include "scl.conf" source s_local { #system(); #internal(); syslog(port(514) transport("udp")); }; rewrite tag_SN { set("ABC-XXXXXXX", value("HOST")); }; destination d_Log_Server { network("abcd.com" port(514) transport(tls) tls( ca-dir("/etc/ssl") peer-verify("required-trusted") ) time-zone("+00:00") template("${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}.${R_MSEC}${TZOFFSET} ${HOST} ${MSGHDR}${MESSAGE}\n") ); }; log { source(s_local); rewrite(tag_SN); destination(d_Log_Server); }; Also, tried debugging using valgrind but did not resolve the issue. ==19671== Memcheck, a memory error detector ==19671== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==19671== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==19671== Command: /sbin/syslog-ng --cfgfile=/var/syslog-ng.conf ==19671== Parent PID: 2190 ==19671== ==19671== Conditional jump or move depends on uninitialised value(s) ==19671== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19671== by 0x4007F33: expand_dynamic_string_token (in /lib/ld-linux.so.3) ==19671== ==19671== Conditional jump or move depends on uninitialised value(s) ==19671== at 0x401A8D8: index (in /lib/ld-linux.so.3) ==19671== by 0x4007F33: expand_dynamic_string_token (in /lib/ld-linux.so.3) ==19671== ==19671== Conditional jump or move depends on uninitialised value(s) ==19671== at 0x4007F38: expand_dynamic_string_token (in /lib/ld-linux.so.3) ==19671== disInstr(arm): unhandled instruction: 0xEC510F1E cond=14(0xE) 27:20=197(0xC5) 4:4=1 3:0=14(0xE) ==19671== valgrind: Unrecognised instruction at address 0x4cd6e68. ==19671== at 0x4CD6E68: _armv7_tick (in /lib/libcrypto.so.1.1) ==19671== Your program just tried to execute an instruction that Valgrind ==19671== did not recognise. There are two possible reasons for this. ==19671== 1. Your program has a bug and erroneously jumped to a non-code ==19671== location. If you are running Memcheck and you just saw a ==19671== warning about a bad jump, it's probably your program's fault. ==19671== 2. The instruction is legitimate but Valgrind doesn't handle it, ==19671== i.e. it's Valgrind's fault. If you think this is the case or ==19671== you are not sure, please let us know and we'll try to fix it. ==19671== Either way, Valgrind will now raise a SIGILL signal which will ==19671== probably kill your program. ==19700== Invalid read of size 4 ==19700== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19700== by 0x40145BB: dl_open_worker (in /lib/ld-linux.so.3) ==19700== Address 0x4f86bb4 is 3 bytes after a block of size 41 alloc'd ==19700== at 0x4847654: malloc (vg_replace_malloc.c:299) ==19700== by 0x49B3BC3: g_malloc (gmem.c:94) ==19700== by 0x49CEA17: g_strdup (gstrfuncs.c:363) ==19700== by 0x49417BF: g_module_open (gmodule.c:543) ==19700== by 0x489AFDB: plugin_dlopen_module_as_filename (plugin.c:252) ==19700== by 0x489B8B3: plugin_dlopen_module_as_dir_and_filename (plugin.c:271) ==19700== by 0x489B8B3: plugin_load_candidate_modules (plugin.c:457) ==19700== by 0x4887017: cfg_lexer_preprocess (cfg-lexer.c:1068) ==19700== by 0x4887017: cfg_lexer_lex (cfg-lexer.c:1110) ==19700== by 0x48A876F: main_parse (cfg-grammar.c:3143) ==19700== by 0x4887D8F: cfg_parser_parse (cfg-parser.c:369) ==19700== by 0x48845BF: cfg_run_parser (cfg.c:504) ==19700== by 0x48846B3: cfg_read_config (cfg.c:556) ==19700== by 0x489766B: main_loop_read_and_init_config (mainloop.c:552) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19700== by 0x40145BB: dl_open_worker (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D8: index (in /lib/ld-linux.so.3) ==19700== by 0x40145BB: dl_open_worker (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x40145C0: dl_open_worker (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401AD9C: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Invalid read of size 4 ==19700== at 0x401AC60: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== Address 0x5185254 is 0 bytes after a block of size 20 alloc'd ==19700== at 0x4849EB4: realloc (vg_replace_malloc.c:785) ==19700== by 0x49B3C9B: g_realloc (gmem.c:159) ==19700== by 0x49FCAF3: _g_gnulib_vasnprintf (vasnprintf.c:5533) ==19700== by 0x49FD263: _g_gnulib_vasprintf (printf.c:142) ==19700== by 0x49F6B0B: g_vasprintf (gprintf.c:310) ==19700== by 0x49CEB7F: g_strdup_vprintf (gstrfuncs.c:514) ==19700== by 0x49CEBAB: g_strdup_printf (gstrfuncs.c:540) ==19700== by 0x489AF7F: plugin_get_module_init_name (plugin.c:234) ==19700== by 0x489B363: plugin_load_module (plugin.c:364) ==19700== by 0x48ABCAB: pragma_parse (pragma-grammar.y:443) ==19700== by 0x4887D8F: cfg_parser_parse (cfg-parser.c:369) ==19700== by 0x4886E9B: cfg_lexer_parse_pragma (cfg-lexer.c:996) ==19700== by 0x4886E9B: cfg_lexer_preprocess (cfg-lexer.c:1040) ==19700== by 0x4886E9B: cfg_lexer_lex (cfg-lexer.c:1110) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401AD80: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Use of uninitialised value of size 4 ==19700== at 0x401ACB8: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Invalid read of size 4 ==19700== at 0x401ADAC: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== Address 0x533bdac is 3 bytes after a block of size 25 alloc'd ==19700== at 0x4849EB4: realloc (vg_replace_malloc.c:785) ==19700== by 0x49B3C9B: g_realloc (gmem.c:159) ==19700== by 0x49FCAF3: _g_gnulib_vasnprintf (vasnprintf.c:5533) ==19700== by 0x49FD263: _g_gnulib_vasprintf (printf.c:142) ==19700== by 0x49F6B0B: g_vasprintf (gprintf.c:310) ==19700== by 0x49CEB7F: g_strdup_vprintf (gstrfuncs.c:514) ==19700== by 0x49CEBAB: g_strdup_printf (gstrfuncs.c:540) ==19700== by 0x489AF7F: plugin_get_module_init_name (plugin.c:234) ==19700== by 0x489B363: plugin_load_module (plugin.c:364) ==19700== by 0x489B6BB: plugin_find (plugin.c:199) ==19700== by 0x489921B: msg_format_options_init (msg-format.c:81) ==19700== by 0x4891A9B: log_reader_options_init (logreader.c:728) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401ADC4: strcpy (in /lib/ld-linux.so.3) ==19700== by 0x400A8D3: _dl_lookup_symbol_x (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D4: index (in /lib/ld-linux.so.3) ==19700== by 0x4008BFB: _dl_map_object (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x401A8D8: index (in /lib/ld-linux.so.3) ==19700== by 0x4008BFB: _dl_map_object (in /lib/ld-linux.so.3) ==19700== ==19700== Conditional jump or move depends on uninitialised value(s) ==19700== at 0x4008C00: _dl_map_object (in /lib/ld-linux.so.3) ==19700== ==19700== Syscall param epoll_pwait(sigmask) points to unaddressable byte(s) ==19700== at 0x4BCC73C: epoll_pwait (in /lib/libc.so.6) ==19700== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==19700== ==19671== ==19671== HEAP SUMMARY: ==19671== in use at exit: 11,980 bytes in 55 blocks ==19671== total heap usage: 179 allocs, 124 frees, 49,157 bytes allocated ==19671== ==19671== 419 (44 direct, 375 indirect) bytes in 1 blocks are definitely lost in loss record 44 of 46 ==19671== at 0x4847654: malloc (vg_replace_malloc.c:299) ==19671== by 0x49B3BC3: g_malloc (gmem.c:94) ==19671== by 0x488BD07: g_process_set_argv_space (gprocess.c:569) ==19671== by 0x1127B: main (main.c:216) ==19671== ==19671== LEAK SUMMARY: ==19671== definitely lost: 44 bytes in 1 blocks ==19671== indirectly lost: 375 bytes in 10 blocks ==19671== possibly lost: 0 bytes in 0 blocks ==19671== still reachable: 11,561 bytes in 44 blocks ==19671== suppressed: 0 bytes in 0 blocks ==19671== Reachable blocks (those to which a pointer was found) are not shown. ==19671== To see them, rerun with: --leak-check=full --show-leak-kinds=all The memory usage keeps increasing. There are no local disks used to store the logs messages. It is directly sent to the Graylog server. Could you please take a look at this issue and help resolving the same. Regards, Vidya Balakrishnan

2 4

my syslog-ng-stable repos updated to 3.27
by Peter Czanik (pczanik) 08 Jun '20

08 Jun '20

Hi, My unofficial syslog-ng-stable repos for openSUSE/SLES and Fedora/RHEL/CentOS are now updated to syslog-ng 3.27. As you might recall, version 3.26 was skipped, as it arrived at a time when nobody was touching systems (beginning of WFH). 3.27 was relesaed a while ago. It seems to run fine everywhere, so I found it as a safe choice for the stable repo. (side note: thanks to those who reminded me to update the repos. The one drawback that I don't update immediately after release, that sometimes I forget it completely... 🙂 ). As usual: let me know, if you run into any problems with the packages. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] Message and Header are being split incorrectly
by Sass, Fabian 05 Jun '20

05 Jun '20

Hi Peter, thanks a lot, this resolved the issue without having to use hacky templates. Still it bugs me out that rewriting $PROGRAM didn't work. Any idea why? According to the other list posting it should've. Cheers, Fabian Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Peter Czanik (pczanik) Gesendet: Freitag, 5. Juni 2020 08:36 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Message and Header are being split incorrectly Hi, FreeBSD 12.1 changed from the legacy syslog protocol to RFC 5424 format. When you use the system() source for local logs, this is handled automagically. But you can also fix it by hand: https://github.com/syslog-ng/syslog-ng/issues/2428 Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com<mailto:peter.czanik@oneidentity.com>> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Sass, Fabian <Fabian.Sass(a)f-i-ts.de<mailto:Fabian.Sass@f-i-ts.de>> Sent: Friday, June 5, 2020 08:28 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Message and Header are being split incorrectly CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers. Jun 5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root) Jun 5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro. $MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line. However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update. The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost. The behavior seems to be analog to this bug: https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.bal…> As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test): @version: 3.5 # # options # options { mark_freq(3600); flush_lines(0); dir_perm(0640); chain_hostnames(off); keep_hostname(yes); create_dirs(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(3600); }; source s_all { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); }; # # rewrite since syslog message splitting is broken since update to freebsd12... # rewrite r_msg { set( "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*")) ); subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE")); }; # # destinations # destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); }; destination d_test2 { file("/var/log/fabian_messages2"); }; destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); }; destination d_loghost { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); }; # # # logging # log { source(s_all); rewrite(r_msg); destination(d_messages); }; log { source(s_all); rewrite(r_msg); destination(d_loghost); }; log { source(s_all); rewrite(r_msg); destination(d_test); }; log { source(s_all); destination(d_test2); }; Am I missing something here, or is syslog-ng somehow behaving unintended? Fabian

1 0

Message and Header are being split incorrectly
by Sass, Fabian 05 Jun '20

05 Jun '20

Hi, since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers. Jun 5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root) Jun 5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro. $MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line. However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update. The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost. The behavior seems to be analog to this bug: https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test): @version: 3.5 # # options # options { mark_freq(3600); flush_lines(0); dir_perm(0640); chain_hostnames(off); keep_hostname(yes); create_dirs(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(3600); }; source s_all { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); }; # # rewrite since syslog message splitting is broken since update to freebsd12... # rewrite r_msg { set( "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*")) ); subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE")); }; # # destinations # destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); }; destination d_test2 { file("/var/log/fabian_messages2"); }; destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); }; destination d_loghost { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); }; # # # logging # log { source(s_all); rewrite(r_msg); destination(d_messages); }; log { source(s_all); rewrite(r_msg); destination(d_loghost); }; log { source(s_all); rewrite(r_msg); destination(d_test); }; log { source(s_all); destination(d_test2); }; Am I missing something here, or is syslog-ng somehow behaving unintended? Fabian

2 1

build on MacOS
by Pal, Laszlo 01 Jun '20

01 Jun '20

Hi, For quite some time I'm trying to build syslog-ng on my MacBook and I'm always getting the following error. Is there any Mac user who can help me with this? Is there a plan to make syslog-ng available again in homebrew? ld: warning: ignoring file > ./modules/dbparser/.libs/libsyslog-ng-patterndb.a, building for > macOS-x86_64 but attempting to link with file built for unknown-unsupported > file format ( 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A 0x2F 0x20 0x20 0x20 > 0x20 0x20 0x20 0x20 ) > Undefined symbols for architecture x86_64: > "_pattern_db_debug_ruleset", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pattern_db_expire_state", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pattern_db_free", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_get_ruleset", referenced from: > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_global_init", referenced from: > _main in pdbtool-pdbtool.o > "_pattern_db_new", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_process", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_reload_ruleset", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > "_pattern_db_set_emit_func", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pdb_example_free", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_file_validate", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_get_filenames", referenced from: > _pdbtool_merge in pdbtool-pdbtool.o > "_pdb_rule_get_name", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_rule_set_load", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > _pdbtool_dictionary in pdbtool-pdbtool.o > "_pdb_sort_filenames", referenced from: > _pdbtool_merge in pdbtool-pdbtool.o > "_ptz_find_clusters", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_free", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_load_file", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_new", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_print_patterndb", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_r_find_all_applicable_nodes", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_r_find_node", referenced from: > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > _pdbtool_dictionary in pdbtool-pdbtool.o > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [modules/dbparser/pdbtool/pdbtool] Error 1 > make[2]: *** Waiting for unfinished jobs.... > ld: warning: ignoring file > ./modules/diskq/.libs/libsyslog-ng-disk-buffer.a, building for macOS-x86_64 > but attempting to link with file built for macOS-x86_64 > Undefined symbols for architecture x86_64: > "_log_queue_disk_load_queue", referenced from: > _open_queue in dqtool.o > "_log_queue_disk_non_reliable_new", referenced from: > _open_queue in dqtool.o > "_log_queue_disk_reliable_new", referenced from: > _open_queue in dqtool.o > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) Thanks L

1 0

How to ensure that only my hosts log to my syslog server?
by Evan Rempel 29 May '20

29 May '20

We are starting to explore laptop logging which means that I have to open up firewalls to public networks as the laptops are moved around. Is there a way to ensure that only computers configured by my organization are able to connect to or send logs to my log server? I looked at "Mutual authentication using TLS" but if I understand that correctly the client is required to have a IP/hostname that matches the CN of the certificate. I couldn't find other information but perhaps I am searching for the wrong terms. -- Evan

2 2

Logs not making it to Splunk, high scratch buffers count
by Faine, Mark R. (MSFC-IS40)[NICS] 28 May '20

28 May '20

We've recently been seeing interruptions or delays to logs going into Splunk. I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing. Here are the stats: SourceName;SourceId;SourceInstance;State;Type;Number destination;d_spol;;a;processed;0 src.internal;s_sys#2;;a;processed;384 src.internal;s_sys#2;;a;stamp;1590690126 center;;received;a;processed;1925133 destination;d_mesg;;a;processed;384 destination;d_mail;;a;processed;2 destination;d_auth;;a;processed;3 destination;d_mlal;;a;processed;0 center;;queued;a;processed;1962257 source;pan_splunk;;a;processed;1924743 destination;d_file;;a;processed;1561759 destination;d_cron;;a;processed;1 global;scratch_buffers_count;;a;queued;150323855375 destination;#anon-destination0;;a;processed;362983 global;payload_reallocs;;a;processed;5 global;sdata_updates;;a;processed;24 global;scratch_buffers_bytes;;a;queued;1536 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;0 source;s_sys;;a;processed;390 global;msg_clones;;a;processed;4662233 destination;d_file_no_location;;a;processed;37125 global;internal_queue_length;;a;processed;0 Here is the source configuration: source pan_splunk { network(port(514) transport("tcp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(port(514) transport("udp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(transport("tls") port(10514) max-connections(100) flags(syslog-protocol) log-iw-size(100000) log-fetch-limit(100) log-msg-size(10485760) tls(peer-verify(optional-untrusted) key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; Here is the destination configuration destination d_file { file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log" create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750) owner("splunk") group("splunk") perm(0640) log-fifo-size(1000) disk-buffer( mem-buf-size(10000) disk-buf-size(2000000000) reliable(yes) dir("/var/log/remote/disk-buffer_dfile") ) flags("threaded", "no-multi-line") ); }; This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference. I would welcome any help you could provide. Thanks, -Mark

1 0

[SOLVED] Strange issue when forwarding using LEEF template
by Pal, Laszlo 28 May '20

28 May '20

Hi, I have a strange issue with templates... Maybe some stupid issue on my side, maybe a bug I don't know *<spoiler> Solutions at the end of this </spoiler>* Here is the scenario Logs coming from Windows using Syslog-ng Agent (syslog destination) Logs received using latest OSE (but same issues at the customer with PE6) Logs must be forwarded to Qradar using a special LEEF template like this (there are also tabs in the template). I've tried both network and syslog destination template t_leefwin { template("<${PRI}>${BSDDATE} ${HOST} LEEF:1.0|Microsoft|Windows|2k8r2|${EVENT_ID}| devTime=${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}GMT${TZOFFSET} devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat=${EVENT_TYPE} sev=${EVENT_LEVEL} resource=${HOST} usrName=${EVENT_USERNAME} application=${EVENT_SOURCE} message=${EVENT_MSG}\n"); }; The forwarded results like this <46> IP-address LEEF:1.0|Microsoft|Windows|2k8r2||devTime=2020-05-28T09:10:02GMT+02:00 devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat= sev= resource=172.17.24.174 usrName= application= message= So, the $macro-s are empty except the one which is not from .sdata (resource) I've created a local json and welf destination and I can see the macros there SDATA="[win(a)18372.4 EVENT_CATEGORY=\"None\" EVENT_FACILITY=\"16\" EVENT_ID=\"4098\" EVENT_LEVEL=\"3\" EVENT_NAME=\"Application\" EVENT_REC_NUM=\"73006\" EVENT_SID=\"S-1-5-18\" EVENT_SID_TYPE=\"User\" EVENT_SOURCE=\"Group Policy Services\" EVENT_TYPE=\"Figyelmeztetés\" EVENT_HOST=\"hostname\" EVENT_USERNAME=\"NT AUTHORITY\\\\SYSTEM\" I remember I used this method years ago to forward logs from syslog-ng agent to Qradar with proper LEEF format and it was ok... but now, some small thing is missing Versions syslog-ng 3 (3.25.1) Config version: 3.25 Installer-Version: 3.25.1 But, we have a similar issue with PE6 at the customer Any idea? --- so after playing with the welf output a bit, I figured out the short version of the macro names are not longer working somehow, but the long ones are ok like this template t_leefwin { template("<${PRI}>${BSDDATE} ${HOST} LEEF:1.0|Microsoft|Windows|2k8r2|${.SDATA.win@18372.4.EVENT_ID }|devTime=${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}GMT${TZOFFSET} devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat=${.SDATA.win(a)18372.4.EVENT_TYPE} sev=${.SDATA.win(a)18372.4.EVENT_LEVEL} resource=${SDATA.win(a)18372.4.EVENT_HOST} usrName=${.SDATA.win(a)18372.4.EVENT_USERNAME} application=${.SDATA.win(a)18372.4.EVENT_SOURCE} message=${MESSAGE}\n"); }; Thanks L:

2 2

High throughput UDP logging configuration.
by Evan Rempel 28 May '20

28 May '20

I know. There is a ton of information online about this. It shouldn't be that difficult, but I'm still having problems. I have one device with a very high logging rate. Approaching 15K msg/sec (7.5MB/sec). I am running on a 2 socket 8 core/CPU system (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz). Not the latest and greatest, but still fairly quick. All of the storage is on SSD. What I have configured 1. OS net.core.rmem_max = 536870912 2. so-reuseport(1) - and 8 sources on the UDP port 3. so_rcvbuf(67108864) - all 9 udp sources. 4. log_iw_size(2M) 5. log_fetch_limit(20k) 6. log_fifo_size(4M) Flow control is NOT enabled. Monitoring the queued messages inside syslog-ng and they remain near zero. Monitoring udp buffer queues in the OS shows that one stream is still overwhelming syslog-ng's ability to read messages. [~]$ sudo netstat -unlp|egrep -e 'PID|syslog' Proto Recv-Q Send-Q Local Address Foreign Addr State PID/Program name udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 134216320 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng A few things of note. 1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf configured ins 64M in size. Is that because the syslog-ng configuration of so_rcvbuf is in characters but the OS buffer is in bytes? 2. In a 1 second interval "cat /proc/net/udp" shows that one UDP stream dropped 6283 packets. So I'm continually dropping approx 50% of the UDP stream. 3. Increasing the log_iw_size or the log_iw_size actually seems to make things worse. All suggestions that help me understand this and help to minimize the drops are welcome. -- Evan Rempel

2 3

Forwarding to Elastic
by Shawn Taylor 28 May '20

28 May '20

I am running ES/Kibana 6.8.9-1 and am struggling with this issue. https://discuss.elastic.co/t/message-failed-to-find-message-in-kibana-logs/… I have added my index to the *Logs Indices* field in the Logs configuration. When I look at the fields in a document I see a field called MESSAGE, but not message. I do not see a way to add this field in the configuration. Is it possible to have this document display in the Logs UI? Can I convert the fields in syslog-ng to lowercase before forwarding them to elastic? Thanks, Shawn

3 4

Extract SNI on incoming connections
by Raghunath Adhyapak 23 May '20

23 May '20

Hi, I would like to run a syslog-ng server that accepts TLS connections. I want to identify clients based on the server name they use to connect and do specific processing for each client. I see the following macros being available to be used in filter, but I don't see .TLS.SN (or something like that to indicate server name used by client to connect) .TLS.X509 *Description:* When using a transport that uses TLS, these macros contain information about the peer's certificate. That way, you can use information from the client certificate in filenames, database values, or as other metadata. If you clients have their own certificates, then these values are unique per client, but unchangeable by the client. The following macros are available in syslog-ng OSE version 3.9 and later. - .TLS.X509_CN: The Common Name of the certificate. - .TLS.X509_O: The value of the Organization field. - .TLS.X509_OU: The value of the Organization Unit field. Thanks Raghu

2 2

Convert json format to syslog rfc5424
by Carlos Lopez 22 May '20

22 May '20

Hi all, Due to some issues with our logs collector server, I need to read some logs in json format and convert them to syslog rfc5424 and forward to our central syslog server. Acctually, all information regarding this type of configuration located in syslog-ng’s web site is related to send json format as a destination. But, is it possible to do the opposite: read a json file and forward to a third server in syslog rfc5424 format? Yes, I will try to catch up only some specific fields, not all fields contained in the json soure file. Any tips? Regards, C. L. Martinez

2 1

getting "Error processing log message" for checkpoint logs
by Russell Fulton 20 May '20

20 May '20

I am trying to ingest logs from a checkpoint firewall over a tcp connection: source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); }; # added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng destination d_debug { file( "/data/russell//test.log" ); }; log { source( s_checkpoint ); destination( d_debug ); }; This is what turns up in the output: May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -….. Any ideas on what is going on or how to find out? Russell

2 5

How to get the number of logged message count
by Monitto Pitchaimani sebastin (mopitcha) 18 May '20

18 May '20

Hi All, Is there a way to get the logged messages to the particular destination? We wanted to get the count of number of logs delivered to the particular destination I.e., network() etc Regards, Monitto

2 1

Insider 2020-05: syslog-ng 3.27; Python HTTP header; DBLD; Sumologic;
by Peter Czanik (pczanik) 14 May '20

14 May '20

Dear syslog-ng users, This is the 81st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.27 of syslog-ng released ---------------------------------- Version 3.27.1 of syslog-ng is now available, adding many smaller features to syslog-ng. These include new macros to check where a given log message arrived, a new rewrite to easily change the syslog facility of a message, a Sumologic destination and an experimental feature to check log integrity. For source code and full details, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.27.1 For ready-to-use packages check https://syslog-ng.com/3rd-party-binaries Introduction to the Python HTTP header -------------------------------------- You can create your own custom headers for the HTTP destination using syslog-ng’s Python HTTP header plugin and Python scripts. The included example configuration only adds a simple counter to the headers, but with a bit of coding, you can resolve authentication problems or fine-tune how data is handled at cloud-based logging and SIEM platforms, like Sumologic. https://www.syslog-ng.com/community/b/blog/posts/introduction-to-the-python… DBLD: a syslog-ng developer tool not just for developers -------------------------------------------------------- DBLD is a central tool when it comes to syslog-ng development, but even after multiple blogs about the tool, it is still not much used outside of the developers’ team. So, what is DBLD and how could it be used even by you? The abbreviation stands for Docker BuiLD. Using containers ensures both that you have an easily reproducible build environment, and also that you do not have to “pollute” your base system with development-related software packages. You can use DBLD to build the release tarball or ready-to-use packages for a number of Linux distributions. It can even be used as a development environment with all necessary tools installed. Still not convinced? Yes, the listed possibilities are mostly interesting for syslog-ng developers and 3rd party packagers. If you are lucky, you will never need DBLD. On the other hand, it can come handy if you reported a problem and the syslog-ng team fixed it. Even if you are not a developer, only a junior sysadmin, you can still easily build fixed syslog-ng packages for testing (and even for production use) until an official release with the fix arrives. https://www.syslog-ng.com/community/b/blog/posts/dbld-a-syslog-ng-developer… Getting your logs into Sumo Logic with syslog-ng made easy ---------------------------------------------------------- Sumo Logic is one of the most popular cloud-based log management and security analytics services. They provide their own log shippers, but also work with others, including syslog-ng. The use of syslog-ng is well documented on their support website, but the actual configuration part can easily be simplified. A new, dedicated Sumo Logic destination is already available as part of syslog-ng 3.27. It is based either on the network() or the http() destination of syslog-ng and implemented as an SCL, hiding away most of the configuration details from the users. https://www.syslog-ng.com/community/b/blog/posts/getting-your-logs-into-sum… syslog-ng PE 7.0.20 available ----------------------------- Version 7.0.20 of syslog-ng Premium Edition was released. It adds Red Hat Enterprise Linux 8 to the supported platforms: https://support.oneidentity.com/technical-documents/syslog-ng-premium-editi… Using RHEL 8 instead of RHEL 7 you can use udp-balancer() source for high performance UDP log collection: https://support.oneidentity.com/technical-documents/syslog-ng-premium-editi… NEW WHITE PAPERS * Upgrade to State-of-the-Art Log Management: https://www.syslog-ng.com/whitepaper/log-management-essentials8137754/ WEBINARS * Learn how to easily integrate Azure Sentinel into your log-data strategy: https://www.syslog-ng.com/webcast-ondemand/learn-how-to-easily-integrate-az… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Missing one log message when destination unavailable for short period of time.
by Ali Shirvani 08 May '20

08 May '20

Hi all, I tried to setup simple relay with syslog-ng, it receives log on one port and forward to the destination. It works fine when destination is available, but when destination unavailable for a short period of time and then became available I miss one log entry. For example assume that I send simple numbers with logger command to syslog-ng, from 0 to 100, when I receive 50 on destination I stop its listener and start it again, in this case the first log entry that I receive is 52 instead of 51. Here is my syslog-ng.conf: @version: 3.26 options { time_reopen(5); }; source s_network { network( ip("127.0.0.1") port(514) transport("udp") ); }; destination d_network { network( "127.0.0.1" port(5514) transport("udp") ); }; log { source(s_network); destination(d_network); }; Would you please help me to resolve this issue? Regards, Ali

3 6

Syslogs not writing to disk
by Rose, Ed 05 May '20

05 May '20

Hello All, I have syslog-ng setup currently setup to write to disk and forward events on to a different syslog server. Syslog isn't writing to disk it seems. I do not see any errors either, unless I am not looking in the right place for the errors. Tcpdump -v host x.x.x.x shows data coming in from the host but a tail -f ${HOST}/log_${HOST} shows no logs going to disk. Thanks Ed syslog-ng 3 (3.14.1) Config version: 3.14 Installer-Version: 3.14.1 Revision: Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Available-Modules: afmongodb,sdjournal,linux-kmsg-format,appmodel,date,pseudofile,afamqp,syslogformat,csvparser,cef,tags-parser,affile,afsocket,afstomp,tfgetent,json-plugin,confgen,system-source,stardate,graphite,dbparser,afprog,xml,add-contextual-data,snmptrapd-parser,cryptofuncs,kvformat,disk-buffer,basicfuncs,afuser,map-value-pairs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on filter f_awanra1 { host("x.x.x.x"); }; filter f_bwanra1 { host("x.x.x.x"); }; filter f_cwanra1 { host("x.x.x.x"); }; destination d_ciscoasa { file("/var/log2/gns/cisco_asa/${HOST}/log_${HOST}" create-dirs(yes) dir-perm(0755) dir-group(splunk) perm(0640) group(splunk)); }; destination d_uid1 { tcp("x.x.x.x" port(514) ); }; destination d_uid2 { tcp("x.x.x.x" port(514) ); }; destination d_uid3 { tcp("x.x.x.x" port(514) ); }; destination d_uid4 { tcp("x.x.x.x" port(514) ); }; log { source(s_net); filter(f_amywanra1); destination(d_ciscoasa); destination(d_uid1); destination(d_uid2); destination(d_uid3); destination(d_uid4); flags(final); }; log { source(s_net); filter(f_cnswanra1); destination(d_ciscoasa); destination(d_uid1); destination(d_uid2); destination(d_uid3); destination(d_uid4); flags(final); }; log { source(s_net); filter(f_demwanra1); destination(d_ciscoasa); destination(d_uid1); destination(d_uid2); destination(d_uid3); destination(d_uid4); flags(final); };

2 8

syslog-ng 3.27 Debian/Ubuntu packages (OBS)
by Laszlo Budai 30 Apr '20

30 Apr '20

Hi, syslog-ng 3.27.1[1] packages are available in OBS repositories[2, 3]. List of supported OSs: * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.27.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng 3.27
by Laszlo Budai 30 Apr '20

30 Apr '20

3.27.1 Highlights * DESTIP/DESTPORT/PROTO: new macros. (#2899<https://github.com/syslog-ng/syslog-ng/pull/2899>) * set-facility(): add new rewrite operation to change the syslog facility associated with the message. (#3136<https://github.com/syslog-ng/syslog-ng/pull/3136>) * network tls: Added ca-file() option. With this option the user can set a bundled CA-file to verify the peer. (#3145<https://github.com/syslog-ng/syslog-ng/pull/3145>) * Forward integrity and confidentiality of logs (#3121<https://github.com/syslog-ng/syslog-ng/pull/3121>): It is an experimental feature currently, we are still working on the final form with the authors. Features * DESTIP/DESTPORT/PROTO: new macros These new macros express the destination ip, destination port and used protocol on a source. The use-case behind the PR is as follows: * someone has an appliance which sends out log messages via both UDP and TCP * the format of the two are different, and he wants to capture either with the simplest possible filter * netmask() doesn't work because the IP addresses are the same * host() doesn't work because the hostnames are the same Example: log { source { network(localip(10.12.15.215) port(5555) transport(udp)); }; destination { file("/dev/stdout" template("destip=$DESTIP destport=$DESTPORT proto=$PROTO\n")); }; }; Output: destip=10.12.15.215 destport=5555 proto=17 (#2899<https://github.com/syslog-ng/syslog-ng/pull/2899>) * set-facility(): add new rewrite operation to change the syslog facility associated with the message. log { source { system(); }; if (program("postfix")) { rewrite { set-facility("mail"); }; }; destination { file("/var/log/mail.log"); }; flags(flow-control); }; (#3136<https://github.com/syslog-ng/syslog-ng/pull/3136>) * network tls: Added ca-file() option. With this option the user can set a bundled CA-file to verify the peer. (#3145<https://github.com/syslog-ng/syslog-ng/pull/3145>) * http: When a HTTP response is received, emit a signal with the HTTP response code. (Later it can be extended to read the response and parse it in a slot...). This PR also extends the Python HTTP header module with the possibility of writing custom HTTP response code handlers. When someone implements an auth header plugin in Python, it could be useful (for example invalidating a cache). Example config, click to expand! (#3159<https://github.com/syslog-ng/syslog-ng/pull/3159>) * java/python: add support for the "arrow" syntax. options("key" => "value") (#3161<https://github.com/syslog-ng/syslog-ng/pull/3161>) (#3247<https://github.com/syslog-ng/syslog-ng/pull/3247>) * python: persist support for python This feature enables users to persist data between reloads or restarts. The intended usage is to support bookmarking and acknowledgement in the future. It is not suitable for local database use cases. (#3171<https://github.com/syslog-ng/syslog-ng/pull/3171>) * rewrite: Added conditional set-tag() option. With this option the user can put condition statement inside set-tag option. rewrite { set-tag("tag" condition(match("test" value("MSG")))); }; (#3190<https://github.com/syslog-ng/syslog-ng/pull/3190>) * scl: add sumologic destinations: sumologic-syslog() and sumologic-http() (#3194<https://github.com/syslog-ng/syslog-ng/pull/3194>) * iterate: new template function The iterate template function generates a series from an initial number and a next function. For example you can generate a sequence of nonnegative numbers with source { example-msg-generator( num(3) template("$(iterate $(+ 1 $_) 0)") ); }; (#3205<https://github.com/syslog-ng/syslog-ng/pull/3205>) * telegram: new max-size option Telegram message will be truncated for max-size size. Telegram does not accept message larger than 4096 utf8 characters. The default value is 4096. (#3206<https://github.com/syslog-ng/syslog-ng/pull/3206>) * example-message-generator : add support for values(name1 => value1, name2 => value2,..) syntax. Example @version: 3.27 log { source { example-msg-generator(template("message parameter") num(10) values("PROGRAM" => "program-name" "current-second" => "$C_SEC" )); }; destination { file(/dev/stdout template("$(format-json --scope all-nv-pairs)\n")); }; }; (#3237<https://github.com/syslog-ng/syslog-ng/pull/3237>) * example-msg-generator: support freq(0) for fast message generation log { source { example-msg-generator(freq(0) num(100)); }; destination { file("/dev/stdout"); }; }; (#3245<https://github.com/syslog-ng/syslog-ng/pull/3245>) Bugfixes * file: changed time-reap() timer's schedule to respect the documentation (expires after last message) (#3133<https://github.com/syslog-ng/syslog-ng/pull/3133>) * dbld: fix building problems * fix rpm package build on centos-7 * fix devshell image build * fix ubuntu-trusty image build * fix deb package build on ubuntu-trusty * fix rpm package build on fedora-30 (#3143<https://github.com/syslog-ng/syslog-ng/pull/3143>) * tls (network): Properly log an error message, when key-file() or cert-file() is missing. (#3145<https://github.com/syslog-ng/syslog-ng/pull/3145>) * loggen: fix crash with invalid parameterization (#3146<https://github.com/syslog-ng/syslog-ng/pull/3146>) * format-json: fix printing of embedded zeros Prior to 2.64.1, g_utf8_get_char_validated() in glib falsely identified embedded zeros as valid utf8 characters. As a result, format json printed the embedded zeroes as \u0000 instead of \x00. This change fixes this problem. (#3175<https://github.com/syslog-ng/syslog-ng/pull/3175>) * configure: fix --with-net-snmp configure option (#3180<https://github.com/syslog-ng/syslog-ng/pull/3180>) * python: fix Py_None reference counting in logger methods (trace, debug, info, warning, error) (#3187<https://github.com/syslog-ng/syslog-ng/pull/3187>) * afmongodb: do not build module when ENABLE_MONGODB=OFF (#3188<https://github.com/syslog-ng/syslog-ng/pull/3188>) * telegram: automatically truncate messages larger than 4096 utf8 characters to avoid telegram destination to get stuck (#3206<https://github.com/syslog-ng/syslog-ng/pull/3206>) * compat/glib: fix recursive call issue on CentOS-6/RHEL-6/platforms (#3212<https://github.com/syslog-ng/syslog-ng/pull/3212>) * timeutils: fix crash in %f conversion when non-numeric character is in usec field (e.g. ".asd123") (#3270<https://github.com/syslog-ng/syslog-ng/pull/3270>) Packaging * macOS: add example startup configuration. (#3172<https://github.com/syslog-ng/syslog-ng/pull/3172>) * rpm: fix --without maxminddb option If maxminddb development package was installed on the build system: rpmbuild fails if --without maxminddb was used. (#3208<https://github.com/syslog-ng/syslog-ng/pull/3208>) Notes to developers * light: Support to relocate reports dir other than current base dir For example python -m pytest -lvs functional_tests/source_drivers/file_source/test_acceptance.py --installdir=/install --reports /tmp/ (#3157<https://github.com/syslog-ng/syslog-ng/pull/3157>) * CONTRIBUTING.md: contribution guide updated (#3174<https://github.com/syslog-ng/syslog-ng/pull/3174>) * libtest: Now we install config_parse_lib.h, fake-time.h, mock-cfg-parser.h and queue_utils_lib.h which help unit testing outside of core. (#3179<https://github.com/syslog-ng/syslog-ng/pull/3179>) * tests: Wait until snmptrapd process able to write traps into output file (#3185<https://github.com/syslog-ng/syslog-ng/pull/3185>) * mongodb: Replaced the deprecated get_server_status() API with command_simple(). This means, that syslog-ng can now be built with -Werror on systems with 1.15 libmongoc. (#3199<https://github.com/syslog-ng/syslog-ng/pull/3199>) * stats: add external stats counters There are situations when someone wants to expose internal variables through stats-api. Without this changeset, we need two distinct variables: one is for the internal state, other is for the stats registration (internal state cannot depends on the life cycle of the StatsCounterGroup). (#3225<https://github.com/syslog-ng/syslog-ng/pull/3225>) Other changes * afsnmp: merge two existing SNMP modules (trapd-parser and destination) into one. Motivation: keep closely related modules together and decrease the number of small packages. Packaging related changes: libsnmptrapd-parser.so has been removed, both the snmp destination and the trapd parser are part of afsnmp.so . * deb: we already had mod-snmp which shipped snmp-dest. From now, this packages installs the snmptrapd-parser syslog-ng module. The syslog-ng-mod-snmptrapd-parser package has been removed. * rpm: snmpdest renamed to afsnmp and the snmptrapd-parser is installed by this package from now (#3142<https://github.com/syslog-ng/syslog-ng/pull/3142>) * dbld: remove ubuntu-cosmic as it has reached EOL (#3143<https://github.com/syslog-ng/syslog-ng/pull/3143>) * afsocket-source: present the number of connections in stats It helps in the debug process if we can see the number of source connections counted by syslog-ng internally. (#3193<https://github.com/syslog-ng/syslog-ng/pull/3193>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Kokan, Laszlo Budai, Laszlo Szemere, László Várady, Mehul Prajapati, Roberto Meléndez, Stephan Marwedel, Steven Haigh, Peter Czanik, Thomas De Schampheleire, Vatsal Sisodiya, Vivin Peris https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.27.1

1 0

Issue with processing syslog event
by Raghunath Adhyapak 29 Apr '20

29 Apr '20

Hello all, I am receiving the following syslog line from one of devices. <134>1 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53 This line contains a version filed immediately following the priority, and then timestamp is in epoch format as against ISO8601 or other standard format. I see the following error in syslog-ng log: [2020-04-28T10:46:15.340911] Outgoing message; message='Apr 28 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53' What could be the possible issue here? My config is as follows: ##======================================== ######################## # Global options ######################## options {keep_hostname (yes); use_dns (no); mark-freq(30);}; ######################## # Sources ######################## source s_syslog { syslog( transport(udp) port(514) max-connections(500) ); }; ######################## # Destinations ######################## destination d_file { file("/var/log/dump.log"); }; ######################## # Log paths ######################## log { source(s_syslog); destination(d_file); flags(flow-control); }; ##======================================== If I check my file /var/log/dump.log, I see that the error line is getting written to it too. root@ip-172-31-240-95:~# tail -f /var/log/dump.log | grep "Error processing" Apr 28 10:44:46 ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53 Apr 28 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53 Thanks Raghu

3 5

Syslog-ng network namespace support
by Monitto Pitchaimani sebastin (mopitcha) 28 Apr '20

28 Apr '20

Hi All, Does syslog-ng supports log forwarding to remote host in different network namespaces? Regards, Monitto

3 5

Re: [syslog-ng] syslog-ng, json and handle escaping properly
by Arthur Lambert 27 Apr '20

27 Apr '20

> log { > source { example-msg-generator(num(1) template("{\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Check pstore partition..\"}")); }; > parser { json-parser(); }; > destination { file(/dev/stdout template("$(format-json --key category,level,message timestamp=${ISODATE})\n")); }; >}; Ok so first I update the syslog-ng config file with your modification diff --git a/board/xxx/xxx/rootfs/etc/syslog-ng.conf b/board/xxx/xxx/rootfs/etc/syslog-ng.conf index c28ea5e59d..6b93ce24b8 100644 --- a/board/xxx/xxx/rootfs/etc/syslog-ng.conf +++ b/board/xxxx/xxx/rootfs/etc/syslog-ng.conf @@ -4,10 +4,6 @@ filter f_middleware { facility("local1"); }; -template t_json_filetemplate { - template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n"); -}; source s_kernel { file("/proc/kmsg" program_override("kernel")); }; @@ -21,7 +17,8 @@ destination d_kernel { }; destination d_middleware { - file("/data/logs/middleware.log", template(t_json_filetemplate)); + file(/data/logs/middleware.log template("$(format-json --key category,level,message timestamp=${ISODATE})\n"));; + }; log { @@ -31,6 +28,7 @@ log { log { source(s_middleware); + parser { json-parser(); }; filter(f_middleware); destination(d_middleware); }; I also add the '{' in my macro log diff --git a/log/log_private.h b/log/log_private.h index b1e2dd31..09e875e1 100644 --- a/log/log_private.h +++ b/log/log_private.h @@ -18,7 +18,7 @@ #define _PRINT_DEBUG(M, ...) \ do { \ - syslog(LOG_DEBUG, "\"category\":\"%s\",\"level\":\"DEBUG\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \ + syslog(LOG_DEBUG, "{\"category\":\"%s\",\"level\":\"DEBUG\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \ } while (0) #else @@ -28,32 +28,32 @@ #define _PRINT_INF_GOTO(M, ...) \ do { \ - syslog(LOG_DEBUG, "\"category\":\"%s\",\"level\":\"INFO\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \ + syslog(LOG_DEBUG, "{\"category\":\"%s\",\"level\":\"INFO\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \ errno = 0; \ goto error; \ (...) #endif /* !LOG_PRIVATE_H_ */ So basically I suppose that your modification allows to syslog-ng to understand by itself that there is an extra pair key/value in my format and that I am using JSON format. So let's do some test now... int main (void) { PRINT_DEBUG ("Test simple."); PRINT_DEBUG ("\nTest with return line\n."); PRINT_DEBUG ("Test with json inside message : {\"key\" : \"%d\", \"key2\" : \"%d\"} ", 42, 21); PRINT_DEBUG ("Test simple2."); PRINT_DEBUG ("Begin. Program name : %s", argv[0]); (...) } Result : # cat /data/logs/middleware.log {"timestamp":"2020-04-24T15:36:28+02:00","message":"Test simple.","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:36:28+02:00","message":"Test simple2.","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:36:28+02:00","message":"Begin. Program name : nano_core","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:36:28+02:00","message":"Board config : nano_proto1","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:36:28+02:00","message":"Initialize GPIO..","level":"INFO","category":"GPIO_MANAGER"} {"timestamp":"2020-04-24T15:36:28+02:00","message":"Synchronize dictionnary with current rootfs content","level":"INFO","category":"AUDIO_MANAGER"} As you can see, the line with \n and the JSON debug output is not in the final log. It was not in a good format for the JSON syslog-ng parser. How can I fix that? Now that syslog-ng is able to understand that there is a category, a level, a message in my JSON. I cannot just ask for syslog-ng to use the escape template only on the column with message key before sending the message to the parser? I am able to fix the \n issue by using unix-dgram instead of unix-stream but without really understand why. New result after using unix-dgram : {"timestamp":"2020-04-24T15:58:09+02:00","message":"Test simple.","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:58:09+02:00","message":"\nTest with return line\n.","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:58:09+02:00","message":"Test simple2.","level":"DEBUG","category":"MPU_CORE"} {"timestamp":"2020-04-24T15:58:09+02:00","message":"Begin. Program name : nano_core","level":"DEBUG","category":"MPU_CORE"}

2 1

syslog-ng, json and handle escaping properly
by Arthur Lambert 24 Apr '20

24 Apr '20

Hello, I tried to migrate from syslog to Syslog-ng. I am trying to deploy it on an embedded target with buildroot. At the same time, I also tried to switch log format to JSON. In my c code, I use Syslog function and init with the openlog function: openlog("middleware", 0, LOG_LOCAL1); I have some macro to generate the json logs: #define _PRINT_DEBUG(M, ...) \ do { \ syslog(LOG_DEBUG, "\"category\":\"%s\",\"level\":\"DEBUG\",\"message\":\"" M "%s\"}\n", __DIR__, __VA_ARGS__); \ } while (0) I define a syslog-ng conf file: @version: 3.2X filter f_middleware { facility("local1"); }; template t_json_filetemplate { template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n"); }; source s_kernel { file("/proc/kmsg" program_override("kernel")); }; source s_middleware { unix-stream("/dev/log"); }; destination d_kernel { file("/data/logs/kern.log"); }; destination d_middleware { file("/data/logs/middleware.log", template(t_json_filetemplate)); }; log { source(s_kernel); destination(d_kernel); }; log { source(s_middleware); filter(f_middleware); destination(d_middleware); }; So basically I want to redirect all my log from my code in /data/logs/middleware.log. I also want to redirect all kernel logs in /data/logs/kern.log I have two issues. The first one is escaping. If I want to print a JSON file the double quote will be not escaped. My first solution was to use escape feature from Syslog-ng: destination d_middleware { file("/data/logs/middleware.log", template("{\"timestamp\":\"${ISODATE}\",${MESSAGE}\n") template-escape(yes)); }; Problem is that now everything is escaped and not only the message, some example: {"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Check pstore partition..\"} {"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Start hardware watchdog daemon...\"} {"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Prepare asound configuration...\"} {"timestamp":"2020-04-21T09:24:02+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Middleware is about to start...\"} {"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Load sdma firmware..\"} {"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Watchdog is about to start...\"} {"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"BOOT\",\"level\":\"INFO\",\"message\":\"Start mpu watchdog..\"} As you can see, everything is escaped because for Syslog-ng, everything is the message and not only the final part. I feel like I am completely using Syslog-ng in the wrong way right? How can I fix that? Do I have to handle escaping by myself manually in my middleware? Do I have to pre-process all logs manually by myself? It seems quite awful. I have a second issue which is probably related? When I tried to print the content of a file, the last '}' is truncated. The reason seems to be related to the presence of \n. I can reproduce the issue by just calling my PRINT_DEBUG macro as follow: PRINT_DEBUG ("YO YO YO KO\n"); PRINT_DEBUG ("YO YO YO OK"); The result: {"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"MPU_CORE\",\"level\":\"DEBUG\",\"message\":\"YO YO YO KO {"timestamp":"2020-04-21T09:24:03+02:00",\"category\":\"MPU_CORE\",\"level\":\"DEBUG\",\"message\":\"YO YO YO OK\"} My syslog version/module information : # syslog-ng -V -s syslog-ng 3 (3.26.1) Config version: 3.22 Installer-Version: 3.26.1 Revision: Compile-Date: Apr 10 2020 16:41:46 Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,examples,graphite,hook-commands,ht tp,kvformat,linux-kmsg-format,map-value-pairs,mod-python,pseudofile,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off -- - Arthur LAMBERT

3 2

syslog-ng-ctl: EOF occurred while reading control socket
by Faine, Mark R. (MSFC-IS40)[NICS] 24 Apr '20

24 Apr '20

I am seeing the message "EOF occurred while reading control socket" intermittently when using the syslog-ng-ctl command. Is there something misconfigured or anything I can do to prevent this? Thanks, -Mark

2 1

mystique with spoof_address
by Edvinas Kairys 16 Apr '20

16 Apr '20

Hello, i installed (yum install) following version on Centos 7 box. syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: Compile-Date: Dec 30 2015 19:57:24 Available-Modules: affile,afprog,afsocket-notls,afsocket-tls,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,linux-kmsg-format,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on My goal is to forward syslog messages 'untouched' but to change the source address to original one. For that case i'm using spoof-address. My conf is like this: options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (no); mark-freq (0); }; source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); }; destination d_syslog_tcp { network("10.13.33.125" transport("udp") port(5140) spoof-source(yes)); }; log { source(s_network); destination(d_syslog_tcp); }; log { source(s_network); filter(f_default); destination(d_mesg); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" Strange thing, that when I enable spoof-source, some packets are not transmitted to the destination. Even TCPDUMP says that it's sent, but i dont see some logs on destination box. Could it be something with spoof_source command ? Also i didn't compiled it because i saw that SPOOF functionality is on in syslog-ng -V output. Any suggestions ? Thanks

2 4

issues with rewrite and subst
by Russell Fulton 14 Apr '20

14 Apr '20

version 3.26.1 I am attempting to use rewrite to copy part of one macro into a new one: rewrite r_user1 { set( "${user}", value("user_name"));}; rewrite r_user2 { subst( "^.+$(.+)$", "${1}", value("user_name")) ;}; The first simply copies user into user_name. The second is intended to extract anything in parentheses and throw away the rest. here is a cleaned up dump of json output including the macros from the regexp: {"user_name":")","user”:”yyyyn@gmail.com(xxx549)","2":")","1":")","0”:”yyyy@gmail.com(xxx549)”} I believe ‘1’ should contain ‘xxx549’ not ‘)’. I have tested the regexpn in ruby’s irb and it works fine there. here is another example this one where the regexp should not have matched at all: {"user_name":"9","user":"ping999","2":"9","1":"9","0":"ping999”} macros 1 and 2 always seem to contain the last character of the original string.

2 1

Regular occurrences of Error writing control channel; error='Broken pipe (32)'
by Ahmed Soyef 09 Apr '20

09 Apr '20

Hi All, We've been running syslog-ng and have recently started to see the above message in our logs. Sometimes running syslog-ng-ctl stats takes very long and results in similar error messages. Does anyone know why this may be the case? Do these errors cause any issues with log flows through the syslog-ng service? Do they cause issues with recording of stats? Kind regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************

3 3

Insider 2020-04: syslog-ng RPM; mult-line-timeout; FreeBSD;
by Peter Czanik (pczanik) 09 Apr '20

09 Apr '20

Dear syslog-ng users, This is the 80th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. In these difficult times, when many of us stay at home, we can decide what to do with that bit of extra time that we don’t spend commuting. Some of us turn to entertainment or hobbies, do long-neglected things around the house, or even focus on studying and honing skills. If you ran out of movies to watch and want to spend your time on something more practical, this might be the best time to learn something new about syslog-ng. One starting point for that is the reference documentation, available for each release separately: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-ed… Another possibility is to check out our ever-growing collection of syslog-ng blogs. These describe a large number of different syslog-ng use cases. Blogs are available at https://www.syslog-ng.com/community/b/blog If you want to deep dive into configuring syslog-ng, here are some of the highlights: * The building blocks of syslog-ng: https://www.syslog-ng.com/community/b/blog/posts/building-blocks-of-syslog-… * Calculate PI with syslog-ng: https://www.syslog-ng.com/community/b/blog/posts/calculate-pi-syslog-ng * Creating heat maps using the new syslog-ng geoip2 parser: https://www.syslog-ng.com/community/b/blog/posts/creating-heat-maps-using-n… * Extend syslog-ng in Python: https://www.syslog-ng.com/community/tags/python Even if it is often not real-time, you can also chat with syslog-ng users and developers. The syslog-ng Gitter channel is a place to ask syslog-ng-related questions, but you are also very welcome to share your experiences about syslog-ng here. A success story not just feels good for developers but can also inspire other users. It is available at https://gitter.im/syslog-ng/syslog-ng NEWS Overview of syslog-ng RPM repositories -------------------------------------- This blog provides you an overview of syslog-ng RPM repositories: why to have unofficial packages at all and what are the advantages and disadvantages of the different unofficial repositories. https://www.syslog-ng.com/community/b/blog/posts/overview-of-syslog-ng-rpm-… Multi-line-timeout: making sure your last multi-line message is not lost -------------------------------- When your application has a problem that it cannot handle, then Java, PHP and other environments often write multi-line error messages. These long messages include information that is useful for developers, but it might be difficult to handle for logging systems. A multi-line log message is usually saved when the beginning of the next message is detected, which can make handling the last message problematic. This is where multi-line-timeout can help. After a pre-configured timeout, the last message is saved by syslog-ng. Losing a message when there are thousands of them is probably not a big deal. But if you are lucky, the error messages are rare. Maybe just a single message in a file. In either case, you can lose most or even all messages. If you do not want to lose these messages, configure multi-line-timeout. https://www.syslog-ng.com/community/b/blog/posts/multi-line-timeout-making-… syslog-ng in FreeBSD ports -------------------------- Recently we explored the different syslog-ng packages available for RPM-based Linux distributions, used by the majority of our Linux users. This time we focus on FreeBSD, the platform of choice for most of our non-Linux users. >From this blog, you can learn about the features that are in the binary syslog-ng packages, how to compile syslog-ng for yourself from ports and the current syslog-ng version policy in ports. NEW WHITE PAPERS * Upgrade to State-of-the-Art Log Management: https://www.syslog-ng.com/whitepaper/log-management-essentials8137754/ WEBINARS * Learn how to easily integrate Azure Sentinel into your log-data strategy: https://www.syslog-ng.com/webcast-ondemand/learn-how-to-easily-integrate-az… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

can not load geoip2 plugin
by Russell Fulton 09 Apr '20

09 Apr '20

HI This is a locally compiled application (version 3.26.1) and I am trying to get the geoip2 stuff working. I had similar problems a week or two back with elasticsearch_http destination plugin. This seems like it might be easier to diagnose since the error says where it is looking for the plugin. rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.26.1/sbin/syslog-ng -s -f /etc/syslog-ng/geoip.conf [2020-04-08T07:30:19.551845] Plugin module not found in 'module-path'; module-path='/usr/local/syslog-ng-3.26.1/lib/syslog-ng', module='geoip2' Error parsing config, syntax error, unexpected LL_ERROR, expecting $end in /etc/syslog-ng/geoip.conf:4:1-5:1: 1 @version: 3.21 2 3 @include "scl.conf" 4-----> @module geoip2 4-----> ^^^^^^^^^^^^^^^ 5-----> # 6 7 options { 8 use_dns (no); 9 use_fqdn (no); rful011@secmgrprd02:~$ ls -l /usr/local/syslog-ng-3.26.1/lib/syslog-ng/libgeoip2-plugin.* -rwxr-xr-x 1 root root 1271 Mar 31 08:57 /usr/local/syslog-ng-3.26.1/lib/syslog-ng/libgeoip2-plugin.la -rwxr-xr-x 1 root root 158912 Mar 31 08:57 /usr/local/syslog-ng-3.26.1/lib/syslog-ng/libgeoip2-plugin.so any ideas why it is not loading the library?

2 2

3.26 RHEL/CentOS 6 packages
by Peter Czanik (pczanik) 02 Apr '20

02 Apr '20

Hi, syslog-ng 3.26 packages are now also available for RHEL/CentOS 6: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng326epel6/ And if you use syslog-ng on RHEL/CentOS I'd appreciate if you could answer a quick poll on Twitter: https://twitter.com/PCzanik/status/1245692669889777667 Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

errors with elasticsearch_http on 2.21 & 3.22
by Russell Fulton 02 Apr '20

02 Apr '20

I have had this config running happily for several months and today I tried to add the geoip2 plugin and managed to break everything. I first realised that I needed to recompile syslog-ng to get the geoip2 plugin but forgot I was running a locally compiled version . I tried apt install syslog-ng-mod-geoip and this broke things badly and I have been trying to recover for the last few hours :( The issue is that any attempt to start syslog-ng with the original config that includes a destination of elasticsearch_http get an error: Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.22/etc/syslog-ng.conf:33:3-33:21 Judging from what I found on th ’Net this is what one would expect if the syslog-ng version was prior to 3.21 and did not support the http dest. I first tried recompiling and installing 3.21 (in a different dir) with geoip2 enabled. That went with out errors but still would not process my config. Aside: At this point I found that in spite of having the geoip2 module included, syslog-ng failed to find it when I copied the setup from the manual. long story short the manual says the module is ‘geoip2’ when in fact it is ‘geoip2-plugin’ and that is the name you have to use in the @module. Weird! I then compiled 3.22 since I had the tar file and this behaved the same. rful011@secmgrprd02:~$ cat short.conf @version: 3.21 @include "scl.conf" @module geoip2-plugin source s_loghost { network( transport("tcp") flags(no-multi-line) port(1514) keep-alive(yes)); }; destination d_elastic { elasticsearch_http( index("auth_${YEAR}.${MONTH}.${DAY}") type("auth") persist-name("auth") template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") ); }; log { source(s_loghost); destination(d_elastic ); flags(flow-control); }; rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s -c ~/short.conf Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.21/etc/syslog-ng.conf:33:3-33:21: 28 file( "/home/rful011/test.log" template(t_db_parsed_test)); 29 }; 30 31 32 destination d_elastic { 33----> elasticsearch_http( 33----> ^^^^^^^^^^^^^^^^^^ 34 index("auth_${YEAR}.${MONTH}.${DAY}") 35 type("auth") 36 persist-name("auth") 37 template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") 38 url("http://secesprd01.its.auckland.ac.nz:9200/_bulk”) rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng --version syslog-ng 3 (3.21.1) Config version: 3.21 Installer-Version: 3.21.1 Revision: Compile-Date: Mar 29 2020 11:57:39 Module-Directory: /usr/local/syslog-ng-3.21/lib/syslog-ng Module-Path: /usr/local/syslog-ng-3.21/lib/syslog-ng Include-Path: /usr/local/syslog-ng-3.21/share/syslog-ng/include Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip2-plugin,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off

3 8

Syslog-NG Mysql Parsing to separate column
by Sync IT 01 Apr '20

01 Apr '20

Hi I am new to syslog-ng. I am receiving my logs perfectly to mysql. now i want to separate the message log into different column in mqsql. Currently i am receiving messages like this R2_DST_RTR forward: in:<pppoe-user1> out:vlan100-Default-Internet, src-mac 11:22:33:44:55:66, proto TCP (ACK,FIN), 172.16.41.159:42670->23.46.16.231:80, NAT (172.16.41.159:42670->9.9.9.9:42670)->23.46.16.231:80, len 52 This is my syslog-ng config source s_net{ udp( ); }; destination d_mtksql { sql(type(mysql) host("localhost") username("dbuser") password("dbpasss") database("syslog") table("${R_YEAR}_${R_MONTH}_${R_DAY}") columns( "id int(11) unsigned not null auto_increment primary key", "host varchar(40) not null", "date datetime", "message text not null") values("0", "$FULLHOST", "$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC", "$MSG") indexes("id")); }; log { source(s_net); destination(d_mtksql); }; i want to separate the message data in different column like- time,username,src mac,src ip,dst ip, nat ip. Any idea how can i achieve that? I am not a pro so if possible kindly write in details how to parse them. Many thanks

2 1

Change Position Information inside Messasge
by William Luiz Ribeiro Vasconcelos Da Silva 27 Mar '20

27 Mar '20

Hello everyone, Is it possible to change a position of an information within the message? From: 2020-03-26; 17: 55: 22; RTCGNMGA0101: RT_SRC_NAT_PBA_ALLOC :; 100.64.1.114:38784-38847; 177.51.112.218; TO: 2020-03-26; 17: 55: 22; RTCGNMGA0101: RT_SRC_NAT_PBA_ALLOC:; 100.64.1.114; 177.51.112.218:38784-38847; Basically, I would like to change the position of the connection port within the message. How to do this? TKS Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson Mobile +55 11 97979-9886 www.ericsson.com<http://www.ericsson.com/> [Descri??o: Descri??o: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informa??es privilegiadas e/ou de car?ter confidencial, n?o podendo ser retransmitida sem autoriza??o do remetente. Se voc? n?o ? o destinat?rio ou pessoa autorizada para receb?-la, informamos que o seu uso, divulga??o, c?pia ou arquivamento s?o proibidos. Portanto, se voc? recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conte?do. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 3

variable host filter in events
by Syslogng 25 Mar '20

25 Mar '20

Hi, We receive aggregated syslog from a server (all logs are send from 1 IP). Also all the events are mixed. The name of the host sending the initial traffic is in each event. ex: 2020/03/23 [notice] [user] New original_source=SERV1.example.com Task=0 .... How to recover SER1 which is always preceded by "original_source=" and followed by "example.com" to save it in a file for example /data/serv1.log I don't want a static filter (I know how to do it) but a dynamic one. If a new event arrives at original_source=SERV2.example.com I would like it to automatically create a /data/serv2.log Could you help me please ? thank you in advance Pit

2 3

Problem to Get UDP Packets - Syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 25 Mar '20

25 Mar '20

Hello Everyone, I installed syslog-ng on a new machine, however in initial tests, there was no collection of UDP packages by syslog-ng. Here are some points I checked: sudo netstat -plunt | grep -e PID -e syslog Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:601 0.0.0.0:* LISTEN 16169/syslog-ng udp 0 0 10.96.145.42:514 0.0.0.0:* 16169/syslog-ng netstat -anu | grep 514 udp 0 0 10.96.145.42:514 0.0.0.0:* [cgnat@mgalnxa01 etc]$ sudo systemctl status syslog-ng -l ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-03-25 12:38:08 -03; 5min ago Process: 114207 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Main PID: 16169 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─16169 /opt/syslog-ng/libexec/syslog-ng -F --enable-core Mar 25 12:38:08 mgalnxa01 systemd[1]: Starting System Logger Daemon... Here is an example of the package received via tcpdump, but it was not captured by syslog-ng: 10:46:13.529331 IP (tos 0x20, ttl 251, id 33055, offset 0, flags [none], proto UDP (17), length 243) 10.96.145.98.syslog > mgalnxa01.9514: [udp sum ok] SYSLOG, length: 215 Facility user (1), Severity info (6) Msg: Mar 24 13:46:13 2020 RTCGNMGA0103 RT_NAT: RT_SRC_NAT_PBA_ALLOC: Subscriber 100.64.1.6 used/maximum [2/2] blocks, allocates port block [47104-47167] from 177.51.116.146 in source pool PUBLIC-NAT-POOL-1 lsys_id: 0 0x0000: 3c31 343e 4d61 7220 3234 2031 333a 3436 0x0010: 3a31 3320 3230 3230 2052 5443 474e 4d47 0x0020: 4130 3130 3320 5254 5f4e 4154 3a20 5254 0x0030: 5f53 5243 5f4e 4154 5f50 4241 5f41 4c4c 0x0040: 4f43 3a20 5375 6273 6372 6962 6572 2031 0x0050: 3030 2e36 342e 312e 3620 7573 6564 2f6d 0x0060: 6178 696d 756d 205b 322f 325d 2062 6c6f 0x0070: 636b 732c 2061 6c6c 6f63 6174 6573 2070 0x0080: 6f72 7420 626c 6f63 6b20 5b34 3731 3034 0x0090: 2d34 3731 3637 5d20 6672 6f6d 2031 3737 0x00a0: 2e35 312e 3131 362e 3134 3620 696e 2073 0x00b0: 6f75 7263 6520 706f 6f6c 2050 5542 4c49 0x00c0: 432d 4e41 542d 504f 4f4c 2d31 206c 7379 0x00d0: 735f 6964 3a20 30 What I need to analyze / verify, for syslog-ng will capture this type of package and convert it to a file. Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson Rua Maria Preste Maia, 300 02879-130, Brazil Phone +55 11 2760-3785 Mobile +55 11 97979-9886 wsilva_ericsson(a)timbrasil.com.br<mailto:wsilva_ericsson@timbrasil.com.br> www.ericsson.com<http://www.ericsson.com/> [Descricao: Descricao: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informacoes privilegiadas e/ou de carater confidencial, nao podendo ser retransmitida sem autorizacao do remetente. Se voce nao e o destinatario ou pessoa autorizada para recebe-la, informamos que o seu uso, divulgacao, copia ou arquivamento sao proibidos. Portanto, se voce recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteudo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

3 8

Remove String from Message + syslog
by William Luiz Ribeiro Vasconcelos Da Silva 25 Mar '20

25 Mar '20

Hello eveyone, We are receiving two strings in the message in UDP packets: "used / maximum [1/2]" or "used / maximum [2/2]". How to configure the subst function, to be able to remove these strings? Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson Mobile +55 11 97979-9886 wsilva_ericsson(a)timbrasil.com.br<mailto:wsilva_ericsson@timbrasil.com.br> www.ericsson.com<http://www.ericsson.com/> [Descri??o: Descri??o: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informa??es privilegiadas e/ou de car?ter confidencial, n?o podendo ser retransmitida sem autoriza??o do remetente. Se voc? n?o ? o destinat?rio ou pessoa autorizada para receb?-la, informamos que o seu uso, divulga??o, c?pia ou arquivamento s?o proibidos. Portanto, se voc? recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conte?do. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 3

Disk Buffering
by Ronald Fenner 24 Mar '20

24 Mar '20

I'm going to enable disk buffer but I'd like to place the files on an EBS volume that persists when a server is shutdown. The idea being we'd have a recovery server that would stream out and items left in the buffer. Besides the buffer itself what other files should be on there so that a different instance with the same config file can load the buffer and send the rest of it. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 1

Mysql is missing log on my Ubuntu 18.04.4LTS
by Sync IT 23 Mar '20

23 Mar '20

Hi I have installed syslog-ng on a ubuntu machine. Mysql has been used to store the logs. But i keep seeing the logs are missed. that means logs are not storing properly. any idea what is wrong with it. it normally happened at the end of the day close to 12am. Kindly help. Thanks

3 2

Interested in contributing to syslog-ng
by Vivin Peris 20 Mar '20

20 Mar '20

I want to start contributing to syslog-ng organization and need a mentor for the same. I have worked extensively with apache kafka and would be interested in the project entailing the addition of support for template feature Please help me out with the same Thanks Sincerely Vivin [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=sig…> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=sig…> 03/20/20, 03:14:47 PM

2 1

MARK messages and filtering
by Alexandre Santos 19 Mar '20

19 Mar '20

Hi, I have the following part of syslog configuration: destination d_localfile_kernel { file("/var/log/kernel.log" flags(syslog-protocol) ); }; filter f_localfile_kernel_kern { facility(kern) and level(info .. emerg); }; filter f_localfile_kernel { filter(f_localfile_kernel_kern); }; log { source(s_src); filter(f_localfile_kernel); destination(d_localfile_kernel); flags(flow-control); }; And I see in my kernel.log file: <46>1 2020-03-18T18:35:28+00:00 localhost - - - [meta sequenceId="1"] -- MARK -- <46>1 2020-03-18T18:55:28+00:00 localhost - - - [meta sequenceId="2"] -- MARK -- <46>1 2020-03-18T19:15:28+00:00 localhost - - - [meta sequenceId="3"] -- MARK -- <46>1 2020-03-18T19:35:28+00:00 localhost - - - [meta sequenceId="4"] -- MARK -- <46>1 2020-03-18T19:55:28+00:00 localhost - - - [meta sequenceId="5"] -- MARK -- <46>1 2020-03-18T20:15:28+00:00 localhost - - - [meta sequenceId="6"] -- MARK -- <46>1 2020-03-18T20:35:28+00:00 localhost - - - [meta sequenceId="7"] -- MARK -- <46>1 2020-03-18T20:55:28+00:00 localhost - - - [meta sequenceId="8"] -- MARK -- <46>1 2020-03-18T21:15:28+00:00 localhost - - - [meta sequenceId="9"] -- MARK -- I was not expecting to see syslog facility messages coming out, since I am filtering by kernel facility. Is this expected behavior? Thanks & regards, Alex

2 1

Insider 2020-03: syslog-ng-stable; macOS; BastilleBSD;
by Peter Czanik (pczanik) 19 Mar '20

19 Mar '20

Dear syslog-ng users, This is the 79th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Introducing the syslog-ng-stable RPM repositories ------------------------------------------------- For many years – especially after syslog-ng changed to a rolling release model – users I talked to asked for up-to-date RPM packages. They also asked for a separate repository for each new release to avoid surprises (a new release might accidentally or even intentionally break old features) and to be able to use a given release if they want to (“if it works, do not fix it”). That is how my unofficial RPM repositories were born. Recently some long-time syslog-ng users and members of the Splunk community started to ask for a repository, which always has the latest syslog-ng version available. Most users still prefer to use separate repositories. That is how I came up with the idea for the syslog-ng-stable repository: I push a new release to this new rolling repo only after at least a week of delay. This is enough to spot most major problems. Once the delay is over and everything seems to be OK, I can push the latest release to the syslog-ng-stable repo. If there is a bigger problem, I can skip the release in the stable repo or wait for a fix. https://www.syslog-ng.com/community/b/blog/posts/introducing-the-syslog-ng-… How to install syslog-ng on macOS --------------------------------- A member of the syslog-ng community wrote a guide on how to enable syslog-ng on macOS: “My goal was to send any kind of Mac logs to a big data platform (Devo). Mac already comes with syslogd, which is the Apple System Log server. It is basically a daemon that processes syslog messages but to be honest, it’s pretty old and basic. As a result, I decided to try syslog-ng.” https://medium.com/macoclock/how-to-install-syslog-ng-on-macos-965127f1c05c Running syslog-ng in BastilleBSD -------------------------------- Bastille is a container management system for FreeBSD. If you are coming from a Linux world, it is a bit like Docker or Podman / Buildah from Red Hat, at least some of its functionality. I learned about BastilleBSD right before my Christmas holidays. Currently my primary work platform is Linux and I am just preparing to learn about Kubernetes and OpenShift. I planned not to do anything work-related during my holidays – which is quite difficult, if your hobby heavily overlaps with your work. Having some strong FreeBSD roots (started to use FreeBSD in 1994), BastilleBSD arrived just on time to be a good excuse to do something IT-related :-) https://www.syslog-ng.com/community/b/blog/posts/running-syslog-ng-in-basti… NEW WHITE PAPERS * Upgrade to State-of-the-Art Log Management: https://www.syslog-ng.com/whitepaper/log-management-essentials8137754/ WEBINARS * Learn how to easily integrate Azure Sentinel into your log-data strategy: https://www.brighttalk.com/webcast/16207/390780 * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

3.26 question
by Peter Czanik (pczanik) 18 Mar '20

18 Mar '20

Hi, Do you use syslog-ng 3.26? Before I add a new release to my syslog-ng-stable repo normally I check for feedback on the release. But here I have a problem. There are no bug reports about syslog-ng 3.26, which is good. On the other hand I could not find any reports that someone is already using 3.26. Do you have any feedback on 3.26? Either positive or negative? Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

use /dev/log on systemd machines
by Matus UHLAR - fantomas 18 Mar '20

18 Mar '20

Hello, if I understand properly, the system() source opens /dev/log socket when machine does not run systemd, am I right? Can I force syslog-ng to open /dev/log socket in this case or have I to configure it explicitly? -- Matus UHLAR - fantomas, uhlar(a)fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. I intend to live forever - so far so good.

2 1

cmake vs autotools
by Chris Packham 18 Mar '20

18 Mar '20

Hi, Question from the buildroot mailing list [1]. Is syslog-ng transitioning to cmake or will both cmake an autotools be supported for the foreseeable future. It's not an issue either way buildroot supports building with either cmake or autotools. But if autotools will be retired we may as well switch now to avoid hassles with future updates. Thanks, Chris -- [1] - http://lists.busybox.net/pipermail/buildroot/2020-March/277459.html

3 2

syslog-ng 3.26 RPM packages
by Peter Czanik (pczanik) 10 Mar '20

10 Mar '20

Hi, A bit slower than usual, but my unofficial syslog-ng 2.26 RPM packages are now available. I'm still traveling, so I did not have a chance for much testing. Any feedback is very welcome! * openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng326 * fedora/RHEL/CentOS: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng326/ Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Fwd: facility override
by Alexandre Santos 09 Mar '20

09 Mar '20

Hi, I tried to rewrite PRI but the macro is not writable: Mar 09 11:27:31 localhost syslog-ng[31986]: Error parsing rewrite expression, PRI is read-only, it cannot be changed in rewrite rules in /etc/syslog-ng/syslog-ng.conf: Mar 09 11:27:31 localhost syslog-ng[31986]: 359 }; Mar 09 11:27:31 localhost syslog-ng[31986]: 360 # end of localFile: netconf-commands Mar 09 11:27:31 localhost syslog-ng[31986]: 361 Mar 09 11:27:31 localhost syslog-ng[31986]: 362 Mar 09 11:27:31 localhost syslog-ng[31986]: 363 # remote: test Mar 09 11:27:31 localhost syslog-ng[31986]: 364---> rewrite r_pri{set("189", value("PRI"));}; Mar 09 11:27:31 localhost syslog-ng[31986]: 364---> ^ Mar 09 11:27:31 localhost syslog-ng[31986]: 365 destination d_test_udp { Mar 09 11:27:31 localhost syslog-ng[31986]: 366 syslog("192.168.122.1" transport("udp") ip-protocol(4) port(514) Mar 09 11:27:31 localhost syslog-ng[31986]: 367 suppress(5) Mar 09 11:27:31 localhost syslog-ng[31986]: 368 disk-buffer( Mar 09 11:27:31 localhost syslog-ng[31986]: 369 mem-buf-size(2097152) I tried to use the following configuration to make a facility override local0 -> local7: # remote: test template t_test { template("$(if ('${LEVEL_NUM}' == '0') '<184>1' $(if ('${LEVEL_NUM}' == '1') '<185>1' $(if ('${LEVEL_NUM}' == '2') '<186>1' $(if ('${LEVEL_NUM}' == '3') '<187>1' $(if ('${LEVEL_NUM}' == '4') '<188>1' $(if ('${LEVEL_NUM}' == '5') '<189>1' $(if ('${LEVEL_NUM}' == '6') '<190>1' '<191>1' ))))))) ${ISODATE} ${HOST} ${PROGRAM} ${PID} $(if ('${MSGID}' == '') '-' '${MSGID}') $(if ('${SDATA}' == '') '-' '${SDATA}') ${MESSAGE}"); }; destination d_test_udp { network("192.168.122.1" transport("udp") ip-protocol(4) port(514) template ("t_test") ); }; filter f_remote_test_udp_local0 { facility(local0) and level(debug .. emerg); }; log { source(s_src); filter(f_remote_test_udp_local0); destination(d_test_udp); flags(flow-control); }; # end of remote server: test The problem is that I get a double header in the syslog server: Mar 9 11:41:52 127.0.0.1 root[31353] <190>1 2020-03-09T11:41:52+00:00 127.0.0.1 root 31353 - - This is a Test 6 Mar 9 11:41:53 127.0.0.1 root[31404] <190>1 2020-03-09T11:41:53+00:00 127.0.0.1 root 31404 - - This is a Test 7 Mar 9 11:41:53 127.0.0.1 root[31448] <190>1 2020-03-09T11:41:53+00:00 127.0.0.1 root 31448 - - This is a Test 8 Mar 9 11:41:54 127.0.0.1 root[31449] <190>1 2020-03-09T11:41:54+00:00 127.0.0.1 root 31449 - - This is a Test 9 Mar 9 11:41:54 127.0.0.1 root[31468] <190>1 2020-03-09T11:41:54+00:00 127.0.0.1 root 31468 - - This is a Test 10 Mar 9 11:41:55 127.0.0.1 root[31485] <190>1 2020-03-09T11:41:55+00:00 127.0.0.1 root 31485 - - This is a Test 11 Mar 9 11:41:55 127.0.0.1 root[31526] <190>1 2020-03-09T11:41:55+00:00 127.0.0.1 root 31526 - - This is a Test 12 Mar 9 11:41:55 127.0.0.1 root[31567] <190>1 2020-03-09T11:41:55+00:00 127.0.0.1 root 31567 - - This is a Test 13 Can you help me? Thanks and regards, Alex On Mon, Jan 20, 2020 at 07:49 AM Attila Szakacs (aszakacs) < Attila.Szakacs(a)oneidentity.com> wrote: > Hi Alex, > > I think ${PRI} does exactly that. > > PRI > Description: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslog messages. > > Regards, > Attila > ________________________________ > From: syslog-ng <syslog-ng-bounces at lists.balabit.hu <https://lists.balabit.hu/mailman/listinfo/syslog-ng>> on behalf of Alexandre Santos <alexandre.rosas.santos at gmail.com <https://lists.balabit.hu/mailman/listinfo/syslog-ng>> > Sent: Friday, January 17, 2020 7:56 PM > To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu <https://lists.balabit.hu/mailman/listinfo/syslog-ng>> > Subject: [syslog-ng] facility override > > CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. > > Hi, > > Is there a way of overriding the facility of the messages on the destination, besides using a template like: > { template("$(if ('${LEVEL_NUM}' == '2') '<186>1' $(if ('${LEVEL_NUM}' == '3') '<187>1' $(if ('${LEVEL_NUM}' == '4') '<188>1' ${HOST}\\@`HOSTIPV4` ${PROGRAM} ${PID} ${SDATA} ${MESSAGE}\n"); }; > > Thanks in advance, > Alex >

2 2

restart vs reload
by Alexandre Santos 09 Mar '20

09 Mar '20

Hi, I am trying to dynamically change the output message format between RFC3164 and RFC5424. root@localhost:~# diff /etc/syslog-ng/syslog-ng.conf /root/syslog-ng.conf.bak 365c365 < network("192.168.122.1" transport("udp") ip-protocol(4) port(514) --- > syslog("192.168.122.1" transport("udp") ip-protocol(4) port(514) If I use: "/usr/sbin/service syslog-ng restart" it works fine, but if I use "/usr/sbin/syslog-ng-ctl reload", the message format does not change. I am using syslog-ng version 3.19 Any thoughts? Thanks & Regards, Alex

2 1

my unofficial 3.26 RPM packages delayed
by Peter Czanik (pczanik) 05 Mar '20

05 Mar '20

Hi, Just a heads up. I'm traveling and at my current location wifi is too slow and unstable to do package building. Once I have a stable wifi I'll first build them in my syslog-ng-githead repositories and once everything builds (I have to disable Python support on SLES 12 as Python 3.4 is too old) I'll create the usual versioned repositories. So lets hope, that wifi will work better at my next hotel / conference and then I can prepare my usual openSUSE/SLES & Fedora/RHEL/CentOS packages. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Not able to link syslog-ng libraries through gcc
by Nitish Saboo 05 Mar '20

05 Mar '20

Hi, I compiled syslog-ng-3.25.1 using the following code: tar -xvzf syslog-ng_3.25.1.tar.gz mkdir -p syslog-ng_3.25.1/build cd syslog-ng_3.25.1/build ../configure --prefix=/home/nsaboo/Desktop/Cproject/syslog/syslog-ng_3.25.1/install make make install Post installation,the folder structure looks like this: nsaboo@ubuntu:~/Desktop/Cproject/syslog/syslog-ng-3.25.1/install$ ls -lrt total 28 drwxrwxr-x 2 nsaboo nsaboo 4096 Feb 11 04:08 sbin drwxrwxr-x 2 nsaboo nsaboo 4096 Feb 11 04:08 bin drwxr-xr-x 2 nsaboo nsaboo 4096 Feb 11 04:08 var drwxr-xr-x 3 nsaboo nsaboo 4096 Feb 11 04:08 share drwxrwxr-x 4 nsaboo nsaboo 4096 Feb 11 04:08 include drwxr-xr-x 3 nsaboo nsaboo 4096 Feb 11 04:08 etc drwxrwxr-x 4 nsaboo nsaboo 4096 Feb 18 00:35 lib nsaboo@ubuntu:~/Desktop/Cproject/syslog/syslog-ng-3.25.1/install$ I have a C code base (syslog-node.c) and header file (node.h) that is internally calling syslog-ng patterndb apis': I have a main.c file that calls methods in 'syslog-node.c'. My header file has following methods: typedef void (*key_value_cb)(const char* key, const char* value, size_t value_len, int data); int initialize_engine(const char* filename, const char* module_path, key_value_cb cb); int reload_pattern_db(const char* filename, key_value_cb cb); int load_pattern_db(const char* filename, key_value_cb cb); void match(const char* pattern, size_t pattern_len, const char* program, size_t program_len); int shutdown(); I am calling following gcc command to create a binary but it is erroring out: gcc -L/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/lib lsyslog-ng -L/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/lib/syslog-ng ldbparser -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/include/syslog-ng -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/include -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/ -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/build/ -I/usr/local/include/eventlog/ syslog-node.c main.c -o main Error ===== nsaboo@ubuntu:~/Desktop/Cproject$ gcc -L/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/lib -lsyslog-ng -L/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/lib/syslog-ng -ldbparser -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/include/syslog-ng -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/install/include -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/ -I/home/nsaboo/Desktop/Cproject/syslog/syslog-ng-3.25.1/build/ -I/usr/local/include/eventlog/ syslog-node.c main.c -o main /tmp/ccam1ao5.o: In function `pdbtool_pdb_emit_accumulate': syslog-node.c:(.text+0x55): undefined reference to `logmsg_registry' syslog-node.c:(.text+0x73): undefined reference to `nv_table_foreach' /tmp/ccam1ao5.o: In function `match': syslog-node.c:(.text+0x93): undefined reference to `log_msg_new_empty' syslog-node.c:(.text+0xb0): undefined reference to `log_msg_set_value' syslog-node.c:(.text+0xc9): undefined reference to `log_msg_set_value' syslog-node.c:(.text+0xdf): undefined reference to `pattern_db_process' syslog-node.c:(.text+0xeb): undefined reference to `log_msg_unref' /tmp/ccam1ao5.o: In function `reload_pattern_db': syslog-node.c:(.text+0x1e2): undefined reference to `pattern_db_free' syslog-node.c:(.text+0x1e7): undefined reference to `pattern_db_new' syslog-node.c:(.text+0x20b): undefined reference to `pattern_db_reload_ruleset' syslog-node.c:(.text+0x225): undefined reference to `pattern_db_set_emit_func' /tmp/ccam1ao5.o: In function `load_pattern_db': syslog-node.c:(.text+0x257): undefined reference to `pattern_db_free' syslog-node.c:(.text+0x25c): undefined reference to `pattern_db_new' syslog-node.c:(.text+0x280): undefined reference to `pattern_db_reload_ruleset' syslog-node.c:(.text+0x29a): undefined reference to `pattern_db_set_emit_func' /tmp/ccam1ao5.o: In function `initialize_engine': syslog-node.c:(.text+0x2ba): undefined reference to `app_startup' syslog-node.c:(.text+0x2c4): undefined reference to `msg_init' syslog-node.c:(.text+0x2cb): undefined reference to `resolvedConfigurablePaths' syslog-node.c:(.text+0x2d0): undefined reference to `resolved_configurable_paths_init' syslog-node.c:(.text+0x2db): undefined reference to `resolvedConfigurablePaths' syslog-node.c:(.text+0x2e5): undefined reference to `cfg_new' syslog-node.c:(.text+0x2fb): undefined reference to `cfg_set_global_paths' syslog-node.c:(.text+0x311): undefined reference to `cfg_load_module' syslog-node.c:(.text+0x327): undefined reference to `cfg_load_module' syslog-node.c:(.text+0x32c): undefined reference to `pattern_db_global_init' syslog-node.c:(.text+0x347): undefined reference to `pattern_db_free' syslog-node.c:(.text+0x34c): undefined reference to `pattern_db_new' syslog-node.c:(.text+0x370): undefined reference to `pattern_db_reload_ruleset' syslog-node.c:(.text+0x38a): undefined reference to `pattern_db_set_emit_func' /tmp/ccam1ao5.o: In function `shutdown': syslog-node.c:(.text+0x39a): undefined reference to `app_shutdown' collect2: error: ld returned 1 exit status Those functions are available in the shared libraries.nm command output shows the same nsaboo@ubuntu:~/Desktop/Cproject$ nm syslog/syslog-ng-3.25.1/install/lib/syslog-ng/libdbparser.so | grep T | grep 'pattern_db_set_emit_func' 0000000000010d40 T pattern_db_set_emit_func nsaboo@ubuntu:~/Desktop/Cproject$ nm syslog/syslog-ng-3.25.1/install/lib/syslog-ng/libdbparser.so | grep T | grep 'pattern_db_new' 0000000000010ef0 T pattern_db_new nsaboo@ubuntu:~/Desktop/Cproject$ nm syslog/syslog-ng-3.25.1/install/lib/syslog-ng/libdbparser.so | grep T | grep 'pattern_db_reload_ruleset' 0000000000010cc0 T pattern_db_reload_ruleset nsaboo@ubuntu:~/Desktop/Cproject$ nm syslog/syslog-ng-3.25.1/install/lib/libsyslog-ng.so.0 | grep T | grep 'log_msg_set_value' 0000000000080df0 T log_msg_set_value 00000000000811d0 T __log_msg_set_value_by_name 0000000000081220 T log_msg_set_value_indirect nsaboo@ubuntu:~/Desktop/Cproject$ nm syslog/syslog-ng-3.25.1/install/lib/libsyslog-ng.so | grep T | grep 'msg_init' 0000000000051990 T msg_init nsaboo@ubuntu:~/Desktop/Cproject$ nm syslog/syslog-ng-3.25.1/install/lib/libsyslog-ng.so | grep T | grep 'cfg_new' 0000000000039200 T cfg_new 0000000000039360 T cfg_new_snippet 0000000000039390 T cfg_new_subordinate I understand this is the C linking error where I am including all the dynamic shared libraries and header files, but not able to figure out what is causing this issue. Can someone please help me to determine what am I missing here ? Thanks, Nitish

1 1

syslog-ng 3.26 Debian/Ubuntu packages (OBS)
by Laszlo Budai 05 Mar '20

05 Mar '20

Hi, syslog-ng 3.26.1[1] packages are available in OBS repositories[2, 3]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.26.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0