- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Message and Header are being split incorrectly
by Sass, Fabian 05 Jun '20

05 Jun '20

Hi Peter, thanks a lot, this resolved the issue without having to use hacky templates. Still it bugs me out that rewriting $PROGRAM didn't work. Any idea why? According to the other list posting it should've. Cheers, Fabian Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Peter Czanik (pczanik) Gesendet: Freitag, 5. Juni 2020 08:36 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Message and Header are being split incorrectly Hi, FreeBSD 12.1 changed from the legacy syslog protocol to RFC 5424 format. When you use the system() source for local logs, this is handled automagically. But you can also fix it by hand: https://github.com/syslog-ng/syslog-ng/issues/2428 Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com<mailto:peter.czanik@oneidentity.com>> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Sass, Fabian <Fabian.Sass(a)f-i-ts.de<mailto:Fabian.Sass@f-i-ts.de>> Sent: Friday, June 5, 2020 08:28 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Message and Header are being split incorrectly CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi, since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers. Jun 5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root) Jun 5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro. $MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line. However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update. The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost. The behavior seems to be analog to this bug: https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.bal…> As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test): @version: 3.5 # # options # options { mark_freq(3600); flush_lines(0); dir_perm(0640); chain_hostnames(off); keep_hostname(yes); create_dirs(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(3600); }; source s_all { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); }; # # rewrite since syslog message splitting is broken since update to freebsd12... # rewrite r_msg { set( "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*")) ); subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE")); }; # # destinations # destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); }; destination d_test2 { file("/var/log/fabian_messages2"); }; destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); }; destination d_loghost { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); }; # # # logging # log { source(s_all); rewrite(r_msg); destination(d_messages); }; log { source(s_all); rewrite(r_msg); destination(d_loghost); }; log { source(s_all); rewrite(r_msg); destination(d_test); }; log { source(s_all); destination(d_test2); }; Am I missing something here, or is syslog-ng somehow behaving unintended? Fabian

1 0

Message and Header are being split incorrectly
by Sass, Fabian 05 Jun '20

05 Jun '20

Hi, since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers. Jun 5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root) Jun 5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro. $MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line. However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update. The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost. The behavior seems to be analog to this bug: https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test): @version: 3.5 # # options # options { mark_freq(3600); flush_lines(0); dir_perm(0640); chain_hostnames(off); keep_hostname(yes); create_dirs(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(3600); }; source s_all { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); }; # # rewrite since syslog message splitting is broken since update to freebsd12... # rewrite r_msg { set( "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*")) ); subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE")); }; # # destinations # destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); }; destination d_test2 { file("/var/log/fabian_messages2"); }; destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); }; destination d_loghost { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); }; # # # logging # log { source(s_all); rewrite(r_msg); destination(d_messages); }; log { source(s_all); rewrite(r_msg); destination(d_loghost); }; log { source(s_all); rewrite(r_msg); destination(d_test); }; log { source(s_all); destination(d_test2); }; Am I missing something here, or is syslog-ng somehow behaving unintended? Fabian

2 1

build on MacOS
by Pal, Laszlo 01 Jun '20

01 Jun '20

Hi, For quite some time I'm trying to build syslog-ng on my MacBook and I'm always getting the following error. Is there any Mac user who can help me with this? Is there a plan to make syslog-ng available again in homebrew? ld: warning: ignoring file > ./modules/dbparser/.libs/libsyslog-ng-patterndb.a, building for > macOS-x86_64 but attempting to link with file built for unknown-unsupported > file format ( 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A 0x2F 0x20 0x20 0x20 > 0x20 0x20 0x20 0x20 ) > Undefined symbols for architecture x86_64: > "_pattern_db_debug_ruleset", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pattern_db_expire_state", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pattern_db_free", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_get_ruleset", referenced from: > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_global_init", referenced from: > _main in pdbtool-pdbtool.o > "_pattern_db_new", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_process", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > "_pattern_db_reload_ruleset", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > _pdbtool_dump in pdbtool-pdbtool.o > "_pattern_db_set_emit_func", referenced from: > _pdbtool_match in pdbtool-pdbtool.o > "_pdb_example_free", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_file_validate", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_get_filenames", referenced from: > _pdbtool_merge in pdbtool-pdbtool.o > "_pdb_rule_get_name", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_pdb_rule_set_load", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > _pdbtool_dictionary in pdbtool-pdbtool.o > "_pdb_sort_filenames", referenced from: > _pdbtool_merge in pdbtool-pdbtool.o > "_ptz_find_clusters", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_free", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_load_file", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_new", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_ptz_print_patterndb", referenced from: > _pdbtool_patternize in pdbtool-pdbtool.o > "_r_find_all_applicable_nodes", referenced from: > _pdbtool_test in pdbtool-pdbtool.o > "_r_find_node", referenced from: > _pdbtool_dump in pdbtool-pdbtool.o > _pdbtool_test in pdbtool-pdbtool.o > _pdbtool_dictionary in pdbtool-pdbtool.o > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [modules/dbparser/pdbtool/pdbtool] Error 1 > make[2]: *** Waiting for unfinished jobs.... > ld: warning: ignoring file > ./modules/diskq/.libs/libsyslog-ng-disk-buffer.a, building for macOS-x86_64 > but attempting to link with file built for macOS-x86_64 > Undefined symbols for architecture x86_64: > "_log_queue_disk_load_queue", referenced from: > _open_queue in dqtool.o > "_log_queue_disk_non_reliable_new", referenced from: > _open_queue in dqtool.o > "_log_queue_disk_reliable_new", referenced from: > _open_queue in dqtool.o > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) Thanks L

1 0

How to ensure that only my hosts log to my syslog server?
by Evan Rempel 29 May '20

29 May '20

We are starting to explore laptop logging which means that I have to open up firewalls to public networks as the laptops are moved around. Is there a way to ensure that only computers configured by my organization are able to connect to or send logs to my log server? I looked at "Mutual authentication using TLS" but if I understand that correctly the client is required to have a IP/hostname that matches the CN of the certificate. I couldn't find other information but perhaps I am searching for the wrong terms. -- Evan

2 2

Logs not making it to Splunk, high scratch buffers count
by Faine, Mark R. (MSFC-IS40)[NICS] 28 May '20

28 May '20

We've recently been seeing interruptions or delays to logs going into Splunk. I have noticed that global;scratch_buffers_count queued increases substantially and doesn't every seem to stop climbing. Here are the stats: SourceName;SourceId;SourceInstance;State;Type;Number destination;d_spol;;a;processed;0 src.internal;s_sys#2;;a;processed;384 src.internal;s_sys#2;;a;stamp;1590690126 center;;received;a;processed;1925133 destination;d_mesg;;a;processed;384 destination;d_mail;;a;processed;2 destination;d_auth;;a;processed;3 destination;d_mlal;;a;processed;0 center;;queued;a;processed;1962257 source;pan_splunk;;a;processed;1924743 destination;d_file;;a;processed;1561759 destination;d_cron;;a;processed;1 global;scratch_buffers_count;;a;queued;150323855375 destination;#anon-destination0;;a;processed;362983 global;payload_reallocs;;a;processed;5 global;sdata_updates;;a;processed;24 global;scratch_buffers_bytes;;a;queued;1536 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;0 source;s_sys;;a;processed;390 global;msg_clones;;a;processed;4662233 destination;d_file_no_location;;a;processed;37125 global;internal_queue_length;;a;processed;0 Here is the source configuration: source pan_splunk { network(port(514) transport("tcp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(port(514) transport("udp") max-connections(100) flags(syslog-protocol) log-fetch-limit(100) log-iw-size(100000) log-msg-size(10485760) ); network(transport("tls") port(10514) max-connections(100) flags(syslog-protocol) log-iw-size(100000) log-fetch-limit(100) log-msg-size(10485760) tls(peer-verify(optional-untrusted) key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; Here is the destination configuration destination d_file { file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log" create_dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750) owner("splunk") group("splunk") perm(0640) log-fifo-size(1000) disk-buffer( mem-buf-size(10000) disk-buf-size(2000000000) reliable(yes) dir("/var/log/remote/disk-buffer_dfile") ) flags("threaded", "no-multi-line") ); }; This includes some recent modifications to try and resolve the issue but nothing we have tried so far has made a difference. I would welcome any help you could provide. Thanks, -Mark

1 0

[SOLVED] Strange issue when forwarding using LEEF template
by Pal, Laszlo 28 May '20

28 May '20

Hi, I have a strange issue with templates... Maybe some stupid issue on my side, maybe a bug I don't know *<spoiler> Solutions at the end of this </spoiler>* Here is the scenario Logs coming from Windows using Syslog-ng Agent (syslog destination) Logs received using latest OSE (but same issues at the customer with PE6) Logs must be forwarded to Qradar using a special LEEF template like this (there are also tabs in the template). I've tried both network and syslog destination template t_leefwin { template("<${PRI}>${BSDDATE} ${HOST} LEEF:1.0|Microsoft|Windows|2k8r2|${EVENT_ID}| devTime=${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}GMT${TZOFFSET} devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat=${EVENT_TYPE} sev=${EVENT_LEVEL} resource=${HOST} usrName=${EVENT_USERNAME} application=${EVENT_SOURCE} message=${EVENT_MSG}\n"); }; The forwarded results like this <46> IP-address LEEF:1.0|Microsoft|Windows|2k8r2||devTime=2020-05-28T09:10:02GMT+02:00 devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat= sev= resource=172.17.24.174 usrName= application= message= So, the $macro-s are empty except the one which is not from .sdata (resource) I've created a local json and welf destination and I can see the macros there SDATA="[win(a)18372.4 EVENT_CATEGORY=\"None\" EVENT_FACILITY=\"16\" EVENT_ID=\"4098\" EVENT_LEVEL=\"3\" EVENT_NAME=\"Application\" EVENT_REC_NUM=\"73006\" EVENT_SID=\"S-1-5-18\" EVENT_SID_TYPE=\"User\" EVENT_SOURCE=\"Group Policy Services\" EVENT_TYPE=\"Figyelmeztetés\" EVENT_HOST=\"hostname\" EVENT_USERNAME=\"NT AUTHORITY\\\\SYSTEM\" I remember I used this method years ago to forward logs from syslog-ng agent to Qradar with proper LEEF format and it was ok... but now, some small thing is missing Versions syslog-ng 3 (3.25.1) Config version: 3.25 Installer-Version: 3.25.1 But, we have a similar issue with PE6 at the customer Any idea? --- so after playing with the welf output a bit, I figured out the short version of the macro names are not longer working somehow, but the long ones are ok like this template t_leefwin { template("<${PRI}>${BSDDATE} ${HOST} LEEF:1.0|Microsoft|Windows|2k8r2|${.SDATA.win@18372.4.EVENT_ID }|devTime=${R_YEAR}-${R_MONTH}-${R_DAY}T${R_HOUR}:${R_MIN}:${R_SEC}GMT${TZOFFSET} devTimeFormat=yyyy-MM-dd'T'HH:mm:ssz cat=${.SDATA.win(a)18372.4.EVENT_TYPE} sev=${.SDATA.win(a)18372.4.EVENT_LEVEL} resource=${SDATA.win(a)18372.4.EVENT_HOST} usrName=${.SDATA.win(a)18372.4.EVENT_USERNAME} application=${.SDATA.win(a)18372.4.EVENT_SOURCE} message=${MESSAGE}\n"); }; Thanks L:

2 2

High throughput UDP logging configuration.
by Evan Rempel 28 May '20

28 May '20

I know. There is a ton of information online about this. It shouldn't be that difficult, but I'm still having problems. I have one device with a very high logging rate. Approaching 15K msg/sec (7.5MB/sec). I am running on a 2 socket 8 core/CPU system (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz). Not the latest and greatest, but still fairly quick. All of the storage is on SSD. What I have configured 1. OS net.core.rmem_max = 536870912 2. so-reuseport(1) - and 8 sources on the UDP port 3. so_rcvbuf(67108864) - all 9 udp sources. 4. log_iw_size(2M) 5. log_fetch_limit(20k) 6. log_fifo_size(4M) Flow control is NOT enabled. Monitoring the queued messages inside syslog-ng and they remain near zero. Monitoring udp buffer queues in the OS shows that one stream is still overwhelming syslog-ng's ability to read messages. [~]$ sudo netstat -unlp|egrep -e 'PID|syslog' Proto Recv-Q Send-Q Local Address Foreign Addr State PID/Program name udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 134216320 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng udp 0 0 11.22.33.44:514 0.0.0.0:* 15591/syslog-ng A few things of note. 1. The OS UDP buffer seems to be 128MB in size and the so_rcvbuf configured ins 64M in size. Is that because the syslog-ng configuration of so_rcvbuf is in characters but the OS buffer is in bytes? 2. In a 1 second interval "cat /proc/net/udp" shows that one UDP stream dropped 6283 packets. So I'm continually dropping approx 50% of the UDP stream. 3. Increasing the log_iw_size or the log_iw_size actually seems to make things worse. All suggestions that help me understand this and help to minimize the drops are welcome. -- Evan Rempel

2 3

Forwarding to Elastic
by Shawn Taylor 28 May '20

28 May '20

I am running ES/Kibana 6.8.9-1 and am struggling with this issue. https://discuss.elastic.co/t/message-failed-to-find-message-in-kibana-logs/… I have added my index to the *Logs Indices* field in the Logs configuration. When I look at the fields in a document I see a field called MESSAGE, but not message. I do not see a way to add this field in the configuration. Is it possible to have this document display in the Logs UI? Can I convert the fields in syslog-ng to lowercase before forwarding them to elastic? Thanks, Shawn

3 4

Extract SNI on incoming connections
by Raghunath Adhyapak 23 May '20

23 May '20

Hi, I would like to run a syslog-ng server that accepts TLS connections. I want to identify clients based on the server name they use to connect and do specific processing for each client. I see the following macros being available to be used in filter, but I don't see .TLS.SN (or something like that to indicate server name used by client to connect) .TLS.X509 *Description:* When using a transport that uses TLS, these macros contain information about the peer's certificate. That way, you can use information from the client certificate in filenames, database values, or as other metadata. If you clients have their own certificates, then these values are unique per client, but unchangeable by the client. The following macros are available in syslog-ng OSE version 3.9 and later. - .TLS.X509_CN: The Common Name of the certificate. - .TLS.X509_O: The value of the Organization field. - .TLS.X509_OU: The value of the Organization Unit field. Thanks Raghu

2 2

Convert json format to syslog rfc5424
by Carlos Lopez 22 May '20

22 May '20

Hi all, Due to some issues with our logs collector server, I need to read some logs in json format and convert them to syslog rfc5424 and forward to our central syslog server. Acctually, all information regarding this type of configuration located in syslog-ng’s web site is related to send json format as a destination. But, is it possible to do the opposite: read a json file and forward to a third server in syslog rfc5424 format? Yes, I will try to catch up only some specific fields, not all fields contained in the json soure file. Any tips? Regards, C. L. Martinez

2 1