Hi Mark, I'm testing it with 3.19.1 and the following config: @version: 3.19 block destination default_file( basepath('/tmp') location("") app("") name("") file_path_args("") ) { file( "`basepath`/`location`/`app`/${HOST}/${HOST}_`app``file_path_args`.log" persist-name(`name`) create_dirs(yes) flags("threaded", "no-multi-line") ); }; destination d_default { default_file( basepath(`BASEPATH`) location("my_location") app("my_app") name('d_default_udp') file_path_args("${session}${some_other_arg}") ); }; destination d_default_2 { default_file( basepath(`BASEPATH`) location("my_location") app("my_app") name('d_default_udp_2') # file_path_args("${session}${some_other_arg}") ); }; log { source { example-msg-generator(); }; destination(d_default); destination(d_default_2); rewrite { set("my_session" value("session")); }; destination(d_default); }; These are the files generated: [09:59][ /tmp/my_location/my_app/alltilla-Precision-5530 ] $ ls -algh total 16K drwx------ 2 alltilla 4,0K máj 20 09:55 . drwx------ 3 alltilla 4,0K máj 20 09:35 .. -rw------- 1 alltilla 2,1K máj 20 09:57 alltilla-Precision-5530_my_app.log -rw------- 1 alltilla 910 máj 20 09:57 alltilla-Precision-5530_my_appmy_session.log And the syslog-ng -Fedtv output: [2021-05-20T09:57:55.503759] Setting value; name='MESSAGE', value='-- Generated message. --', msg='0x557306766d60' [2021-05-20T09:57:55.503808] Incoming generated message; msg='-- Generated message. --' [2021-05-20T09:57:55.503840] >>>>>> Source side message processing begin; instance='internal', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:39:12', msg='0x557306766d60' [2021-05-20T09:57:55.503864] Setting value; name='HOST_FROM', value='alltilla-Precision-5530', msg='0x557306766d60' [2021-05-20T09:57:55.503879] Setting value; name='HOST', value='alltilla-Precision-5530', msg='0x557306766d60' [2021-05-20T09:57:55.503901] Setting value; name='SOURCE', value='#anon-source0', msg='0x557306766d60' [2021-05-20T09:57:55.503986] Initializing destination file writer; template='/tmp/my_location/my_app/${HOST}/${HOST}_my_app${session}${some_other_arg}.log', filename='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log' [2021-05-20T09:57:55.504102] affile_open_file; path='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log', fd='12' [2021-05-20T09:57:55.504193] Initializing destination file writer; template='/tmp/my_location/my_app/${HOST}/${HOST}_my_app.log', filename='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log' [2021-05-20T09:57:55.504243] affile_open_file; path='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log', fd='13' [2021-05-20T09:57:55.504277] >>>>>> rewrite rule evaluation begin; rule='#anon-rewrite0', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:42:13', msg='0x557306766d60' [2021-05-20T09:57:55.504295] Message was cloned; original_msg='0x557306766d60', new_msg='0x55730675ec00' [2021-05-20T09:57:55.504306] Setting value; name='session', value='my_session', msg='0x55730675ec00' [2021-05-20T09:57:55.504322] <<<<<< rewrite rule evaluation finished; rule='#anon-rewrite0', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:42:13', msg='0x55730675ec00' [2021-05-20T09:57:55.504350] Initializing destination file writer; template='/tmp/my_location/my_app/${HOST}/${HOST}_my_app${session}${some_other_arg}.log', filename='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_appmy_session.log' [2021-05-20T09:57:55.504395] affile_open_file; path='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_appmy_session.log', fd='14' [2021-05-20T09:57:55.504421] <<<<<< Source side message processing finish; instance='internal', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:39:12', msg='0x557306766d60' [2021-05-20T09:57:55.504903] Outgoing message; message='May 20 09:57:55 alltilla-Precision-5530 -- Generated message. --\x0a' [2021-05-20T09:57:55.504984] Outgoing message; message='May 20 09:57:55 alltilla-Precision-5530 -- Generated message. --\x0a' [2021-05-20T09:57:55.505017] Outgoing message; message='May 20 09:57:55 alltilla-Precision-5530 -- Generated message. --\x0a' I do not have double quotes appended. Can you send me a similar config, which reproduces your issue? Thanks! Attila ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov> Sent: Wednesday, May 19, 2021 2:30 PM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] [EXTERNAL] Re: using VARARGS correctly CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Thanks, I didn't see anything about the "…" in the documentation. I'll take another look. Your suggestion is what I tried originally, however, the problem is that sometimes the argument is not wanted and I was trying to avoid multiple log paths with separate destinations. If I have a destination in which I sometimes do not add the session argument I get "" appended to my path. So, in circumstances where I do pass a session it works fine and the session number is appended, however, in cases where the session is not passed I get "" appended to the end of the log file name. I was trying to use VARARGS only as a way to work around that problem. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell) From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Attila Szakacs (aszakacs) Sent: Wednesday, May 19, 2021 01:40 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: [EXTERNAL] Re: [syslog-ng] using VARARGS correctly Hi Mark, VARARGS is used to pass unknown amount of options to the underlying driver through the block. For example: block destination ewmm(ip('127.0.0.1') transport(tcp) port(514) ...) { network("`ip`" transport(`transport`) port(`port`) template("$(format-ewmm)") frac-digits(3) `__VARARGS__` ); }; "..." and "__VARARGS​​__" must be used together. In the example above, any option given to the ewmm destination other than ip(), transport() and port(), is passed to the underlying network() destination. For your use case, I think a single option would suffice: block destination default_file( basepath('/var/log/remote/backup') location("") app("") name("") file_path_args("")) { file( "`basepath`/`location`/`app`/${HOST}/${HOST}_`app``file_path_args`.log" persist-name(`name`) create_dirs(yes) flags("threaded", "no-multi-line") ); }; destination d_default { default_file( basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') file_path_args("${session}${some_other_arg}") ); }; With the file_path_args() option you can set any number of optionally available macros in the order you like. If a macro is not available, it will resolve to empty string. Does this take care of your needs? Cheers, Attila ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov<mailto:mark.faine@nasa.gov>> Sent: Tuesday, May 18, 2021 8:00 PM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] using VARARGS correctly CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Syslog-ng 3.19 block destination d_default(basepath('/var/log/remote/backup') location("") app("") name("")) { file("`basepath`/`location`/`app`/${HOST}/${HOST}_`app``__VARARGS__`.log" persist-name(`name`) create_dirs(yes) flags("threaded", "no-multi-line")); }; However, when I call it like so: d_default(basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') "$session"); I get a syntax error when checking with --syntax-only Error parsing block reference, syntax error, unexpected LL_STRING, expecting ')' in /etc/syslog-ng/conf.d/splunk.conf: 23 categorize_loc(); 24 categorize_app(); 25 }; 26 }; 27 destination { 28----> d_default(basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') "$session"); 28----> ^^^^^^^^^^ From looking at the documentation, it looks like it's a valid way to use it. $session here is just a number from 0-9 that is set from rewriting a user defined macro from the message's sessionid field. It's only going to exist for a certain kind of message. Thanks, -Mark ______________________________________________________________________________ Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7Cd76c6bdac4c64ec8a0fd08d91a26d9ae%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637569576443135501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sPCS%2BJNfaqQRl0lODAcJx9vKtYk9W2nCvLhh73%2BS3Rg%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7C4bd6b9fe203b4c924fd408d91ac1d6f3%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637570242110426974%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=OJ1VrDSpxZ3cWMzp3TSj%2FD5AbPrDGTu3%2BjYi3mNtug8%3D&reserved=0> Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7Cd76c6bdac4c64ec8a0fd08d91a26d9ae%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637569576443135501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZntsaJqT%2Bh5EaUoQf8gb%2BF%2BdXy6LUIgkuLQzxKANitY%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7C4bd6b9fe203b4c924fd408d91ac1d6f3%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637570242110436963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9qI6ydL0hL36r5JhPlcQ4OfKUZFCw0WstgsazYfV6kY%3D&reserved=0> FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7Cd76c6bdac4c64ec8a0fd08d91a26d9ae%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637569576443135501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EvHK7TAM%2B8QsfR8pZBCYNiJ%2BolHT7b2wo3UMwPoorGM%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7C4bd6b9fe203b4c924fd408d91ac1d6f3%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637570242110436963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=N9W7jBvRDdmivJPsAiSAh7GhTHM0Ryylj%2Ft7ZEZW%2F3k%3D&reserved=0>

Hi Daniel, I think the problem is that these messages do not conform to a known syslog format, so syslog-ng does its best to guess which part of the message means what. I observed, that if adding changing "2021-05-19T1:14:14Z " to "2021-05-19T01:14:14Z" (so adding a 0 before the hour), it works as expected. However, doing this automatically is quite hard. If a log message comes with a specific format, we can write the parsing ourselves. There are multiple ways of doing that, I prepared one, which uses the match() filter with regexp and date-parser(). See my example config below: source s_src { network( transport("tcp") port(12378) flags(no-parse) ); }; filter f_regexp { match("^([^ ]+) ([^ ]+) ([^ ]+) (.*)$" template("${MESSAGE}") flags(store-matches)); }; rewrite r_set_regexp { set("$2" value("HOST")); set("$3" value("PROGRAM")); set("$4" value("MESSAGE")); }; parser p_date { date-parser(template("$1")); }; destination d_local { file("/dev/stdout"); }; log { source(s_src); filter(f_regexp); parser(p_date); rewrite(r_set_regexp); destination(d_local); }; The interesing parts are f_set_regexp, r_regexp, p_date and the flags(no-parse) in the source. In a nutshell: 1. The flags(no-parse) says, that we want to parse our incoming message manually, we do not need automatic message parsing. 2. The f_regexp filter matches the log for the pattern given, and stores the matches in $0, $1, $2... 3. The p_date parser parses the first found entry from f_regexp (that will be the date). 4. the r_set_regexp rewrite sets the rest of the regexp matches to the correct macros. This config correctly handled both of your messages for me. Please give it a try! Cheers, Attila ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> Sent: Thursday, May 20, 2021 1:35 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] using VARARGS correctly CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi Everyone, I am having an issue when the Zulu timestamp is between 10 and 23:59. i.e. the logs format differently before 10AM and after 10AM. I have captured in a tcpdump the syslogs coming in and they both seem the same. We're at GMT+10 so this event was as 11:14:14 on 19th May. Msg: 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d This event was 09:07:47 the next day the 20th May: Msg: 2021-05-19T23:07:46Z 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d In the output files, both events go to the 0519.log file, until 10AM or 00:00:00Z the next day. First event logs as: May 19 11:14:14 10.18.0.14 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O Second event logs as: May 19 23:07:46 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O I assume some built-in filtering is changing the way these are parsed in syslog-ng? I have tried to play with raw message filtering but it doesn't take the conf file: @version:3.5 @include "scl.conf" # syslog-ng configuration file. # options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); dns_cache(yes); keep_hostname(yes); log_fifo_size(2048); log_msg_size(8192); perm(0644); time_reopen (10); use_dns(yes); use_fqdn(yes); }; source s_network { udp(port(514)); }; source attivo { tcp(port(514)); }; ### DESTINATIONS destination d_files_splunk { file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes)); }; destination d_files_nti { file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes) template(t_nti)); }; ### FILTERS filter nti { host("10.18.0.14" type(glob)); }; filter splunk { not (filter(nti)); }; ### LOG log { source(s_network); #filter(splunk); destination(d_files_splunk); }; log { source(s_network); filter(nti); destination(d_files_nti); }; log { source(attivo); # filter(splunk); destination(d_files_splunk); }; ### TEMPLATES template t_nti { template("${RAWMSG}\n") }; Any help is appreciated. Thanks Daniel Ehrlich __________________________________________________________________ This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email. The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt. The University of Southern Queensland is a registered provider of education with the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)