Re: [syslog-ng] [EXTERNAL] Re: using VARARGS correctly

20 May 2021

      Hi Mark,

I'm testing it with 3.19.1 and the following config:
@version: 3.19

block destination default_file(
  basepath('/tmp')
  location("")
  app("")
  name("")
  file_path_args("")
) {
  file(
    "`basepath`/`location`/`app`/${HOST}/${HOST}_`app``file_path_args`.log"
    persist-name(`name`)
    create_dirs(yes)
    flags("threaded", "no-multi-line")
  );
};

destination d_default {
  default_file(
    basepath(`BASEPATH`)
    location("my_location")
    app("my_app")
    name('d_default_udp')
    file_path_args("${session}${some_other_arg}")
  );
};

destination d_default_2 {
  default_file(
    basepath(`BASEPATH`)
    location("my_location")
    app("my_app")
    name('d_default_udp_2')
    # file_path_args("${session}${some_other_arg}")
  );
};

log {
  source { example-msg-generator(); };
  destination(d_default);
  destination(d_default_2);
  rewrite { set("my_session" value("session")); };
  destination(d_default);
};

These are the files generated:
[09:59][ /tmp/my_location/my_app/alltilla-Precision-5530 ] $ ls -algh
total 16K
drwx------ 2 alltilla 4,0K máj   20 09:55 .
drwx------ 3 alltilla 4,0K máj   20 09:35 ..
-rw------- 1 alltilla 2,1K máj   20 09:57 alltilla-Precision-5530_my_app.log
-rw------- 1 alltilla  910 máj   20 09:57 alltilla-Precision-5530_my_appmy_session.log

And the syslog-ng -Fedtv output:
[2021-05-20T09:57:55.503759] Setting value; name='MESSAGE', value='-- Generated message. --', msg='0x557306766d60'
[2021-05-20T09:57:55.503808] Incoming generated message; msg='-- Generated message. --'
[2021-05-20T09:57:55.503840] >>>>>> Source side message processing begin; instance='internal', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:39:12', msg='0x557306766d60'
[2021-05-20T09:57:55.503864] Setting value; name='HOST_FROM', value='alltilla-Precision-5530', msg='0x557306766d60'
[2021-05-20T09:57:55.503879] Setting value; name='HOST', value='alltilla-Precision-5530', msg='0x557306766d60'
[2021-05-20T09:57:55.503901] Setting value; name='SOURCE', value='#anon-source0', msg='0x557306766d60'
[2021-05-20T09:57:55.503986] Initializing destination file writer; template='/tmp/my_location/my_app/${HOST}/${HOST}_my_app${session}${some_other_arg}.log', filename='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log'
[2021-05-20T09:57:55.504102] affile_open_file; path='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log', fd='12'
[2021-05-20T09:57:55.504193] Initializing destination file writer; template='/tmp/my_location/my_app/${HOST}/${HOST}_my_app.log', filename='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log'
[2021-05-20T09:57:55.504243] affile_open_file; path='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_app.log', fd='13'
[2021-05-20T09:57:55.504277] >>>>>> rewrite rule evaluation begin; rule='#anon-rewrite0', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:42:13', msg='0x557306766d60'
[2021-05-20T09:57:55.504295] Message was cloned; original_msg='0x557306766d60', new_msg='0x55730675ec00'
[2021-05-20T09:57:55.504306] Setting value; name='session', value='my_session', msg='0x55730675ec00'
[2021-05-20T09:57:55.504322] <<<<<< rewrite rule evaluation finished; rule='#anon-rewrite0', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:42:13', msg='0x55730675ec00'
[2021-05-20T09:57:55.504350] Initializing destination file writer; template='/tmp/my_location/my_app/${HOST}/${HOST}_my_app${session}${some_other_arg}.log', filename='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_appmy_session.log'
[2021-05-20T09:57:55.504395] affile_open_file; path='/tmp/my_location/my_app/alltilla-Precision-5530/alltilla-Precision-5530_my_appmy_session.log', fd='14'
[2021-05-20T09:57:55.504421] <<<<<< Source side message processing finish; instance='internal', location='/home/alltilla/Work/install/OSE-3.19/etc/syslog-ng.conf:39:12', msg='0x557306766d60'
[2021-05-20T09:57:55.504903] Outgoing message; message='May 20 09:57:55 alltilla-Precision-5530 -- Generated message. --\x0a'
[2021-05-20T09:57:55.504984] Outgoing message; message='May 20 09:57:55 alltilla-Precision-5530 -- Generated message. --\x0a'
[2021-05-20T09:57:55.505017] Outgoing message; message='May 20 09:57:55 alltilla-Precision-5530 -- Generated message. --\x0a'

I do not have double quotes appended. Can you send me a similar config, which reproduces your issue?

Thanks!
Attila
________________________________
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov>
Sent: Wednesday, May 19, 2021 2:30 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: Re: [syslog-ng] [EXTERNAL] Re: using VARARGS correctly

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Thanks, I didn't see anything about the "…" in the documentation.  I'll take another look.  Your suggestion is what I tried originally, however, the problem is that sometimes the argument is not wanted and I was trying to avoid multiple log paths with separate destinations.  If I have a destination in which I sometimes do not add the session argument I get "" appended to my path.

So, in circumstances where I do pass a session it works fine and the session number is appended, however, in cases where the session is not passed I get "" appended to the end of the log file name.   I was trying to use VARARGS only as a way to work around that problem.

Thanks,

-Mark

Mark Faine

System Administrator

SAIC/NICS

215 Wynn Dr. 5065

Huntsville, AL 35805

256-961-1295 (Desk)

256-617-4861 (Work Cell)

From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Attila Szakacs (aszakacs)
Sent: Wednesday, May 19, 2021 01:40
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: [EXTERNAL] Re: [syslog-ng] using VARARGS correctly

Hi Mark,

VARARGS is used to pass unknown amount of options to the underlying driver through the block.

For example:

block destination ewmm(ip('127.0.0.1') transport(tcp) port(514) ...) {

        network("`ip`" transport(`transport`) port(`port`)

                template("$(format-ewmm)")

                frac-digits(3)

                `__VARARGS__`

        );

};

"..." and "__VARARGS__" must be used together.

In the example above, any option given to the ewmm destination other than ip(), transport() and port(), is passed to the underlying network() destination.

For your use case, I think a single option would suffice:

block destination default_file(

  basepath('/var/log/remote/backup')

  location("")

  app("")

  name("")

  file_path_args("")) {

  file(

    "`basepath`/`location`/`app`/${HOST}/${HOST}_`app``file_path_args`.log"

    persist-name(`name`)

    create_dirs(yes)

    flags("threaded", "no-multi-line")

  );

};

destination d_default {

  default_file(

    basepath(`BASEPATH`)

    location("$location")

    app("$app")

    name('d_default_udp')

    file_path_args("${session}${some_other_arg}")

  );

};

With the file_path_args() option you can set any number of optionally available macros in the order you like. If a macro is not available, it will resolve to empty string.

Does this take care of your needs?

Cheers,

Attila

________________________________

From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov<mailto:mark.faine@nasa.gov>>
Sent: Tuesday, May 18, 2021 8:00 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>
Subject: [syslog-ng] using VARARGS correctly

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Syslog-ng 3.19

block destination d_default(basepath('/var/log/remote/backup') location("") app("") name("")) {
    file("`basepath`/`location`/`app`/${HOST}/${HOST}_`app``__VARARGS__`.log"
    persist-name(`name`) create_dirs(yes)
    flags("threaded", "no-multi-line"));
};

However, when I call it like so:

d_default(basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') "$session");

I get a syntax error when checking with --syntax-only

Error parsing block reference, syntax error, unexpected LL_STRING, expecting ')' in /etc/syslog-ng/conf.d/splunk.conf:
23                  categorize_loc();
24                  categorize_app();
25              };
26          };
27          destination {
28---->       d_default(basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') "$session");
28---->                                                                                    ^^^^^^^^^^

From looking at the documentation, it looks like it's a valid way to use it.  $session here is just a number from 0-9 that is set from rewriting a user defined macro from the message's sessionid field.  It's only going to exist for a certain kind of message.

Thanks,
-Mark
______________________________________________________________________________
Member info: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7Cd76c6bdac4c64ec8a0fd08d91a26d9ae%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637569576443135501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sPCS%2BJNfaqQRl0lODAcJx9vKtYk9W2nCvLhh73%2BS3Rg%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7C4bd6b9fe203b4c924fd408d91ac1d6f3%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637570242110426974%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=OJ1VrDSpxZ3cWMzp3TSj%2FD5AbPrDGTu3%2BjYi3mNtug8%3D&reserved=0>
Documentation: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7Cd76c6bdac4c64ec8a0fd08d91a26d9ae%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637569576443135501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZntsaJqT%2Bh5EaUoQf8gb%2BF%2BdXy6LUIgkuLQzxKANitY%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7C4bd6b9fe203b4c924fd408d91ac1d6f3%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637570242110436963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9qI6ydL0hL36r5JhPlcQ4OfKUZFCw0WstgsazYfV6kY%3D&reserved=0>
FAQ: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7CAttila.Szakacs%40oneidentity.com%7Cd76c6bdac4c64ec8a0fd08d91a26d9ae%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637569576443135501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EvHK7TAM%2B8QsfR8pZBCYNiJ%2BolHT7b2wo3UMwPoorGM%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cattila.szakacs%40oneidentity.com%7C4bd6b9fe203b4c924fd408d91ac1d6f3%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637570242110436963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=N9W7jBvRDdmivJPsAiSAh7GhTHM0Ryylj%2Ft7ZEZW%2F3k%3D&reserved=0>