is it exist some way to extract the filename listened by file() ?
Hi Team, Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload. My current approach is. 1. I have the below destination() receiving the file name as a parameter. <snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip> 2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn() <snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; }; log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; }; ......................... </snip> But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()? -- Jorge Pereira
Hi, It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from. With a quick search I didn't find it documented. On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Thanks so much!! exactly, I didn find it. -- Jorge Pereira On Sat, Apr 8, 2017 at 3:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from.
With a quick search I didn't find it documented.
On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Btw, somebody knows which the best way to extract only the file name? I mean something like the function basename() ? -- Jorge Pereira On Sat, Apr 8, 2017 at 3:50 AM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi,
Thanks so much!! exactly, I didn find it.
-- Jorge Pereira
On Sat, Apr 8, 2017 at 3:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from.
With a quick search I didn't find it documented.
On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I've just sent a PR to implement basename and dirname for your use-case: https://github.com/balabit/syslog-ng/pull/1420 You can probably apply the same to 3.7, but still you'd need to recompile from source. -- Bazsi On Sat, Apr 8, 2017 at 9:09 AM, Jorge Pereira <jpereiran@gmail.com> wrote:
Btw, somebody knows which the best way to extract only the file name? I mean something like the function basename() ?
-- Jorge Pereira
On Sat, Apr 8, 2017 at 3:50 AM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi,
Thanks so much!! exactly, I didn find it.
-- Jorge Pereira
On Sat, Apr 8, 2017 at 3:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from.
With a quick search I didn't find it documented.
On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, AFAIK, the FILE_NAME macro is only available in syslog-ng Premium Edition 6. CzP published a workaround a while back, that I never got to add to the official docs: https://czanik.blogs.balabit.com/2015/03/using-rfc5424-syslog-to-forward-fil... I'm not sure if it works in your case. Robert On Sat, Apr 8, 2017 at 8:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from.
With a quick search I didn't find it documented.
On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
yes! I am using and have worked perfectly! -- Jorge Pereira On Sat, Apr 8, 2017 at 5:07 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Hi,
AFAIK, the FILE_NAME macro is only available in syslog-ng Premium Edition 6. CzP published a workaround a while back, that I never got to add to the official docs: https://czanik.blogs.balabit.com/2015/03/using-rfc5424- syslog-to-forward-file-names/
I'm not sure if it works in your case.
Robert
On Sat, Apr 8, 2017 at 8:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from.
With a quick search I didn't find it documented.
On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I've checked the source and it is there. -- Bazsi On Sat, Apr 8, 2017 at 10:07 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Hi,
AFAIK, the FILE_NAME macro is only available in syslog-ng Premium Edition 6. CzP published a workaround a while back, that I never got to add to the official docs: https://czanik.blogs.balabit.com/2015/03/using-rfc5424- syslog-to-forward-file-names/
I'm not sure if it works in your case.
Robert
On Sat, Apr 8, 2017 at 8:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
It seems indeed ugly. We do have a FILE_NAME macro that gets set to the name of the file the message was read from.
With a quick search I didn't find it documented.
On Apr 8, 2017 07:27, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Hi Team,
Well, I am working on a POC using the syslog-ng 3.7.1, basically, I have many of log files that the filename is /path/<file> and I need to append the file name into the syslog payload.
My current approach is.
1. I have the below destination() receiving the file name as a parameter.
<snip> block destination d_collector_with_fn(__filename("")) { tcp("192.168.2.44" port(514) keep-alive(on) template("$DATE $HOST $MSGHDR $(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude SOURCEIP \ --exclude PROGRAM \ --pair SYSLOG_WEBAPP_DOMAIN='`__filename`' \ --pair SOURCE=${SOURCE} )\n") template-escape(no) ); }; </snip>
2. My simple script called by confgen create some dynamic "log {}" statements listening to the files and appending the filename as a parameter to the d_collector_with_fn()
<snip> log { source { file("/path/thisisafile001.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("thisisafile001.net")); }; };
log { source { file("caipirinha4ever.net" program_override("mytag") follow_freq(1) flags(no-parse) ); }; destination { d_collector_with_fn(__filename("caipirinha4ever.net")); }; };
......................... </snip>
But, I have more than 5k files and my current approach creating multiples log { } statement resulting in one connection to the collector by each file!!! in this case, I have 5k connections... this is terrible, someone has some other suggestion? exist some way to catch the filename by some internal ${variable} and pass for a single destination()?
-- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Fekete, Róbert
-
Jorge Pereira
-
Scheidler, Balázs