Re: [syslog-ng] Is it possible to include a external python script?
Hi Balázs, Thanks for the answer. But, My question is: It is possible to include a external python script. e.g: $ cat foobar.conf: python { @include "caipirinha.py" } template-function foobar "$(python foobar)" $ cat caipirinha.py from foobar import foobar_template_function # my python class/code is separated ........... $ got it? maybe using the "confgen" module? -- Jorge Pereira On Mon, Mar 20, 2017 at 2:35 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
The example Python code should be unindented, or you see a different error?
Also, you can always import code in the python block.
With destinations you can specify a path to a python file, so the python block is not needed. But with template functions I've figured the $(python) call would be too crowded if we were passing explicit filename parameters.
But you can encapsulate the python code as well as a wrapper to the python code into a separate file.
Foobar.conf:
python {
from foobar import foobar_template_function
}
template-function foobar "$(python foobar)"
----------
Then you can use:
@include "Foobar.conf"
... template("$(foobar)"); ...
Unfortunately then template function declarations cant take parameters yet from the callsite, which may be a serious limitation in your use-case. But with a conceot above you can completely hide implementation details your actual config.
On Mar 20, 2017 03:53, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Well, based on this[1] document. We should use the *python {}* statement. But, I would like to keep my python script externally in a separated file and do something like.
python filter_001 { /my/script/python001.py }
log { .... filter(filter_001); .... };
p.s: Btw, this sample https://www.balabit.com /documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose-guide- admin/html-single/index.html#example-python-template-functions don't work in the syslog-ng 3.9.1
[1] https://www.balabit.com/sites/default/files/documents/syslog -ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide-admin/html-si ngle/index.html#example-python-template-functions -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, No there's not, you can include one using the syslog-ng @include mechanism, but in that case the python block needs to be part of that file. And yes, you can achieve the same using the confgen module as well, but that would execute an external script. It probably wouldn't be very difficult to implement this syntax: python "python-script.py"; Which would execute that file instead of the python block (look at python-main.c and its evaluate_global_code function), but there's no such functionality now. Cheers, Bazsi -- Bazsi On Mon, Mar 20, 2017 at 5:34 PM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi Balázs,
Thanks for the answer. But, My question is: It is possible to include a external python script.
e.g:
$ cat foobar.conf:
python { @include "caipirinha.py" }
template-function foobar "$(python foobar)"
$ cat caipirinha.py from foobar import foobar_template_function # my python class/code is separated ...........
$
got it? maybe using the "confgen" module? -- Jorge Pereira
On Mon, Mar 20, 2017 at 2:35 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
The example Python code should be unindented, or you see a different error?
Also, you can always import code in the python block.
With destinations you can specify a path to a python file, so the python block is not needed. But with template functions I've figured the $(python) call would be too crowded if we were passing explicit filename parameters.
But you can encapsulate the python code as well as a wrapper to the python code into a separate file.
Foobar.conf:
python {
from foobar import foobar_template_function
}
template-function foobar "$(python foobar)"
----------
Then you can use:
@include "Foobar.conf"
... template("$(foobar)"); ...
Unfortunately then template function declarations cant take parameters yet from the callsite, which may be a serious limitation in your use-case. But with a conceot above you can completely hide implementation details your actual config.
On Mar 20, 2017 03:53, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Well, based on this[1] document. We should use the *python {}* statement. But, I would like to keep my python script externally in a separated file and do something like.
python filter_001 { /my/script/python001.py }
log { .... filter(filter_001); .... };
p.s: Btw, this sample https://www.balabit.com /documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose-guide-a dmin/html-single/index.html#example-python-template-functions don't work in the syslog-ng 3.9.1
[1] https://www.balabit.com/sites/default/files/documents/syslog -ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide-admin/html-si ngle/index.html#example-python-template-functions -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
hi Bazsi, yes! I saw the code and looks to be easy. btw, is it possible to rewrite the $MESSAGE var using python? I have tried the below code without success. python { import socket def resolve_host(logmsg, hostname): try: logmsg["MESSAGE"] = "My caipirinha!" return socket.gethostbyaddr(hostname)[0] except (socket.herror, socket.error): return 'unknown' }; destination d_python { file("/tmp/test.txt" template("@@@ ${ISODATE} # $(python resolve_host 200.221.2.44) # ${MESSAGE} \n")); }; -- Jorge Pereira On Tue, Mar 21, 2017 at 2:27 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
No there's not, you can include one using the syslog-ng @include mechanism, but in that case the python block needs to be part of that file. And yes, you can achieve the same using the confgen module as well, but that would execute an external script.
It probably wouldn't be very difficult to implement this syntax:
python "python-script.py";
Which would execute that file instead of the python block (look at python-main.c and its evaluate_global_code function), but there's no such functionality now.
Cheers, Bazsi
-- Bazsi
On Mon, Mar 20, 2017 at 5:34 PM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi Balázs,
Thanks for the answer. But, My question is: It is possible to include a external python script.
e.g:
$ cat foobar.conf:
python { @include "caipirinha.py" }
template-function foobar "$(python foobar)"
$ cat caipirinha.py from foobar import foobar_template_function # my python class/code is separated ...........
$
got it? maybe using the "confgen" module? -- Jorge Pereira
On Mon, Mar 20, 2017 at 2:35 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
The example Python code should be unindented, or you see a different error?
Also, you can always import code in the python block.
With destinations you can specify a path to a python file, so the python block is not needed. But with template functions I've figured the $(python) call would be too crowded if we were passing explicit filename parameters.
But you can encapsulate the python code as well as a wrapper to the python code into a separate file.
Foobar.conf:
python {
from foobar import foobar_template_function
}
template-function foobar "$(python foobar)"
----------
Then you can use:
@include "Foobar.conf"
... template("$(foobar)"); ...
Unfortunately then template function declarations cant take parameters yet from the callsite, which may be a serious limitation in your use-case. But with a conceot above you can completely hide implementation details your actual config.
On Mar 20, 2017 03:53, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Well, based on this[1] document. We should use the *python {}* statement. But, I would like to keep my python script externally in a separated file and do something like.
python filter_001 { /my/script/python001.py }
log { .... filter(filter_001); .... };
p.s: Btw, this sample https://www.balabit.com /documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose-guide-a dmin/html-single/index.html#example-python-template-functions don't work in the syslog-ng 3.9.1
[1] https://www.balabit.com/sites/default/files/documents/syslog -ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide-admin/html-si ngle/index.html#example-python-template-functions -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
In template functions you can't and should not do that. There's a python parser they was recently merged where you can change values. But you'll need the latest master for that. Bazsi On Mar 22, 2017 05:45, "Jorge Pereira" <jpereiran@gmail.com> wrote:
hi Bazsi,
yes! I saw the code and looks to be easy.
btw, is it possible to rewrite the $MESSAGE var using python? I have tried the below code without success.
python { import socket
def resolve_host(logmsg, hostname): try: logmsg["MESSAGE"] = "My caipirinha!" return socket.gethostbyaddr(hostname)[0] except (socket.herror, socket.error): return 'unknown'
};
destination d_python { file("/tmp/test.txt" template("@@@ ${ISODATE} # $(python resolve_host 200.221.2.44) # ${MESSAGE} \n")); };
-- Jorge Pereira
On Tue, Mar 21, 2017 at 2:27 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
No there's not, you can include one using the syslog-ng @include mechanism, but in that case the python block needs to be part of that file. And yes, you can achieve the same using the confgen module as well, but that would execute an external script.
It probably wouldn't be very difficult to implement this syntax:
python "python-script.py";
Which would execute that file instead of the python block (look at python-main.c and its evaluate_global_code function), but there's no such functionality now.
Cheers, Bazsi
-- Bazsi
On Mon, Mar 20, 2017 at 5:34 PM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi Balázs,
Thanks for the answer. But, My question is: It is possible to include a external python script.
e.g:
$ cat foobar.conf:
python { @include "caipirinha.py" }
template-function foobar "$(python foobar)"
$ cat caipirinha.py from foobar import foobar_template_function # my python class/code is separated ...........
$
got it? maybe using the "confgen" module? -- Jorge Pereira
On Mon, Mar 20, 2017 at 2:35 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
The example Python code should be unindented, or you see a different error?
Also, you can always import code in the python block.
With destinations you can specify a path to a python file, so the python block is not needed. But with template functions I've figured the $(python) call would be too crowded if we were passing explicit filename parameters.
But you can encapsulate the python code as well as a wrapper to the python code into a separate file.
Foobar.conf:
python {
from foobar import foobar_template_function
}
template-function foobar "$(python foobar)"
----------
Then you can use:
@include "Foobar.conf"
... template("$(foobar)"); ...
Unfortunately then template function declarations cant take parameters yet from the callsite, which may be a serious limitation in your use-case. But with a conceot above you can completely hide implementation details your actual config.
On Mar 20, 2017 03:53, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Well, based on this[1] document. We should use the *python {}* statement. But, I would like to keep my python script externally in a separated file and do something like.
python filter_001 { /my/script/python001.py }
log { .... filter(filter_001); .... };
p.s: Btw, this sample https://www.balabit.com /documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose-guide-a dmin/html-single/index.html#example-python-template-functions don't work in the syslog-ng 3.9.1
[1] https://www.balabit.com/sites/default/files/documents/syslog -ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide-admin/html-si ngle/index.html#example-python-template-functions -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks Bazsi! -- Jorge Pereira On Wed, Mar 22, 2017 at 1:57 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
In template functions you can't and should not do that. There's a python parser they was recently merged where you can change values.
But you'll need the latest master for that.
Bazsi
On Mar 22, 2017 05:45, "Jorge Pereira" <jpereiran@gmail.com> wrote:
hi Bazsi,
yes! I saw the code and looks to be easy.
btw, is it possible to rewrite the $MESSAGE var using python? I have tried the below code without success.
python { import socket
def resolve_host(logmsg, hostname): try: logmsg["MESSAGE"] = "My caipirinha!" return socket.gethostbyaddr(hostname)[0] except (socket.herror, socket.error): return 'unknown'
};
destination d_python { file("/tmp/test.txt" template("@@@ ${ISODATE} # $(python resolve_host 200.221.2.44) # ${MESSAGE} \n")); };
-- Jorge Pereira
On Tue, Mar 21, 2017 at 2:27 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
No there's not, you can include one using the syslog-ng @include mechanism, but in that case the python block needs to be part of that file. And yes, you can achieve the same using the confgen module as well, but that would execute an external script.
It probably wouldn't be very difficult to implement this syntax:
python "python-script.py";
Which would execute that file instead of the python block (look at python-main.c and its evaluate_global_code function), but there's no such functionality now.
Cheers, Bazsi
-- Bazsi
On Mon, Mar 20, 2017 at 5:34 PM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi Balázs,
Thanks for the answer. But, My question is: It is possible to include a external python script.
e.g:
$ cat foobar.conf:
python { @include "caipirinha.py" }
template-function foobar "$(python foobar)"
$ cat caipirinha.py from foobar import foobar_template_function # my python class/code is separated ...........
$
got it? maybe using the "confgen" module? -- Jorge Pereira
On Mon, Mar 20, 2017 at 2:35 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
The example Python code should be unindented, or you see a different error?
Also, you can always import code in the python block.
With destinations you can specify a path to a python file, so the python block is not needed. But with template functions I've figured the $(python) call would be too crowded if we were passing explicit filename parameters.
But you can encapsulate the python code as well as a wrapper to the python code into a separate file.
Foobar.conf:
python {
from foobar import foobar_template_function
}
template-function foobar "$(python foobar)"
----------
Then you can use:
@include "Foobar.conf"
... template("$(foobar)"); ...
Unfortunately then template function declarations cant take parameters yet from the callsite, which may be a serious limitation in your use-case. But with a conceot above you can completely hide implementation details your actual config.
On Mar 20, 2017 03:53, "Jorge Pereira" <jpereiran@gmail.com> wrote:
Well, based on this[1] document. We should use the *python {}* statement. But, I would like to keep my python script externally in a separated file and do something like.
python filter_001 { /my/script/python001.py }
log { .... filter(filter_001); .... };
p.s: Btw, this sample https://www.balabit.com /documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose-guide-a dmin/html-single/index.html#example-python-template-functions don't work in the syslog-ng 3.9.1
[1] https://www.balabit.com/sites/default/files/documents/syslog -ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide-admin/html-si ngle/index.html#example-python-template-functions -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Wed, Mar 22, 2017 at 05:57:04AM +0100, Scheidler, Balázs wrote:
In template functions you can't and should not do that. There's a python parser they was recently merged where you can change values.
But you'll need the latest master for that.
If you can't wait for the next release, there's a trick: use your template python code to generate JSON, then chain it using the json-parser :-)
Or, if you use one of the RPM distributions, I have a repo compiled once in a while from git head. You can read more about it at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ A couple of python parser related bug fixes were merged recently, so I try to update it in the coming days (I'm on sick leave right now...). Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Wed, Mar 22, 2017 at 7:11 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Mar 22, 2017 at 05:57:04AM +0100, Scheidler, Balázs wrote:
In template functions you can't and should not do that. There's a python parser they was recently merged where you can change values.
But you'll need the latest master for that.
If you can't wait for the next release, there's a trick: use your template python code to generate JSON, then chain it using the json-parser :-)
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Czanik, Péter
-
Fabien Wernli
-
Jorge Pereira
-
Scheidler, Balázs