[GSoC] Interested in Python support for Correlation Library
Hi, I'm Akshay, computer science student from India. I have decent experience with Python and system programming. Being a Mozillian, I have always been a big fan of Rust and Servo project. I have previously dabbled with Rust – and think of it as next system programming language. I hope to improve my Rust skills as part of this project and write production quality code. I have also been Google Summer of Code student at Inclusive Design Institute in 2013 and Mozilla in 2014. Few questions and ideas related to the project - - While the reason to switch away from handlebars makes sense - Why do we want to using Python as templating language? What are some of the use cases, where Python could be beneficial over any other templating language? - Are there any existing projects using Python as templating language? How is this going to be different than passing the context data to a Python script? - For the SQL-like syntax mentioned in project description - Can we design it as Python library using Rust via CFFI [0]. I have filled the student application form. I'm already trying to understand the correlation library and playing around with existing tests. How can I get more involved and better understand the requirements of the project? [0] - https://github.com/tremolo/rust_python_integration -- - Akshay Aurora ᐧ
Hi, See my answers inline.
- While the reason to switch away from handlebars makes sense - Why do we want to using Python as templating language? What are some of the use cases, where Python could be beneficial over any other templating language?
Users can freely write their own functions. We can provide some functions either in Rust or Python (it should be transparent to the users), but we can't cover everything. I know some people who would definitely write their own extensions. And the Python language is a way more powerfool tool than a simple templating language. - Are there any existing projects using Python as templating language? How
is this going to be different than passing the context data to a Python script?
Maybe the "templating language" term I used in the project description was a little bit ambigious. I'd like to use Python as a general purpose language to process the context data and generate a new log message from it. I think embedding Python (or Lua) into softwares written in other languages is quite common. I'd prefer this [0] library for the Rust <-> Python interaction. I used it for a side project [1], it's quite flexible and actively developed/maintained. You can check the examples in the documentation and the tests in the source code. [0] https://github.com/dgrunwald/rust-cpython [1] https://github.com/ihrwein/python-parser I'd like to see the final solution to be "composable". I mean, by providing small, reusable components users can compose the generated log message as they want. For example, the following line can generate a log message: logmsg(ctx, "context name is {context.name}, max(foo)={max_foo}", foo="bar", quz="baz", max_foo=max(ctx, "foo")) where ctx contains the context data, logmsg and max is a python/Rust function. The syntax isn't settled down :)
- For the SQL-like syntax mentioned in project description - Can we design
it as Python library using Rust via CFFI [0].
see my previous answer
I have filled the student application form. I'm already trying to understand the correlation library and playing around with existing tests. How can I get more involved and better understand the requirements of the project?
First, I'd check what information is available in the contexts (your context data): https://github.com/ihrwein/correlation/blob/master/src/config/action/message... You you can use {{context_len}} to get the number of messages in the context, etc... IMHO the hardest part is using the [0] libray. It's well designed, but if you are new to Rust you may find it difficult to use. So I'd try that first. I don't know about any bugs in the correlation library, but you may try to refactor some code (just to see the internals). The requirements are listed in the idea page as "Tasks". BR, Tibor
Hi Tibor, Thanks for replying, I'll try and get my hands dirty with rust-cpython. - As a user looking for a solution for correlating messages, Python as template language definitely makes a lot of sense. It allows us to offer a more complete solution - allowing user to write their own utilities while we can also provide a good helper library piggybacking on Rust. - Whats you take on a generic script solution that takes context information as arguments, this allows user to use language of his choice. Is performance the only concern here? This shouldn't be very difficult to implement. Are we expecting, users to still write python code (template) in JSON? It might get ugly defining functions within JSON. We could probably consider giving users config option to include their Python module. ᐧ On Sun, Mar 20, 2016 at 8:01 PM, Tibor Benke <ihrwein@gmail.com> wrote:
Hi,
See my answers inline.
- While the reason to switch away from handlebars makes sense - Why do we want to using Python as templating language? What are some of the use cases, where Python could be beneficial over any other templating language?
Users can freely write their own functions. We can provide some functions either in Rust or Python (it should be transparent to the users), but we can't cover everything. I know some people who would definitely write their own extensions.
And the Python language is a way more powerfool tool than a simple templating language.
- Are there any existing projects using Python as templating language? How
is this going to be different than passing the context data to a Python script?
Maybe the "templating language" term I used in the project description was a little bit ambigious. I'd like to use Python as a general purpose language to process the context data and generate a new log message from it.
I think embedding Python (or Lua) into softwares written in other languages is quite common.
I'd prefer this [0] library for the Rust <-> Python interaction. I used it for a side project [1], it's quite flexible and actively developed/maintained. You can check the examples in the documentation and the tests in the source code.
[0] https://github.com/dgrunwald/rust-cpython [1] https://github.com/ihrwein/python-parser
I'd like to see the final solution to be "composable". I mean, by providing small, reusable components users can compose the generated log message as they want. For example, the following line can generate a log message:
logmsg(ctx, "context name is {context.name}, max(foo)={max_foo}", foo="bar", quz="baz", max_foo=max(ctx, "foo"))
where ctx contains the context data, logmsg and max is a python/Rust function. The syntax isn't settled down :)
- For the SQL-like syntax mentioned in project description - Can we design
it as Python library using Rust via CFFI [0].
see my previous answer
I have filled the student application form. I'm already trying to understand the correlation library and playing around with existing tests. How can I get more involved and better understand the requirements of the project?
First, I'd check what information is available in the contexts (your context data): https://github.com/ihrwein/correlation/blob/master/src/config/action/message...
You you can use {{context_len}} to get the number of messages in the context, etc...
IMHO the hardest part is using the [0] libray. It's well designed, but if you are new to Rust you may find it difficult to use. So I'd try that first. I don't know about any bugs in the correlation library, but you may try to refactor some code (just to see the internals).
The requirements are listed in the idea page as "Tasks".
BR, Tibor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- - Akshay Aurora
- Whats you take on a generic script solution that takes context information as arguments, this allows user to use language of his choice. Is performance the only concern here? This shouldn't be very difficult to implement.
The context information can be really huge (thousands of messages) so I'd be best to pass around this information without any heavy conversation. A generic script solution would be very inefficient in this case.
Are we expecting, users to still write python code (template) in JSON? It might get ugly defining functions within JSON. We could probably consider giving users config option to include their Python module.
That's a good idea! Note, that we could support even YAML configurations, if we upgrade the serde dependency to 0.7 (this is also a possible way of contribution to the project).
participants (2)
-
Akshay Aurora
-
Tibor Benke