Hi Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code? Cheers, m.
Hi Marc, We have a Getting Started Guide in a very early stage, you can find it here: https://www.gitbook.com/book/syslog-ng/getting-started/details Can I ask you about what kind of module you want to develop? Cheers, Tibor 2016-01-22 11:03 GMT+01:00 Marc Falzon <m@baha.mu>:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Tibor On Friday 22 January 2016 at 11:23, Tibor Benke wrote:
Hi Marc,
We have a Getting Started Guide in a very early stage, you can find it here: https://www.gitbook.com/book/syslog-ng/getting-started/details
Great, thank you.
Can I ask you about what kind of module you want to develop?
I'm evaluating the possibility of developing a native C HTTP simple destination using libcurl, since I find the Java-based HTTP module too overkill for my use case. From what I've seen so far from the existing code base, it would be in the same spirit of the `afsmtp` module. Cheers, m.
2016-01-22 11:03 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :) if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough, -- Bazsi On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Bazsi, On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved. m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935... 2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:
m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935...
Awesome, thank you! m.
2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:m@baha.mu) (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I successfully generated a module template using the `create_plugin.sh` script, however I don't have the slightest idea on how to get started from here (I mean for the 'plugin glue', I can manage the cURL part)... The most problematic step for me is the whole parser/grammar thing, to which I believe to involve YACC/Bison parsing but I don't know how it works and it's too big of a learning curve for what I want to achieve at this point... Since the Gitbook doesn't provide any useful information on module development at the moment, is there any way to get me started on this topic? I'm not a developer (sysadmin), and trying to learn from existing module doesn't quite cut it. Thank you, m. On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935...
2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:m@baha.mu) (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
If you commit your code to a branch I can help with the basic skeleton of the parser/grammar On Jan 23, 2016 5:54 PM, "Marc Falzon" <m@baha.mu> wrote:
I successfully generated a module template using the `create_plugin.sh` script, however I don't have the slightest idea on how to get started from here (I mean for the 'plugin glue', I can manage the cURL part)... The most problematic step for me is the whole parser/grammar thing, to which I believe to involve YACC/Bison parsing but I don't know how it works and it's too big of a learning curve for what I want to achieve at this point... Since the Gitbook doesn't provide any useful information on module development at the moment, is there any way to get me started on this topic? I'm not a developer (sysadmin), and trying to learn from existing module doesn't quite cut it.
Thank you,
m.
On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935...
2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:
m@baha.mu) (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Jan 23, 2016, at 21:23, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
If you commit your code to a branch I can help with the basic skeleton of the parser/grammar
Thank you Balázs, my code is available here: https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl m.
On Jan 23, 2016 5:54 PM, "Marc Falzon" <m@baha.mu> wrote: I successfully generated a module template using the `create_plugin.sh` script, however I don't have the slightest idea on how to get started from here (I mean for the 'plugin glue', I can manage the cURL part)... The most problematic step for me is the whole parser/grammar thing, to which I believe to involve YACC/Bison parsing but I don't know how it works and it's too big of a learning curve for what I want to achieve at this point... Since the Gitbook doesn't provide any useful information on module development at the moment, is there any way to get me started on this topic? I'm not a developer (sysadmin), and trying to learn from existing module doesn't quite cut it.
Thank you,
m.
On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935...
2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:m@baha.mu) (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I'll check that out, sorry I've been very distracted yesterday and even today. On Jan 23, 2016 9:27 PM, <m@baha.mu> wrote:
On Jan 23, 2016, at 21:23, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
If you commit your code to a branch I can help with the basic skeleton of the parser/grammar
Thank you Balázs, my code is available here: https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl
m.
On Jan 23, 2016 5:54 PM, "Marc Falzon" <m@baha.mu> wrote: I successfully generated a module template using the `create_plugin.sh`
script, however I don't have the slightest idea on how to get started from here (I mean for the 'plugin glue', I can manage the cURL part)... The most problematic step for me is the whole parser/grammar thing, to which I believe to involve YACC/Bison parsing but I don't know how it works and it's too big of a learning curve for what I want to achieve at this point... Since the Gitbook doesn't provide any useful information on module development at the moment, is there any way to get me started on this topic? I'm not a developer (sysadmin), and trying to learn from existing module doesn't quite cut it.
Thank you,
m.
On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
There is also a shell script which generates a basic destination
skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935...
2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source
code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is
preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a
threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think
I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:
m@baha.mu) (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I have added code to your module, it compiles and is able to accept messages to be sent, and prints a debug message whenever that happens. I didn't have too much time to clean it up, but I hope this helps to start the ball rolling. https://github.com/balabit/syslog-ng/compare/f/curl-module Bazsi -- Bazsi On Sat, Jan 23, 2016 at 9:27 PM, <m@baha.mu> wrote:
On Jan 23, 2016, at 21:23, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
If you commit your code to a branch I can help with the basic skeleton of the parser/grammar
Thank you Balázs, my code is available here: https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl
m.
On Jan 23, 2016 5:54 PM, "Marc Falzon" <m@baha.mu> wrote: I successfully generated a module template using the `create_plugin.sh`
script, however I don't have the slightest idea on how to get started from here (I mean for the 'plugin glue', I can manage the cURL part)... The most problematic step for me is the whole parser/grammar thing, to which I believe to involve YACC/Bison parsing but I don't know how it works and it's too big of a learning curve for what I want to achieve at this point... Since the Gitbook doesn't provide any useful information on module development at the moment, is there any way to get me started on this topic? I'm not a developer (sysadmin), and trying to learn from existing module doesn't quite cut it.
Thank you,
m.
On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
There is also a shell script which generates a basic destination
skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c90620935...
2016-01-22 11:33 GMT+01:00 Marc Falzon <m@baha.mu (mailto:m@baha.mu)>:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source
code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is
preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a
threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think
I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:
m@baha.mu) (mailto:m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Well, I think libcurl only has a synchronous interface, at least last I've checked. -- Bazsi On Fri, Jan 22, 2016 at 11:33 AM, Marc Falzon <m@baha.mu> wrote:
Hi Bazsi,
On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
m.
On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m@baha.mu (mailto:
m@baha.mu)> wrote:
Hi
Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
Cheers,
m.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
❦ 22 janvier 2016 11:28 +0100, "Scheidler, Balázs" <balazs.scheidler@balabit.com> :
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Oh, there is a way to have an asynchronous implementation? Which module would be a simple but correct example of this? -- Never laugh at live dragons. -- Bilbo Baggins [J.R.R. Tolkien, "The Hobbit"]
well, the afsocket one. that's a completely nonblocking destination that implements udp/tcp/unix-stream/unix-dgram/syslog/network sources and destinations. It basically boils down to using a LogWriter class that does the heavy lifting at least as long as your transport protocol is simple enough. Probably the simplest is to create a LogProto implementation and let the afsocket driver do the rest. Asynchronous sources/destinations operate on a set of worker threads (in contrast to a dedicated thread) and are driven by the epoll() based event loop. -- Bazsi On Fri, Jan 22, 2016 at 12:13 PM, Vincent Bernat <bernat@luffy.cx> wrote:
❦ 22 janvier 2016 11:28 +0100, "Scheidler, Balázs" < balazs.scheidler@balabit.com> :
if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
Oh, there is a way to have an asynchronous implementation? Which module would be a simple but correct example of this? -- Never laugh at live dragons. -- Bilbo Baggins [J.R.R. Tolkien, "The Hobbit"]
❦ 22 janvier 2016 12:43 +0100, "Scheidler, Balázs" <balazs.scheidler@balabit.com> :
well, the afsocket one. that's a completely nonblocking destination that implements udp/tcp/unix-stream/unix-dgram/syslog/network sources and destinations.
It basically boils down to using a LogWriter class that does the heavy lifting at least as long as your transport protocol is simple enough. Probably the simplest is to create a LogProto implementation and let the afsocket driver do the rest.
Asynchronous sources/destinations operate on a set of worker threads (in contrast to a dedicated thread) and are driven by the epoll() based event loop.
It's for the librdkafka-based kafka destination. So, the third-party library is taking care of all the transport stuff. Is the afsocket one still a good example for async stuff in this case? I didn't have a look yet. -- Don't compare floating point numbers just for equality. - The Elements of Programming Style (Kernighan & Plauger)
Well, LogWriter assumes a specific semantics from the underlying transport. Async APIs usually give you access to the fd, and the needed I/O event mask. If that's the case, you can probably integrate it to syslog-ng's mainloop, which is ivykis based (an async event lib). But you can create a separate event loop if that's easier. E.g. use a single thread for all kafka destinations. Can you explain your use-case why the existing kafka destination is not good enough? Is it performance, is it java? ❦ 22 janvier 2016 12:43 +0100, "Scheidler, Balázs" < balazs.scheidler@balabit.com> :
well, the afsocket one. that's a completely nonblocking destination that implements udp/tcp/unix-stream/unix-dgram/syslog/network sources and destinations.
It basically boils down to using a LogWriter class that does the heavy lifting at least as long as your transport protocol is simple enough. Probably the simplest is to create a LogProto implementation and let the afsocket driver do the rest.
Asynchronous sources/destinations operate on a set of worker threads (in contrast to a dedicated thread) and are driven by the epoll() based event loop.
It's for the librdkafka-based kafka destination. So, the third-party library is taking care of all the transport stuff. Is the afsocket one still a good example for async stuff in this case? I didn't have a look yet. -- Don't compare floating point numbers just for equality. - The Elements of Programming Style (Kernighan & Plauger) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
❦ 26 janvier 2016 08:20 +0100, "Scheidler, Balázs" <balazs.scheidler@balabit.com> :
Well, LogWriter assumes a specific semantics from the underlying transport. Async APIs usually give you access to the fd, and the needed I/O event mask. If that's the case, you can probably integrate it to syslog-ng's mainloop, which is ivykis based (an async event lib).
librdkafka is able to give access to the underlying file descriptor, so it should be OK. I thought that LogWriter assumed that you'll have to format packets yourself, hence my question.
Can you explain your use-case why the existing kafka destination is not good enough? Is it performance, is it java?
It's Java and its memory footprint. -- Rebellion lay in his way, and he found it. -- William Shakespeare, "Henry IV"
participants (5)
-
m＠baha.mu
-
Marc Falzon
-
Scheidler, Balázs
-
Tibor Benke
-
Vincent Bernat