[syslog-ng] Module development guide
Scheidler, Balázs
balazs.scheidler at balabit.com
Sat Feb 6 21:42:17 CET 2016
I'll definitely review this, and I am happy that you reached your goal. I
think that an http dest is useful so much so that I once started doing it
but didn't have time to finish it.
I would even consider this to become preferred over the java implementation
as native code is better than jvm based one.
Bazsi
On Feb 6, 2016 9:17 PM, "Marc Falzon" <m at baha.mu> wrote:
> Hi
>
> Thanks to your tips I managed to reach my goal, here is the "final"
> version of the module:
>
>
> https://github.com/falzm/syslog-ng/commit/28a8649002f3d005dc2dfd2a8b53ef713a44eb40
>
> It can be used with the following configuration:
>
> destination d_curl {
> curl(
> url("http://localhost:8001/logs")
> headers(
> "X-Custom-Header1: plop"
> "X-Custom-Header2: blah"
> )
> body("$MESSAGE")
> );
> };
>
> In practice, logs being forwarded to a curl destination produce HTTP
> "POST" requests similar to:
>
> $ logger -p local0.error -t blah this is a very nice log message
>
> [H] Accept: */*
> [H] Content-Length: 68
> [H] Content-Type: application/x-www-form-urlencoded
> [H] Host: localhost:8001
> [H] User-Agent: syslog-ng 3.8.0alpha0/libcurl 7.46.0
> [H] X-Custom-Header1: plop
>
> [H] X-Custom-Header2: blah
>
> [H] X-Syslog-Facility: local0
> [H] X-Syslog-Host: cobalt
> [H] X-Syslog-Level: err
> [H] X-Syslog-Program: blah
> [B] this is a very nice log message
>
>
>
> I also implemented `user`/`password` driver arguments to allow HTTP basic
> authentication.
>
> I intend to start field-testing it soon ; in the meantime, could you tell
> me if you spot potential bugs/improvements? Again I'm not a pro developer,
> this has been done with a basic knowledge of C...
>
> Providing that it works as expected and that I document it extensively,
> would you consider merging this module in the mainline tree? I have a very
> practical usage for it, but I imagine it could be useful to other users.
> Alternatively, could it be possible to merge it into the syslog-ng
> incubator branch? I'd like to avoid patching and repackaging (Debian)
> syslog-ng every time there is a new release, if possible :)
>
> Thank you very much for your help along the way, looking forward to
> reading your feedback!
>
> Cheers,
>
> m.
>
>
> On Tuesday 2 February 2016 at 23:37, Scheidler, Balázs wrote:
>
> > Well, you just create a LogTemplate object using log_template_new(),
> compile an expression using log_template_compile() and then at runtime
> evaluate using log_template_format().
> > The name argument to _new() should be NULL.
> > The other arguments should be pretty straightforward, just grep for a
> call site for examples.
> > The assert in stats fails probably because the SCS identifier of the
> module is not added to the possible statistic sources. I know its not the
> best API there (all modules that want to report statistics has to be
> hardwired in the core).
> > On Feb 2, 2016 4:22 PM, "Marc Falzon" <m at baha.mu (mailto:m at baha.mu)>
> wrote:
> > > Hi
> > >
> > > Thank you, it's been very helpful: I now have a almost usable module :)
> > >
> > > I just have a last question for you: currently my module gets the log
> message using `log_msg_get_value(msg, LM_V_PROGRAM, NULL)` – which was
> enough for prototyping, however I guess it would be better to use the
> templating system that all other modules use... Do you mind explaining the
> basics of this system?
> > >
> > > One more thing: when I terminate syslog-ng running my module I get
> this message, do you have any idea what could cause this?
> > >
> > > **
> > > ERROR:lib/stats/stats-cluster.c:159:stats_cluster_untrack_counter:
> assertion failed: (self && (self->live_mask & (1 << type)) &&
> &self->counters[type] == (*counter))
> > > Aborted
> > >
> > >
> > >
> > > Cheers,
> > >
> > > m.
> > >
> > >
> > > On Tuesday 2 February 2016 at 09:22, Scheidler, Balázs wrote:
> > >
> > > > Hi,
> > > >
> > > > Let me try to describe what is needed to parse options from the
> configuration file:
> > > > curl-grammar.ym: is a yacc/bison source file that parses the curl
> specific portion of the configuration. It is able to generate parsers for
> LALR(1) languages.
> > > > curl-parser.c: contains the set of keywords that curl understands.
> This maps keywords found in the configuration into tokens that the grammar
> is able to process. A token is basically a numeric identifier.
> > > >
> > > > So for instance to add a "header" option, you will need this stuff:
> > > > 1) define a token in the grammar, so it becomes available as a macro
> > > > diff --git a/modules/curl/curl-grammar.ym
> b/modules/curl/curl-grammar.ym
> > > > index c3aaf00..eab317c 100644
> > > > --- a/modules/curl/curl-grammar.ym
> > > > +++ b/modules/curl/curl-grammar.ym
> > > > @@ -47,6 +47,7 @@
> > > > %parse-param {gpointer arg}
> > > >
> > > > %token KW_CURL
> > > > +%token KW_HEADER
> > > >
> > > > %type <ptr> driver
> > > > %type <ptr> curl_destination
> > > >
> > > > If you recompile the curl module, this will make KW_HEADER to become
> available in curl-grammar.h
> > > > 2) add a keyword to curl-parser.c
> > > > This will tell the syslog-ng lexer (a component that reads the
> configuration file and breaks them into tokens) to map the string "header"
> into a symbolic token named KW_HEADER.
> > > > diff --git a/modules/curl/curl-parser.c b/modules/curl/curl-parser.c
> > > > index 6958a8d..b60ef0b 100644
> > > > --- a/modules/curl/curl-parser.c
> > > > +++ b/modules/curl/curl-parser.c
> > > > @@ -30,6 +30,7 @@ int curl_parse(CfgLexer *lexer, LogDriver
> **instance, gpointer arg);
> > > >
> > > > static CfgLexerKeyword curl_keywords[] = {
> > > > { "curl", KW_CURL },
> > > > + { "header", KW_HEADER },
> > > > { NULL }
> > > > };
> > > >
> > > > 3) add rules to the grammar to parse this option
> > > > First we need a small refactor in the curl-grammar file to save the
> just instantiated CurlDestinationDriver (I hand edited this hunk for
> clarity, so it will not apply automatically):
> > > >
> > > > @@ -71,7 +72,15 @@ driver
> > > > ;
> > > >
> > > > curl_destination
> > > > - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); }
> > > > + : KW_CURL
> > > > + {
> > > > + last_driver = curl_dd_new(configuration);
> > > > + }
> > > > + '(' ')' { $$ = last_driver; }
> > > > + ;
> > > >
> > > >
> > > > This basically just stores the driver instance in a variable named
> "last_driver" (defined by the core cfg-grammar.y file). So we can later use
> this to call setters on.
> > > > And now let's add option parsing (this time the entire hunk):
> > > > @@ -71,7 +72,20 @@ driver
> > > > ;
> > > >
> > > > curl_destination
> > > > - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); }
> > > > + : KW_CURL
> > > > + {
> > > > + last_driver = curl_dd_new(configuration);
> > > > + }
> > > > + '(' curl_options ')' { $$ = last_driver; }
> > > > + ;
> > > > +
> > > > +curl_options
> > > > + : curl_option curl_options
> > > > + |
> > > > + ;
> > > > +
> > > > +curl_option
> > > > + : KW_HEADER '(' string string ')' {
> curl_dd_set_header(last_driver, $3, $4); }
> > > > ;
> > > >
> > > > /* INCLUDE_RULES */
> > > >
> > > > What this does is that we added a rule "curl_options" within the
> parentheses of our original "curl" rule, which is the rule that must match
> the configuration as we parse it. (think of it as a recursive tree).
> > > > curl_options is satisfied with either an empty rule (e.g. no
> options) or recursing itself with a curl_option rule. Note the plural and
> singular names, curl_options is responsible to match a list of
> "curl_option"s.
> > > > curl_option is responsible for parsing individual options, for now
> only a single rule is there, KW_HEADER, which when parsed will call
> curl_dd_set_header().
> > > > You can add further options like this:
> > > > +curl_option
> > > > + : KW_HEADER '(' string string ')' {
> curl_dd_set_header(last_driver, $3, $4); }
> > > > + | KW_URL '(' string ')' { curl_dd_set_url(last_driver, $3); }
> > > >
> > > >
> > > > Variables like $1, $2 and so on represent the Nth token in the rule,
> e.g. $3 in the KW_HEADER branch is the first string value. $4, is the 2nd.
> > > > Hope this helps. You will definitely be able to find tutorials that
> describe both yacc/lex, however there are a number of quirks how syslog-ng
> applies them, primarily to make it possible to extend the base grammar with
> plugins.
> > > > Cheers,
> > > > Bazsi
> > > >
> > > >
> > > > --
> > > > Bazsi
> > > >
> > > > On Mon, Feb 1, 2016 at 10:01 PM, Marc Falzon <m at baha.mu (mailto:
> m at baha.mu) (mailto:m at baha.mu)> wrote:
> > > > > Hi Bazsi
> > > > >
> > > > > Sorry to bother you again, but now that I'm done prototyping the
> "curl" part of my module using hardcoded values for testing I'm now facing
> the "grammar/parser" problem: could you please drive me though the basics
> of how this works? I tried to understand looking at other modules but I
> don't understand at all.
> > > > >
> > > > > Basically, I'd like to be able to specify some configuration
> settings in the syslog-ng configuration, such as url, some HTTP headers...
> It would look like this:
> > > > >
> > > > > destination d_curl {
> > > > > curl(url("http://logs.example.net:5140/" header("Content-Type:
> application/json") header("X-Custom-Header: blah")));
> > > > > };
> > > > >
> > > > >
> > > > > My code is still available at
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl
> > > > >
> > > > > Thank you in advance,
> > > > >
> > > > > m.
> > > > >
> > > > >
> > > > > On Saturday 30 January 2016 at 10:17, m at baha.mu (mailto:m at baha.mu)
> (mailto:m at baha.mu) wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > Great! I should be able to get going from here, thank you for
> your help.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > m.
> > > > > >
> > > > > > > On Jan 30, 2016, at 08:56, Scheidler, Balázs <
> balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com)
> (mailto:balazs.scheidler at balabit.com) (mailto:balazs.scheidler at balabit.com)>
> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have added code to your module, it compiles and is able to
> accept messages to be sent, and prints a debug message whenever that
> happens.
> > > > > > >
> > > > > > > I didn't have too much time to clean it up, but I hope this
> helps to start the ball rolling.
> > > > > > >
> > > > > > > https://github.com/balabit/syslog-ng/compare/f/curl-module <
> https://github.com/balabit/syslog-ng/compare/f/curl-module>
> > > > > > >
> > > > > > >
> > > > > > > Bazsi
> > > > > > >
> > > > > > > --
> > > > > > > Bazsi
> > > > > > >
> > > > > > > On Sat, Jan 23, 2016 at 9:27 PM, <m at baha.mu (mailto:m at baha.mu)
> (mailto:m at baha.mu) <mailto:m at baha.mu>> wrote:
> > > > > > >
> > > > > > > > On Jan 23, 2016, at 21:23, Scheidler, Balázs <
> balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com)
> (mailto:balazs.scheidler at balabit.com) <mailto:balazs.scheidler at balabit.com>>
> wrote:
> > > > > > > >
> > > > > > > > If you commit your code to a branch I can help with the
> basic skeleton of the parser/grammar
> > > > > > >
> > > > > > >
> > > > > > > Thank you Balázs, my code is available here:
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl <
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl>
> > > > > > >
> > > > > > > m.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > On Jan 23, 2016 5:54 PM, "Marc Falzon" <m at baha.mu (mailto:
> m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu>> wrote:
> > > > > > > > I successfully generated a module template using the
> `create_plugin.sh (http://create_plugin.sh) (http://create_plugin.sh) (
> http://create_plugin.sh)` script, however I don't have the slightest idea
> on how to get started from here (I mean for the 'plugin glue', I can manage
> the cURL part)... The most problematic step for me is the whole
> parser/grammar thing, to which I believe to involve YACC/Bison parsing but
> I don't know how it works and it's too big of a learning curve for what I
> want to achieve at this point... Since the Gitbook doesn't provide any
> useful information on module development at the moment, is there any way to
> get me started on this topic? I'm not a developer (sysadmin), and trying to
> learn from existing module doesn't quite cut it.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > > > > > > m.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
> > > > > > > >
> > > > > > > > > There is also a shell script which generates a basic
> destination skeleton. You can find its source on this brach:
> https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator <
> https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator> . Check
> the dev-utils/plugin_skeleton_creator directory and this commit for the
> usage:
> https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010
> <
> https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010
> >
> > > > > > > > >
> > > > > > > > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m at baha.mu (mailto:
> m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu> (mailto:m at baha.mu
> <mailto:m at baha.mu>)>:
> > > > > > > > > > Hi Bazsi,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs
> wrote:
> > > > > > > > > >
> > > > > > > > > > > not really, but maybe I would read the "official"
> modules source code instead. and of course you can ask questions on the
> mailing list. :)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes that's what I've started to do, but a proper
> documentation is preferable, saves time and avoid confusion ;)
> > > > > > > > > >
> > > > > > > > > > > if you want to create a destination, the easiest is to
> create a threaded destination, where the output is an independent thread,
> that can use a synchronous API. Of course this is going to be slower than
> using an asynchronous implementation, but that usually is good enough,
> > > > > > > > > >
> > > > > > > > > > Thank you for the hint. Given my use case (HTTP
> destination) I think I'm safe to go with an asynchronous implementation
> since each request is stateless, no connections/locks involved.
> > > > > > > > > >
> > > > > > > > > > m.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <
> m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu>
> (mailto:m at baha.mu <mailto:m at baha.mu>) (mailto:m at baha.mu <mailto:m at baha.mu>)>
> wrote:
> > > > > > > > > > > > Hi
> > > > > > > > > > > >
> > > > > > > > > > > > Is there any documentation on how to develop new
> syslog-ng module using the native C API, besides reading the incubator
> modules source code?
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > m.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> ______________________________________________________________________________
> > > > > > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> ______________________________________________________________________________
> > > > > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> ______________________________________________________________________________
> > > > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> ______________________________________________________________________________
> > > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> ______________________________________________________________________________
> > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > >
> > > > > > > >
> ______________________________________________________________________________
> > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> ______________________________________________________________________________
> > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160206/453ed11d/attachment-0001.htm
More information about the syslog-ng
mailing list