[syslog-ng] Module development guide

Scheidler, Balázs balazs.scheidler at balabit.com
Tue Feb 2 23:37:11 CET 2016 

Well, you just create a LogTemplate object using log_template_new(),
compile an expression using log_template_compile() and then at runtime
evaluate using log_template_format().

The name argument to _new() should be NULL.

The other arguments should be pretty straightforward, just grep for a call
site for examples.

The assert in stats fails probably because the SCS identifier of the module
is not added to the possible statistic sources. I know its not the best API
there (all modules that want to report statistics has to be hardwired in
the core).
On Feb 2, 2016 4:22 PM, "Marc Falzon" <m at baha.mu> wrote:

> Hi
>
> Thank you, it's been very helpful: I now have a almost usable module :)
>
> I just have a last question for you: currently my module gets the log
> message using `log_msg_get_value(msg, LM_V_PROGRAM, NULL)` – which was
> enough for prototyping, however I guess it would be better to use the
> templating system that all other modules use... Do you mind explaining the
> basics of this system?
>
> One more thing: when I terminate syslog-ng running my module I get this
> message, do you have any idea what could cause this?
>
> **
> ERROR:lib/stats/stats-cluster.c:159:stats_cluster_untrack_counter:
> assertion failed: (self && (self->live_mask & (1 << type)) &&
> &self->counters[type] == (*counter))
> Aborted
>
>
>
> Cheers,
>
> m.
>
>
> On Tuesday 2 February 2016 at 09:22, Scheidler, Balázs wrote:
>
> > Hi,
> >
> > Let me try to describe what is needed to parse options from the
> configuration file:
> > curl-grammar.ym: is a yacc/bison source file that parses the curl
> specific portion of the configuration. It is able to generate parsers for
> LALR(1) languages.
> > curl-parser.c: contains the set of keywords that curl understands. This
> maps keywords found in the configuration into tokens that the grammar is
> able to process. A token is basically a numeric identifier.
> >
> > So for instance to add a "header" option, you will need this stuff:
> > 1) define a token in the grammar, so it becomes available as a macro
> > diff --git a/modules/curl/curl-grammar.ym b/modules/curl/curl-grammar.ym
> > index c3aaf00..eab317c 100644
> > --- a/modules/curl/curl-grammar.ym
> > +++ b/modules/curl/curl-grammar.ym
> > @@ -47,6 +47,7 @@
> > %parse-param {gpointer arg}
> >
> > %token KW_CURL
> > +%token KW_HEADER
> >
> > %type <ptr> driver
> > %type <ptr> curl_destination
> >
> > If you recompile the curl module, this will make KW_HEADER to become
> available in curl-grammar.h
> > 2) add a keyword to curl-parser.c
> > This will tell the syslog-ng lexer (a component that reads the
> configuration file and breaks them into tokens) to map the string "header"
> into a symbolic token named KW_HEADER.
> > diff --git a/modules/curl/curl-parser.c b/modules/curl/curl-parser.c
> > index 6958a8d..b60ef0b 100644
> > --- a/modules/curl/curl-parser.c
> > +++ b/modules/curl/curl-parser.c
> > @@ -30,6 +30,7 @@ int curl_parse(CfgLexer *lexer, LogDriver **instance,
> gpointer arg);
> >
> > static CfgLexerKeyword curl_keywords[] = {
> > { "curl", KW_CURL },
> > + { "header", KW_HEADER },
> > { NULL }
> > };
> >
> > 3) add rules to the grammar to parse this option
> > First we need a small refactor in the curl-grammar file to save the just
> instantiated CurlDestinationDriver (I hand edited this hunk for clarity, so
> it will not apply automatically):
> >
> > @@ -71,7 +72,15 @@ driver
> > ;
> >
> > curl_destination
> > - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); }
> > + : KW_CURL
> > + {
> > + last_driver = curl_dd_new(configuration);
> > + }
> > + '(' ')' { $$ = last_driver; }
> > + ;
> >
> >
> > This basically just stores the driver instance in a variable named
> "last_driver" (defined by the core cfg-grammar.y file). So we can later use
> this to call setters on.
> > And now let's add option parsing (this time the entire hunk):
> > @@ -71,7 +72,20 @@ driver
> > ;
> >
> > curl_destination
> > - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); }
> > + : KW_CURL
> > + {
> > + last_driver = curl_dd_new(configuration);
> > + }
> > + '(' curl_options ')' { $$ = last_driver; }
> > + ;
> > +
> > +curl_options
> > + : curl_option curl_options
> > + |
> > + ;
> > +
> > +curl_option
> > + : KW_HEADER '(' string string ')' { curl_dd_set_header(last_driver,
> $3, $4); }
> > ;
> >
> > /* INCLUDE_RULES */
> >
> > What this does is that we added a rule "curl_options" within the
> parentheses of our original "curl" rule, which is the rule that must match
> the configuration as we parse it. (think of it as a recursive tree).
> > curl_options is satisfied with either an empty rule (e.g. no options) or
> recursing itself with a curl_option rule. Note the plural and singular
> names, curl_options is responsible to match a list of "curl_option"s.
> > curl_option is responsible for parsing individual options, for now only
> a single rule is there, KW_HEADER, which when parsed will call
> curl_dd_set_header().
> > You can add further options like this:
> > +curl_option
> > + : KW_HEADER '(' string string ')' { curl_dd_set_header(last_driver,
> $3, $4); }
> > + | KW_URL '(' string ')' { curl_dd_set_url(last_driver, $3); }
> >
> >
> > Variables like $1, $2 and so on represent the Nth token in the rule,
> e.g. $3 in the KW_HEADER branch is the first string value. $4, is the 2nd.
> > Hope this helps. You will definitely be able to find tutorials that
> describe both yacc/lex, however there are a number of quirks how syslog-ng
> applies them, primarily to make it possible to extend the base grammar with
> plugins.
> > Cheers,
> > Bazsi
> >
> >
> > --
> > Bazsi
> >
> > On Mon, Feb 1, 2016 at 10:01 PM, Marc Falzon <m at baha.mu (mailto:
> m at baha.mu)> wrote:
> > > Hi Bazsi
> > >
> > > Sorry to bother you again, but now that I'm done prototyping the
> "curl" part of my module using hardcoded values for testing I'm now facing
> the "grammar/parser" problem: could you please drive me though the basics
> of how this works? I tried to understand looking at other modules but I
> don't understand at all.
> > >
> > > Basically, I'd like to be able to specify some configuration settings
> in the syslog-ng configuration, such as url, some HTTP headers... It would
> look like this:
> > >
> > > destination d_curl {
> > > curl(url("http://logs.example.net:5140/" header("Content-Type:
> application/json") header("X-Custom-Header: blah")));
> > > };
> > >
> > >
> > > My code is still available at
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl
> > >
> > > Thank you in advance,
> > >
> > > m.
> > >
> > >
> > > On Saturday 30 January 2016 at 10:17, m at baha.mu (mailto:m at baha.mu)
> wrote:
> > >
> > > > Hi
> > > >
> > > > Great! I should be able to get going from here, thank you for your
> help.
> > > >
> > > > Cheers,
> > > >
> > > > m.
> > > >
> > > > > On Jan 30, 2016, at 08:56, Scheidler, Balázs <
> balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com)
> (mailto:balazs.scheidler at balabit.com)> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I have added code to your module, it compiles and is able to
> accept messages to be sent, and prints a debug message whenever that
> happens.
> > > > >
> > > > > I didn't have too much time to clean it up, but I hope this helps
> to start the ball rolling.
> > > > >
> > > > > https://github.com/balabit/syslog-ng/compare/f/curl-module <
> https://github.com/balabit/syslog-ng/compare/f/curl-module>
> > > > >
> > > > >
> > > > > Bazsi
> > > > >
> > > > > --
> > > > > Bazsi
> > > > >
> > > > > On Sat, Jan 23, 2016 at 9:27 PM, <m at baha.mu (mailto:m at baha.mu)
> <mailto:m at baha.mu>> wrote:
> > > > >
> > > > > > On Jan 23, 2016, at 21:23, Scheidler, Balázs <
> balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com)
> <mailto:balazs.scheidler at balabit.com>> wrote:
> > > > > >
> > > > > > If you commit your code to a branch I can help with the basic
> skeleton of the parser/grammar
> > > > >
> > > > >
> > > > > Thank you Balázs, my code is available here:
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl <
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl>
> > > > >
> > > > > m.
> > > > >
> > > > >
> > > > > >
> > > > > > On Jan 23, 2016 5:54 PM, "Marc Falzon" <m at baha.mu (mailto:
> m at baha.mu) <mailto:m at baha.mu>> wrote:
> > > > > > I successfully generated a module template using the
> `create_plugin.sh (http://create_plugin.sh) (http://create_plugin.sh)`
> script, however I don't have the slightest idea on how to get started from
> here (I mean for the 'plugin glue', I can manage the cURL part)... The most
> problematic step for me is the whole parser/grammar thing, to which I
> believe to involve YACC/Bison parsing but I don't know how it works and
> it's too big of a learning curve for what I want to achieve at this
> point... Since the Gitbook doesn't provide any useful information on module
> development at the moment, is there any way to get me started on this
> topic? I'm not a developer (sysadmin), and trying to learn from existing
> module doesn't quite cut it.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > m.
> > > > > >
> > > > > >
> > > > > > On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
> > > > > >
> > > > > > > There is also a shell script which generates a basic
> destination skeleton. You can find its source on this brach:
> https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator <
> https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator> . Check
> the dev-utils/plugin_skeleton_creator directory and this commit for the
> usage:
> https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010
> <
> https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010
> >
> > > > > > >
> > > > > > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m at baha.mu (mailto:
> m at baha.mu) <mailto:m at baha.mu> (mailto:m at baha.mu <mailto:m at baha.mu>)>:
> > > > > > > > Hi Bazsi,
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
> > > > > > > >
> > > > > > > > > not really, but maybe I would read the "official" modules
> source code instead. and of course you can ask questions on the mailing
> list. :)
> > > > > > > >
> > > > > > > >
> > > > > > > > Yes that's what I've started to do, but a proper
> documentation is preferable, saves time and avoid confusion ;)
> > > > > > > >
> > > > > > > > > if you want to create a destination, the easiest is to
> create a threaded destination, where the output is an independent thread,
> that can use a synchronous API. Of course this is going to be slower than
> using an asynchronous implementation, but that usually is good enough,
> > > > > > > >
> > > > > > > > Thank you for the hint. Given my use case (HTTP destination)
> I think I'm safe to go with an asynchronous implementation since each
> request is stateless, no connections/locks involved.
> > > > > > > >
> > > > > > > > m.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m at baha.mu
> (mailto:m at baha.mu) <mailto:m at baha.mu> (mailto:m at baha.mu <mailto:m at baha.mu>)
> (mailto:m at baha.mu <mailto:m at baha.mu>)> wrote:
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > Is there any documentation on how to develop new
> syslog-ng module using the native C API, besides reading the incubator
> modules source code?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > m.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> ______________________________________________________________________________
> > > > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> ______________________________________________________________________________
> > > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> ______________________________________________________________________________
> > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> ______________________________________________________________________________
> > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> ______________________________________________________________________________
> > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > >
> > > > > >
> ______________________________________________________________________________
> > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> ______________________________________________________________________________
> > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > >
> > >
> >
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160202/c111a4c4/attachment-0001.htm

More information about the syslog-ng mailing list