[syslog-ng] Module development guide

Scheidler, Balázs balazs.scheidler at balabit.com
Tue Feb 2 09:22:42 CET 2016 

Hi,

Let me try to describe what is needed to parse options from the
configuration file:

   - curl-grammar.ym: is a yacc/bison source file that parses the curl
   specific portion of the configuration. It is able to generate parsers for
   LALR(1) languages.
   - curl-parser.c: contains the set of keywords that curl understands.
   This maps keywords found in the configuration into tokens that the grammar
   is able to process. A token is basically a numeric identifier.

So for instance to add a "header" option, you will need this stuff:

1) define a token in the grammar, so it becomes available as a macro

diff --git a/modules/curl/curl-grammar.ym b/modules/curl/curl-grammar.ym
index c3aaf00..eab317c 100644
--- a/modules/curl/curl-grammar.ym
+++ b/modules/curl/curl-grammar.ym
@@ -47,6 +47,7 @@
 %parse-param {gpointer arg}

 %token KW_CURL
+%token KW_HEADER

 %type   <ptr> driver
 %type   <ptr> curl_destination


If you recompile the curl module, this will make KW_HEADER to become
available in curl-grammar.h

2) add a keyword to curl-parser.c

This will tell the syslog-ng lexer (a component that reads the
configuration file and breaks them into tokens) to map the string "header"
into a symbolic token named KW_HEADER.

diff --git a/modules/curl/curl-parser.c b/modules/curl/curl-parser.c
index 6958a8d..b60ef0b 100644
--- a/modules/curl/curl-parser.c
+++ b/modules/curl/curl-parser.c
@@ -30,6 +30,7 @@ int curl_parse(CfgLexer *lexer, LogDriver **instance,
gpointer arg);

 static CfgLexerKeyword curl_keywords[] = {
   { "curl", KW_CURL },
+  { "header", KW_HEADER },
   { NULL }
 };


3) add rules to the grammar to parse this option

First we need a small refactor in the curl-grammar file to save the just
instantiated CurlDestinationDriver (I hand edited this hunk for clarity, so
it will not apply automatically):


@@ -71,7 +72,15 @@ driver
        ;

 curl_destination
-       : KW_CURL '(' ')'                       { $$ =
curl_dd_new(configuration); }
+       : KW_CURL
+         {
+            last_driver = curl_dd_new(configuration);
+          }
+         '('  ')'                             { $$ = last_driver; }
+       ;


This basically just stores the driver instance in a variable named
"last_driver" (defined by the core cfg-grammar.y file). So we can later use
this to call setters on.

And now let's add option parsing (this time the entire hunk):

@@ -71,7 +72,20 @@ driver
        ;

 curl_destination
-       : KW_CURL '(' ')'                       { $$ =
curl_dd_new(configuration); }
+       : KW_CURL
+         {
+            last_driver = curl_dd_new(configuration);
+          }
+         '(' curl_options ')'                             { $$ =
last_driver; }
+       ;
+
+curl_options
+       : curl_option curl_options
+       |
+       ;
+
+curl_option
+       : KW_HEADER '(' string string ')'                  {
curl_dd_set_header(last_driver, $3, $4); }
        ;

 /* INCLUDE_RULES */

What this does is that we added a rule "curl_options" within the
parentheses of our original "curl" rule, which is the rule that must match
the configuration as we parse it. (think of it as a recursive tree).

curl_options is satisfied with either an empty rule (e.g. no options) or
recursing itself with a curl_option rule. Note the plural and singular
names, curl_options is responsible to match a list of "curl_option"s.

curl_option is responsible for parsing individual options, for now only a
single rule is there, KW_HEADER, which when parsed will call
curl_dd_set_header().

You can add further options like this:
+curl_option
+       : KW_HEADER '(' string string ')'                  {
curl_dd_set_header(last_driver, $3, $4); }
+       | KW_URL '(' string ')'                            {
curl_dd_set_url(last_driver, $3); }


Variables like $1, $2 and so on represent the Nth token in the rule, e.g.
$3 in the KW_HEADER branch is the first string value. $4, is the 2nd.

Hope this helps. You will definitely be able to find tutorials that
describe both yacc/lex, however there are a number of quirks how syslog-ng
applies them, primarily to make it possible to extend the base grammar with
plugins.

Cheers,

Bazsi



-- 
Bazsi

On Mon, Feb 1, 2016 at 10:01 PM, Marc Falzon <m at baha.mu> wrote:

> Hi Bazsi
>
> Sorry to bother you again, but now that I'm done prototyping the "curl"
> part of my module using hardcoded values for testing I'm now facing the
> "grammar/parser" problem: could you please drive me though the basics of
> how this works? I tried to understand looking at other modules but I don't
> understand at all.
>
> Basically, I'd like to be able to specify some configuration settings in
> the syslog-ng configuration, such as url, some HTTP headers... It would
> look like this:
>
> destination d_curl {
>   curl(url("http://logs.example.net:5140/" header("Content-Type:
> application/json") header("X-Custom-Header: blah")));
> };
>
>
> My code is still available at
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl
>
> Thank you in advance,
>
> m.
>
>
> On Saturday 30 January 2016 at 10:17, m at baha.mu wrote:
>
> > Hi
> >
> > Great! I should be able to get going from here, thank you for your help.
> >
> > Cheers,
> >
> > m.
> >
> > > On Jan 30, 2016, at 08:56, Scheidler, Balázs <
> balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com)> wrote:
> > >
> > > Hi,
> > >
> > > I have added code to your module, it compiles and is able to accept
> messages to be sent, and prints a debug message whenever that happens.
> > >
> > > I didn't have too much time to clean it up, but I hope this helps to
> start the ball rolling.
> > >
> > > https://github.com/balabit/syslog-ng/compare/f/curl-module <
> https://github.com/balabit/syslog-ng/compare/f/curl-module>
> > >
> > >
> > > Bazsi
> > >
> > > --
> > > Bazsi
> > >
> > > On Sat, Jan 23, 2016 at 9:27 PM, <m at baha.mu <mailto:m at baha.mu>> wrote:
> > >
> > > > On Jan 23, 2016, at 21:23, Scheidler, Balázs <
> balazs.scheidler at balabit.com <mailto:balazs.scheidler at balabit.com>> wrote:
> > > >
> > > > If you commit your code to a branch I can help with the basic
> skeleton of the parser/grammar
> > >
> > >
> > > Thank you Balázs, my code is available here:
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl <
> https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl>
> > >
> > > m.
> > >
> > >
> > > >
> > > > On Jan 23, 2016 5:54 PM, "Marc Falzon" <m at baha.mu <mailto:m at baha.mu>>
> wrote:
> > > > I successfully generated a module template using the
> `create_plugin.sh (http://create_plugin.sh)` script, however I don't have
> the slightest idea on how to get started from here (I mean for the 'plugin
> glue', I can manage the cURL part)... The most problematic step for me is
> the whole parser/grammar thing, to which I believe to involve YACC/Bison
> parsing but I don't know how it works and it's too big of a learning curve
> for what I want to achieve at this point... Since the Gitbook doesn't
> provide any useful information on module development at the moment, is
> there any way to get me started on this topic? I'm not a developer
> (sysadmin), and trying to learn from existing module doesn't quite cut it.
> > > >
> > > > Thank you,
> > > >
> > > > m.
> > > >
> > > >
> > > > On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
> > > >
> > > > > There is also a shell script which generates a basic destination
> skeleton. You can find its source on this brach:
> https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator <
> https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator> . Check
> the dev-utils/plugin_skeleton_creator directory and this commit for the
> usage:
> https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010
> <
> https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010
> >
> > > > >
> > > > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m at baha.mu <mailto:
> m at baha.mu> (mailto:m at baha.mu <mailto:m at baha.mu>)>:
> > > > > > Hi Bazsi,
> > > > > >
> > > > > >
> > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
> > > > > >
> > > > > > > not really, but maybe I would read the "official" modules
> source code instead. and of course you can ask questions on the mailing
> list. :)
> > > > > >
> > > > > >
> > > > > > Yes that's what I've started to do, but a proper documentation
> is preferable, saves time and avoid confusion ;)
> > > > > >
> > > > > > > if you want to create a destination, the easiest is to create
> a threaded destination, where the output is an independent thread, that can
> use a synchronous API. Of course this is going to be slower than using an
> asynchronous implementation, but that usually is good enough,
> > > > > >
> > > > > > Thank you for the hint. Given my use case (HTTP destination) I
> think I'm safe to go with an asynchronous implementation since each request
> is stateless, no connections/locks involved.
> > > > > >
> > > > > > m.
> > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m at baha.mu
> <mailto:m at baha.mu> (mailto:m at baha.mu <mailto:m at baha.mu>) (mailto:m at baha.mu
> <mailto:m at baha.mu>)> wrote:
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > Is there any documentation on how to develop new syslog-ng
> module using the native C API, besides reading the incubator modules source
> code?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > m.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> ______________________________________________________________________________
> > > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> ______________________________________________________________________________
> > > > > > > Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> ______________________________________________________________________________
> > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> ______________________________________________________________________________
> > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> ______________________________________________________________________________
> > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > > >
> > > >
> ______________________________________________________________________________
> > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> > >
> > >
> > >
> > >
> ______________________________________________________________________________
> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng <
> http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> >
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160202/dd4391ce/attachment-0001.htm

More information about the syslog-ng mailing list