[syslog-ng] Module development guide
Marc Falzon
m at baha.mu
Sat Feb 6 21:17:32 CET 2016
Hi
Thanks to your tips I managed to reach my goal, here is the "final" version of the module:
https://github.com/falzm/syslog-ng/commit/28a8649002f3d005dc2dfd2a8b53ef713a44eb40
It can be used with the following configuration:
destination d_curl {
curl(
url("http://localhost:8001/logs")
headers(
"X-Custom-Header1: plop"
"X-Custom-Header2: blah"
)
body("$MESSAGE")
);
};
In practice, logs being forwarded to a curl destination produce HTTP "POST" requests similar to:
$ logger -p local0.error -t blah this is a very nice log message
[H] Accept: */*
[H] Content-Length: 68
[H] Content-Type: application/x-www-form-urlencoded
[H] Host: localhost:8001
[H] User-Agent: syslog-ng 3.8.0alpha0/libcurl 7.46.0
[H] X-Custom-Header1: plop
[H] X-Custom-Header2: blah
[H] X-Syslog-Facility: local0
[H] X-Syslog-Host: cobalt
[H] X-Syslog-Level: err
[H] X-Syslog-Program: blah
[B] this is a very nice log message
I also implemented `user`/`password` driver arguments to allow HTTP basic authentication.
I intend to start field-testing it soon ; in the meantime, could you tell me if you spot potential bugs/improvements? Again I'm not a pro developer, this has been done with a basic knowledge of C...
Providing that it works as expected and that I document it extensively, would you consider merging this module in the mainline tree? I have a very practical usage for it, but I imagine it could be useful to other users. Alternatively, could it be possible to merge it into the syslog-ng incubator branch? I'd like to avoid patching and repackaging (Debian) syslog-ng every time there is a new release, if possible :)
Thank you very much for your help along the way, looking forward to reading your feedback!
Cheers,
m.
On Tuesday 2 February 2016 at 23:37, Scheidler, Balázs wrote:
> Well, you just create a LogTemplate object using log_template_new(), compile an expression using log_template_compile() and then at runtime evaluate using log_template_format().
> The name argument to _new() should be NULL.
> The other arguments should be pretty straightforward, just grep for a call site for examples.
> The assert in stats fails probably because the SCS identifier of the module is not added to the possible statistic sources. I know its not the best API there (all modules that want to report statistics has to be hardwired in the core).
> On Feb 2, 2016 4:22 PM, "Marc Falzon" <m at baha.mu (mailto:m at baha.mu)> wrote:
> > Hi
> >
> > Thank you, it's been very helpful: I now have a almost usable module :)
> >
> > I just have a last question for you: currently my module gets the log message using `log_msg_get_value(msg, LM_V_PROGRAM, NULL)` – which was enough for prototyping, however I guess it would be better to use the templating system that all other modules use... Do you mind explaining the basics of this system?
> >
> > One more thing: when I terminate syslog-ng running my module I get this message, do you have any idea what could cause this?
> >
> > **
> > ERROR:lib/stats/stats-cluster.c:159:stats_cluster_untrack_counter: assertion failed: (self && (self->live_mask & (1 << type)) && &self->counters[type] == (*counter))
> > Aborted
> >
> >
> >
> > Cheers,
> >
> > m.
> >
> >
> > On Tuesday 2 February 2016 at 09:22, Scheidler, Balázs wrote:
> >
> > > Hi,
> > >
> > > Let me try to describe what is needed to parse options from the configuration file:
> > > curl-grammar.ym: is a yacc/bison source file that parses the curl specific portion of the configuration. It is able to generate parsers for LALR(1) languages.
> > > curl-parser.c: contains the set of keywords that curl understands. This maps keywords found in the configuration into tokens that the grammar is able to process. A token is basically a numeric identifier.
> > >
> > > So for instance to add a "header" option, you will need this stuff:
> > > 1) define a token in the grammar, so it becomes available as a macro
> > > diff --git a/modules/curl/curl-grammar.ym b/modules/curl/curl-grammar.ym
> > > index c3aaf00..eab317c 100644
> > > --- a/modules/curl/curl-grammar.ym
> > > +++ b/modules/curl/curl-grammar.ym
> > > @@ -47,6 +47,7 @@
> > > %parse-param {gpointer arg}
> > >
> > > %token KW_CURL
> > > +%token KW_HEADER
> > >
> > > %type <ptr> driver
> > > %type <ptr> curl_destination
> > >
> > > If you recompile the curl module, this will make KW_HEADER to become available in curl-grammar.h
> > > 2) add a keyword to curl-parser.c
> > > This will tell the syslog-ng lexer (a component that reads the configuration file and breaks them into tokens) to map the string "header" into a symbolic token named KW_HEADER.
> > > diff --git a/modules/curl/curl-parser.c b/modules/curl/curl-parser.c
> > > index 6958a8d..b60ef0b 100644
> > > --- a/modules/curl/curl-parser.c
> > > +++ b/modules/curl/curl-parser.c
> > > @@ -30,6 +30,7 @@ int curl_parse(CfgLexer *lexer, LogDriver **instance, gpointer arg);
> > >
> > > static CfgLexerKeyword curl_keywords[] = {
> > > { "curl", KW_CURL },
> > > + { "header", KW_HEADER },
> > > { NULL }
> > > };
> > >
> > > 3) add rules to the grammar to parse this option
> > > First we need a small refactor in the curl-grammar file to save the just instantiated CurlDestinationDriver (I hand edited this hunk for clarity, so it will not apply automatically):
> > >
> > > @@ -71,7 +72,15 @@ driver
> > > ;
> > >
> > > curl_destination
> > > - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); }
> > > + : KW_CURL
> > > + {
> > > + last_driver = curl_dd_new(configuration);
> > > + }
> > > + '(' ')' { $$ = last_driver; }
> > > + ;
> > >
> > >
> > > This basically just stores the driver instance in a variable named "last_driver" (defined by the core cfg-grammar.y file). So we can later use this to call setters on.
> > > And now let's add option parsing (this time the entire hunk):
> > > @@ -71,7 +72,20 @@ driver
> > > ;
> > >
> > > curl_destination
> > > - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); }
> > > + : KW_CURL
> > > + {
> > > + last_driver = curl_dd_new(configuration);
> > > + }
> > > + '(' curl_options ')' { $$ = last_driver; }
> > > + ;
> > > +
> > > +curl_options
> > > + : curl_option curl_options
> > > + |
> > > + ;
> > > +
> > > +curl_option
> > > + : KW_HEADER '(' string string ')' { curl_dd_set_header(last_driver, $3, $4); }
> > > ;
> > >
> > > /* INCLUDE_RULES */
> > >
> > > What this does is that we added a rule "curl_options" within the parentheses of our original "curl" rule, which is the rule that must match the configuration as we parse it. (think of it as a recursive tree).
> > > curl_options is satisfied with either an empty rule (e.g. no options) or recursing itself with a curl_option rule. Note the plural and singular names, curl_options is responsible to match a list of "curl_option"s.
> > > curl_option is responsible for parsing individual options, for now only a single rule is there, KW_HEADER, which when parsed will call curl_dd_set_header().
> > > You can add further options like this:
> > > +curl_option
> > > + : KW_HEADER '(' string string ')' { curl_dd_set_header(last_driver, $3, $4); }
> > > + | KW_URL '(' string ')' { curl_dd_set_url(last_driver, $3); }
> > >
> > >
> > > Variables like $1, $2 and so on represent the Nth token in the rule, e.g. $3 in the KW_HEADER branch is the first string value. $4, is the 2nd.
> > > Hope this helps. You will definitely be able to find tutorials that describe both yacc/lex, however there are a number of quirks how syslog-ng applies them, primarily to make it possible to extend the base grammar with plugins.
> > > Cheers,
> > > Bazsi
> > >
> > >
> > > --
> > > Bazsi
> > >
> > > On Mon, Feb 1, 2016 at 10:01 PM, Marc Falzon <m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu)> wrote:
> > > > Hi Bazsi
> > > >
> > > > Sorry to bother you again, but now that I'm done prototyping the "curl" part of my module using hardcoded values for testing I'm now facing the "grammar/parser" problem: could you please drive me though the basics of how this works? I tried to understand looking at other modules but I don't understand at all.
> > > >
> > > > Basically, I'd like to be able to specify some configuration settings in the syslog-ng configuration, such as url, some HTTP headers... It would look like this:
> > > >
> > > > destination d_curl {
> > > > curl(url("http://logs.example.net:5140/" header("Content-Type: application/json") header("X-Custom-Header: blah")));
> > > > };
> > > >
> > > >
> > > > My code is still available at https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl
> > > >
> > > > Thank you in advance,
> > > >
> > > > m.
> > > >
> > > >
> > > > On Saturday 30 January 2016 at 10:17, m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu) wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Great! I should be able to get going from here, thank you for your help.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > m.
> > > > >
> > > > > > On Jan 30, 2016, at 08:56, Scheidler, Balázs <balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com) (mailto:balazs.scheidler at balabit.com) (mailto:balazs.scheidler at balabit.com)> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have added code to your module, it compiles and is able to accept messages to be sent, and prints a debug message whenever that happens.
> > > > > >
> > > > > > I didn't have too much time to clean it up, but I hope this helps to start the ball rolling.
> > > > > >
> > > > > > https://github.com/balabit/syslog-ng/compare/f/curl-module <https://github.com/balabit/syslog-ng/compare/f/curl-module>
> > > > > >
> > > > > >
> > > > > > Bazsi
> > > > > >
> > > > > > --
> > > > > > Bazsi
> > > > > >
> > > > > > On Sat, Jan 23, 2016 at 9:27 PM, <m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu>> wrote:
> > > > > >
> > > > > > > On Jan 23, 2016, at 21:23, Scheidler, Balázs <balazs.scheidler at balabit.com (mailto:balazs.scheidler at balabit.com) (mailto:balazs.scheidler at balabit.com) <mailto:balazs.scheidler at balabit.com>> wrote:
> > > > > > >
> > > > > > > If you commit your code to a branch I can help with the basic skeleton of the parser/grammar
> > > > > >
> > > > > >
> > > > > > Thank you Balázs, my code is available here: https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl <https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl>
> > > > > >
> > > > > > m.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Jan 23, 2016 5:54 PM, "Marc Falzon" <m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu>> wrote:
> > > > > > > I successfully generated a module template using the `create_plugin.sh (http://create_plugin.sh) (http://create_plugin.sh) (http://create_plugin.sh)` script, however I don't have the slightest idea on how to get started from here (I mean for the 'plugin glue', I can manage the cURL part)... The most problematic step for me is the whole parser/grammar thing, to which I believe to involve YACC/Bison parsing but I don't know how it works and it's too big of a learning curve for what I want to achieve at this point... Since the Gitbook doesn't provide any useful information on module development at the moment, is there any way to get me started on this topic? I'm not a developer (sysadmin), and trying to learn from existing module doesn't quite cut it.
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > m.
> > > > > > >
> > > > > > >
> > > > > > > On Friday 22 January 2016 at 11:35, Tibor Benke wrote:
> > > > > > >
> > > > > > > > There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator <https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator> . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010 <https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c906209355f1b737a28010>
> > > > > > > >
> > > > > > > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu> (mailto:m at baha.mu <mailto:m at baha.mu>)>:
> > > > > > > > > Hi Bazsi,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote:
> > > > > > > > >
> > > > > > > > > > not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;)
> > > > > > > > >
> > > > > > > > > > if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough,
> > > > > > > > >
> > > > > > > > > Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved.
> > > > > > > > >
> > > > > > > > > m.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m at baha.mu (mailto:m at baha.mu) (mailto:m at baha.mu) <mailto:m at baha.mu> (mailto:m at baha.mu <mailto:m at baha.mu>) (mailto:m at baha.mu <mailto:m at baha.mu>)> wrote:
> > > > > > > > > > > Hi
> > > > > > > > > > >
> > > > > > > > > > > Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > m.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ______________________________________________________________________________
> > > > > > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ______________________________________________________________________________
> > > > > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ______________________________________________________________________________
> > > > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ______________________________________________________________________________
> > > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ______________________________________________________________________________
> > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > > >
> > > > > > > ______________________________________________________________________________
> > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ______________________________________________________________________________
> > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> > > > >
> > > >
> > >
> >
>
More information about the syslog-ng
mailing list