[syslog-ng] syslog-ng for filtering and execute external application

Péter, Kókai peter.kokai at oneidentity.com
Mon Apr 29 05:40:06 UTC 2019 

Hello,

Does you get the messages over network or file ? (In your configuration I
saw network, but in your example messages I saw file like result)
or maybe does it send via network the same messages ? (withouth any header
and so on)

Okay, so if you want to trigger a shell script when "exit child, pid=1287"
is recieved (assuming that $MESSAGE macro contains the previous text).

You could simply do a filter and program destination (one possible
solution), as follows:

log {
   source(not-sure-about-this);
   filter { message("exit child"); };
   destination {
       program("your-executable-to-call");
   };

(I would recommend a simple code like this, if it is enough)

If you need the *pid* value parsed, and supplied to your application, you
can also do an additional *kv-parse*-ing step:

log {
        ....
        parser { kv-parser(prefix(".proc.")
extract-stray-words-into('.proc.action')); };


        destination {
                program("/tmp/print" template("$MESSAGE ${.proc.pid}
${.proc.action}\n"));
        };

This way the *${.proc.pid}* macro going to give you back the pid number.

In case you have to know that it has been started, or you have to do a
trigger if there is a possibility not to get "exit child"; you could use
for example *grouping-by*:

log {
        source { stdin(flags(no-parse)); };

        parser { kv-parser(prefix(".proc.")
extract-stray-words-into('.proc.action')); };

        parser { grouping-by(
                        key("${.proc.pid}")
                        aggregate(
                                inherit-mode(last-message)
                                value(".proc.trigger" "1")
                        )
                        timeout(60) #if no exit child is recieved, it is
going to trigger a message anyway
                        trigger( message("exit child") )
                        inject-mode("pass-through")
                );
        };

        filter { match("1" value(".proc.trigger")); };

        destination {
                program("/tmp/print" template("$MESSAGE ${.proc.pid}
${.proc.trigger}\n"));
        };
};


This last example either going to trigger if there has been any message
with pid=9999 and one with "exit child" or the 60 sec timeout triggered.

Note: this last example probably could be improved uppon, providing more
filter, and proper aggregation if needed; but also probably an overkill
just to trigger message on "exit child".

--
Kokan

On Fri, Apr 26, 2019 at 9:03 PM Thomas Schmiedl <thomas.schmiedl at web.de>
wrote:

> Hello Péter,
>
> could you send me an example, it's complicated for a beginner. xupnpd2
> sends such messages:
>
> run child, pid=1283
> exit child, pid=1283
> run child, pid=1284
> exit child, pid=1284
> run child, pid=1285
> exit child, pid=1285
> run child, pid=1286
> exit child, pid=1286
> run child, pid=1287
> using handler 'hls' for
> 'http://sochi-strk.ru:1936/strk/strk.stream/playlist.m3u8'
> exit child, pid=1287
> run child, pid=1289
> run child, pid=1290
> run child, pid=1291
> run child, pid=1293
> run child, pid=1292
> run child, pid=1294
> exit child, pid=1289
> exit child, pid=1290
>
> On "exit child, pid=1287" should be executed the shellscript to stop
> ffmpeg.
>
> Thanks,
> Thomas
>
> Am 26.04.2019 um 18:26 schrieb Péter, Kókai:
> > Hello,
> >
> > I think you are looking for *grouping-by*, that can group together
> multiple
> > messages.
> > You could use *kv-parse* to parse the exit child, pid=1234 and use *pid*
> as
> > key to group the messages.
> >
> >
> > --
> > Kokan
> >
> > On Fri, Apr 26, 2019 at 5:25 PM Thomas Schmiedl <thomas.schmiedl at web.de>
> > wrote:
> >
> >> Hallo,
> >>
> >> for a first test, I wrote this small conf, which matches the first
> >> stream in
> >> https://github.com/clark15b/xupnpd2/blob/master/media/example.m3u.
> >>
> >> @version: 3.20
> >>
> >> source s0 { udp(ip(0.0.0.0) port(514)); };
> >>
> >> destination d0 { file("/home/user/syslog-ng-intel/test"); };
> >>
> >> log { source(s0); filter { message(".*\[(.*)\].*strk\.stream.*"
> >> flags("store-matches")); }; destination(d0); };
> >>
> >> Is it possible to get the variable value of the stored match from the
> >> log for a second log to match "exit child, pid=<variable>"?
> >>
> >> Thanks,
> >> Thomas
> >>
> >> Am 25.04.2019 um 08:46 schrieb Péter, Kókai:
> >>> Hello,
> >>>
> >>> Assuming the log file looks like this:
> >>>
> >>> ```
> >>> run child, pid = 1715
> >>> using handler 'hls' for 'http://localhost/'
> >>> exit child, pid = 1715
> >>> ```
> >>>
> >>> Filters and source could be something like this:
> >>> ```
> >>> @version: 3.20
> >>> @include "scl.conf"
> >>>
> >>> source xupnpd2 {
> >>>     file("/tmp/xup" flags(no-parse)); # no-parse needed to set
> $MESSAGE,
> >> if
> >>> the file does contain only the above lines it is fine not to parse
> >>> };
> >>>
> >>> log { source(xupnpd2); filter { message("run child"); };
> >> destination(d0); };
> >>> log { source(xupnpd2); filter { message("exit child"); } ;
> >> destination(d0);
> >>> };
> >>> ```
> >>>
> >>> If needed syslog-ng could parse, pid and you can transfare more
> >> structured
> >>> to the program destination.
> >>>
> >>> --
> >>> Kokan
> >>>
> >>> On Wed, Apr 24, 2019 at 4:15 PM Thomas Schmiedl <
> thomas.schmiedl at web.de>
> >>> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> thanks Péter for your reply. Please could you write the filters
> >>>>
> >>>> 1. for the start (2 lines with a variable 'pid' number):
> >>>> run child, pid = 1715
> >>>> using handler 'hls' for '... URL ...'
> >>>>
> >>>> 2. for the stop:
> >>>> exit child, pid = 1715
> >>>>
> >>>> Thanks,
> >>>> Thomas
> >>>>
> >>>> Am 24.04.2019 um 15:05 schrieb Péter, Kókai:
> >>>>> Hello,
> >>>>>
> >>>>> 'I have the question, if I could use syslog-ng to filter the start
> and
> >>>> stop
> >>>>> of the stream'
> >>>>> Yes, it can filter those messages (disclaimer without seeing those
> >>>>> messages).
> >>>>>
> >>>>> 'and execute an action "ffmpeg stop" and "delete video segments".'
> >>>>> It was not something syslog-ng is designed to, but for example you
> >> could
> >>>>> use program destination to execute arbitrary executable (like s small
> >>>>> script to call ffmpeg stop).
> >>>>>
> >>>>> You could do something like this (not tested) to have separate things
> >> to
> >>>> do
> >>>>> based on start/stop:
> >>>>>
> >>>>> @version: 3.20
> >>>>> source xupnpd2 {
> >>>>>      stdin(flags(no-parse));
> >>>>> };
> >>>>>
> >>>>> destination start {
> >>>>>      program("/usr/bin/do-start-magic.sh");
> >>>>> };
> >>>>>
> >>>>> destination stop {
> >>>>>      program("/usr/bin/do-stop-magic.sh");
> >>>>> };
> >>>>>
> >>>>> log { source(xupnpd2); filter { program("xupnpd2") AND
> >> message("start");
> >>>> };
> >>>>> destination(start); };
> >>>>> log { source(xupnpd2); filter { program("xupnpd2") AND
> message("stop");
> >>>> } ;
> >>>>> destination(stop); };
> >>>>>
> >>>>>
> >>>>> Each log/event is a single new line to the program stdin, which it
> >> should
> >>>>> process.
> >>>>>
> >>>>> --
> >>>>> Kokan
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 24, 2019 at 2:36 PM Thomas Schmiedl <
> >> thomas.schmiedl at web.de>
> >>>>> wrote:
> >>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> I use the mediaserver xupnpd2 (https://github.com/clark15b/xupnpd2)
> >> on
> >>>>>> the router to show HLS streams on the TV. Unfortunately, the
> software
> >> is
> >>>>>> no longer maintained by the original developer (I'm not a developer
> >>>>>> myself). The HLS processing would have to be revised.
> >>>>>>
> >>>>>> My idea is to start a ffmpeg and a local web server on the router,
> if
> >> I
> >>>>>> want to display the stream (the script xupnpd.lua will be executed,
> >>>>>> which starts ffmpeg etc.). Unfortunately there is not such a script
> >> when
> >>>>>> terminating the stream (exit ffmpeg etc.).
> >>>>>>
> >>>>>> xupnpd2 uses a log where you can track the start as well as the
> ending
> >>>>>> of the stream. Since the router has only a small amount of internal
> >>>>>> memory to write and analyze a logfile, I have the question, if I
> could
> >>>>>> use syslog-ng to filter the start and stop of the stream and execute
> >> an
> >>>>>> action "ffmpeg stop" and "delete video segments". I have very little
> >>>>>> Linux knowledge, maybe you can help me.
> >>>>>>
> >>>>>> Here is the excerpt from the xupnpd2 log (Loglevel 8).
> >>>>>>
> >>>>>> When starting the stream:
> >>>>>> run child, pid = 1715
> >>>>>> using handler 'hls' for '... URL ...'
> >>>>>>
> >>>>>> When stopping the stream:
> >>>>>> exit child, pid = 1715
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Thomas Schmiedl
> >>>>>>
> >>>>>>
> >>>>
> >>
> ______________________________________________________________________________
> >>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>>>> Documentation:
> >>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> ______________________________________________________________________________
> >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>>> Documentation:
> >>>> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>>
> >>>>
> >>>>
> >>
> ______________________________________________________________________________
> >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>> Documentation:
> >>>> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> ______________________________________________________________________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> Documentation:
> >> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>
> >>
> >>
> ______________________________________________________________________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation:
> >> http://www.balabit.com/support/documentation/?product=syslog-ng
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>
> >>
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190429/31bf7e46/attachment-0001.html>

More information about the syslog-ng mailing list