[syslog-ng] Help with filtering at destination based on custom macro value match

[Balazs Scheidler]

The standard syslog destinations do not send all name value pairs
automatically.

E.g. if you have MSG_TAG on the client, it will only be available there and
not on the server, unless the transport you are using to deliver it to
another syslog-ng instance does this for you.

You can roll your own template () on the client side which can be parsed on
the server, but again this parsing does not happen automatically (but
please read on as an alternative). I can see that you are using a custom
template (t_global) which indeed sends the value of MSG_TAG) as a prefix to
the normal message. When syslog-ng parses this on the server, it will put
this value into the PROGRAM name-value pair (and not MSG_TAG that you want
to filter on), since that's the spot the client side template inserted this
value.

An alternative to all of this is to use the syslog-ng () destination
driver, which will use a JSON based format to include all client-side name
value pairs.

The server automatically processes this if you used the
default-network-drivers() as source on the server (this opens all relevant
network ports and enabled automatic parsing of incoming messages).

If you don't want to use the whole of default-network-drivers (), you can
stick to a simpler source and then apply parsing of the ewmm() format,
using the ewmm-parser().

With that all name-value pairs would automatically make it to the server,
where you can trivially continue filtering on any fields that have already
been extracted.

Hope this helps,
Bazsi




On Sat, Jan 30, 2021, 04:36 Akshay Joshi <auj89in at gmail.com> wrote:

> My client is sending logs and it has the following config :
>
> *template t_global {template("<${PRI}>${LOGHOST} ${MSG_TAG}${MSGHDR}${MSG}\n"); };*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *# Global logging remote destination:#-----------------------------------destination d_global_remote {    tcp("proxy.dc.nuagedemo.net <http://proxy.dc.nuagedemo.net>" port(10514)        template(t_global)        tls(peer-verify(required-untrusted)            ca-dir('/etc/default/bootstrap/keys')            cert_file('/etc/default/bootstrap/keys/cert.pem')            key-file('/etc/default/bootstrap/keys/key.pem')        )        flags("threaded")    );};source s_nuageDiag {    file("/home/user/nuage/nuage_diagnostics_daemon.log"         follow-freq(10) default-facility(local1) default-priority(info) tags("nuageDiag"));};rewrite w_nuageDiag { set("nuage-diag: ", value("MSG_TAG") condition(tags("nuageDiag"))); };*
>
> On the destination, I have this :
>
>
>
>
>
>
>
>
>
>
>
>
> *source s_network {    tcp(        port(10514)        max-connections(1000)        tls(            peer-verify(required-untrusted)            key-file("/opt/proxy/config/keys/proxy-Key.pem")            cert_file("/opt/proxy/config/keys/proxyCert.pem")            ca-dir("/opt/proxy/config/keys/proxy-CA.pem")        )    );};*
>
>
> *filter nsg_diag {    match("nuage-diag: " value("MSG_TAG"));};filter f_messages { (level(info..warn) and filter (nsg_diag)); };*
>
>
>
>
>
>
>
> *destination d_logs {        file(            "/var/log/syslog-ng/logs.txt"            owner("root")            group("root")            perm(0777)            );};log { source(s_sys); source(s_network); filter(f_messages); destination(d_logs); };*
>
> I can write logs locally without the filtering. But with filtering, it
> does not match "nuage-diag: " macro.
> This "MSG_TAG" does not seem to be a standard header but a custom one. I
> couldn't find many straightforward examples on forums etc.. as well. I am
> missing a trick or two config-wise for sure.
>
> Any pointers / help will be much appreciated.
>
>
>
>
> --
> Regards,
> Akshay Joshi
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210130/0c90b11c/attachment.html>