<div dir="auto">The standard syslog destinations do not send all name value pairs automatically.<div dir="auto"><br></div><div dir="auto">E.g. if you have MSG_TAG on the client, it will only be available there and not on the server, unless the transport you are using to deliver it to another syslog-ng instance does this for you.</div><div dir="auto"><br></div><div dir="auto">You can roll your own template () on the client side which can be parsed on the server, but again this parsing does not happen automatically (but please read on as an alternative). I can see that you are using a custom template (t_global) which indeed sends the value of MSG_TAG) as a prefix to the normal message. When syslog-ng parses this on the server, it will put this value into the PROGRAM name-value pair (and not MSG_TAG that you want to filter on), since that's the spot the client side template inserted this value.</div><div dir="auto"><br></div><div dir="auto">An alternative to all of this is to use the syslog-ng () destination driver, which will use a JSON based format to include all client-side name value pairs.</div><div dir="auto"><br></div><div dir="auto">The server automatically processes this if you used the default-network-drivers() as source on the server (this opens all relevant network ports and enabled automatic parsing of incoming messages).</div><div dir="auto"><br></div><div dir="auto">If you don't want to use the whole of default-network-drivers (), you can stick to a simpler source and then apply parsing of the ewmm() format, using the ewmm-parser().</div><div dir="auto"><br></div><div dir="auto">With that all name-value pairs would automatically make it to the server, where you can trivially continue filtering on any fields that have already been extracted.</div><div dir="auto"><br></div><div dir="auto">Hope this helps,</div><div dir="auto">Bazsi</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Sat, Jan 30, 2021, 04:36 Akshay Joshi <<a href="mailto:auj89in@gmail.com">auj89in@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>My client is sending logs and it has the following config :</div><div><br></div><div><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29)"><i>template t_global {template("<${PRI}>${LOGHOST} ${MSG_TAG}${MSGHDR}${MSG}\n"); };</i><span style="box-sizing:inherit;display:block;height:unset"></span><i># Global logging remote destination:<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>#-----------------------------------<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>destination d_global_remote {<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> tcp("<a href="http://proxy.dc.nuagedemo.net" target="_blank" rel="noreferrer">proxy.dc.nuagedemo.net</a>" port(10514)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> template(t_global)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> tls(peer-verify(required-untrusted)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> ca-dir('/etc/default/bootstrap/keys')<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> cert_file('/etc/default/bootstrap/keys/cert.pem')<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> key-file('/etc/default/bootstrap/keys/key.pem')<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> )<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> flags("threaded")<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> );<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>};<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>source s_nuageDiag {<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> file("/home/user/nuage/nuage_diagnostics_daemon.log"<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> follow-freq(10) default-facility(local1) default-priority(info) tags("nuageDiag"));<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>};<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>rewrite w_nuageDiag { set("nuage-diag: ", value("MSG_TAG") condition(tags("nuageDiag"))); };</i></pre><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29)"><font face="arial, sans-serif">On the destination, I have this :</font></pre><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29)"><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;white-space:pre-wrap;word-break:normal;border-radius:4px"><i>source s_network {<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> tcp(<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> port(10514)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> max-connections(1000)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> tls(<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> peer-verify(required-untrusted)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> key-file("/opt/proxy/config/keys/proxy-Key.pem")<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> cert_file("/opt/proxy/config/keys/proxyCert.pem")<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> ca-dir("/opt/proxy/config/keys/proxy-CA.pem")<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> )<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> );<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>};</i><span style="box-sizing:inherit;display:block;height:unset"></span><i>filter nsg_diag {<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> match("nuage-diag: " value("MSG_TAG"));<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>};<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>filter f_messages { (level(info..warn) and filter (nsg_diag)); };</i><span style="box-sizing:inherit;display:block;height:unset"></span><i>destination d_logs {<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> file(<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> "/var/log/syslog-ng/logs.txt"<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> owner("root")<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> group("root")<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> perm(0777)<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span> );<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>};<span style="box-sizing:inherit"><br style="box-sizing:inherit"></span>log { source(s_sys); source(s_network); filter(f_messages); destination(d_logs); };</i></pre>
<font face="arial, sans-serif">I can write logs locally without the filtering. But with filtering, it does not match "nuage-diag: " macro.<br>This "MSG_TAG" does not seem to be a standard header but a custom one. I couldn't find many straightforward examples on forums etc.. as well. I am missing a trick or two config-wise for sure.</font>
</pre><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29)"><font face="arial, sans-serif">Any pointers / help will be much appreciated.</font></pre><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29)"><i><br></i></pre></div><br clear="all"><div><br></div>-- <br><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,<br></div>Akshay Joshi<br></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br><br>
</blockquote></div></div>