[syslog-ng] Suppress UTF-8 BOM in incoming network messages

[Balazs Scheidler]

Hi,

You could add flags(syslog-protocol) to your network() source driver, so it
recognizes the new 5424 style input.

With that the BOM will be removed from MSG and you won't need the rewrite
either.

Balazs

On Mon, Sep 23, 2024, 21:17 Jeremy Utley <jutley at seedbox.com> wrote:

> I'm trying to use Syslog-NG to receive logs from our anti-virus web
> console (ESET Protect).  I've got the logs coming in successfully, but they
> seem to be adding a UTF-8 BOM as part of the message:
>
> [2024-09-23T19:03:36.709950] Incoming log entry; input='<15>1
> 2024-09-23T19:03:36.455Z b35f2351-8f78-4414-b964-415939a7b0fb ERAServer 43
> - - <feff>{"event_type" : "Audit_Event","ipv4" : "172.17.0.11","ipv6" :
> "","hostname" : "b35f2351-8f78-4414-b964-415939a7b0fb","source_uuid" :
> "b35f2351-8f78-4414-b964-415939a7b0fb","os_name" : "","occured" :
> "23-Sep-2024 19:03:36","group_name" : "","group_description" :
> "","severity" : "Information","domain" : "Mapped account","action" :
> "Logout","target" : "35396c68-5017-4ea9-b268-d209f8040de9","detail" :
> "Logging out mapped account \'Jeremy Utley\'.","user" : "Jeremy
> Utley","result" : "Success"}', msg='0x7f6020022150', rcptid='0'
>
> Because I only care about the JSON data being provided, I'm setting the
> "flags(no-parse)" option in my network source, and a rewrite filter to
> remove the initial data as follows:
>
> source s_network {
>         network(
>                 port(6514)
>                 transport("tls")
>                 flags(no-parse)
>                 tls(
>                         key-file("/etc/syslog-ng/certs/syslog.key")
>                         cert-file("/etc/syslog-ng/certs/syslog.crt")
>                         peer-verify(optional-untrusted)
>                 )
>         );
> };
>
> rewrite r_trimjson {
>   subst("^(.*)- - ", "", value("MESSAGE"));
> };
>
> destination d_json {
>     file("/var/log/json_data.log" template("${MESSAGE}\n"));
> };
>
> But I can't seem to eliminate the BOM from the output, and that is messing
> with JSON parsing, since that is added to the beginning of every line (but
> the first) in my json_data.log file.  I tried adding "\xfe\xff" to the
> rewrite in an attempt to eliminate it, but that didn't work - the rewrite
> does not even match at that point.
>
> Anyone have any suggestions on how I can deal with this?
>
> Thanks!
>
> Jeremy Utley
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20240925/5fa2ef86/attachment.htm>