Re: [syslog-ng] Message and Header are being split incorrectly

5 Jun 2020

      Hi Peter,

thanks a lot, this resolved the issue without having to use hacky templates.

Still it bugs me out that rewriting $PROGRAM didn't work. Any idea why?
According to the other list posting it should've.

Cheers,
Fabian

Von: syslog-ng <syslog-ng-bounces@lists.balabit.hu> Im Auftrag von Peter Czanik (pczanik)
Gesendet: Freitag, 5. Juni 2020 08:36
An: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Betreff: Re: [syslog-ng] Message and Header are being split incorrectly

Hi,

FreeBSD 12.1 changed from the legacy syslog protocol to RFC 5424 format. When you use the system() source for local logs, this is handled automagically. But you can also fix it by hand: https://github.com/syslog-ng/syslog-ng/issues/2428

Bye,

Peter Czanik (CzP) <peter.czanik@oneidentity.com<mailto:peter.czanik@oneidentity.com>>
Balabit (a OneIdentity company) / syslog-ng upstream
https://syslog-ng.com/community/
https://twitter.com/PCzanik

________________________________
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Sass, Fabian <Fabian.Sass@f-i-ts.de<mailto:Fabian.Sass@f-i-ts.de>>
Sent: Friday, June 5, 2020 08:28
To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>
Subject: [syslog-ng] Message and Header are being split incorrectly

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi,

since updating the Base OS to FreeBSD 12.1 syslog-ng 3.27.1 fails splitting a message and its headers.

Jun  5 08:12:00 myhostname 1 2020-06-05T08:12:00.042109+02:00 myhostname /usr/sbin/cron 71149 - - (root) RELOAD (tabs/root)

Jun  5 08:12:01 myhostname 1 2020-06-05T08:12:01.546089+02:00 myhostname named 54403 - - client @0xfffffff 0.0.0.0: update 'some.domain/IN' denied

Using templates for the destination reveals that the $MSGHDR Macro only holds the value "1", which is the wrongly extracted $PROGRAM macro.

$MESSAGE itself contains almost all information of the lines above, to be precise everything from (including) the $ISODATE to the end of the line.
However using templates and rewrite rules is in this case sufficient to restore the logformat that was used before the update.

The bigger issue is that changing the value of $PROGRAM has no effect when sending it to antoher syslog-ng loghost.
The behavior seems to be analog to this bug:
https://lists.balabit.hu/pipermail/syslog-ng/2011-August/017132.html<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fpipermail%2Fsyslog-ng%2F2011-August%2F017132.html&data=02%7C01%7CPeter.Czanik%40oneidentity.com%7C4be342ff90b940df59d008d80919b8e7%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637269353395272488&sdata=h4MEu8GH%2BKK6tbdxi8BvV6r5eBvoZ%2BF44JfzMLj6TbU%3D&reserved=0>

As you can see in my syslog-ng.conf the $PROGRAM macro is overwritten to "named" if named was logging to the local syslog-ng. The successful overwriting is verified using a separate logfile (destination d_test):

@version: 3.5

#

# options

#

options {

  mark_freq(3600);

  flush_lines(0);

  dir_perm(0640);

  chain_hostnames(off);

  keep_hostname(yes);

  create_dirs(yes);

  use_dns(yes);

  dns_cache(yes);

  dns_cache_expire(3600);

};
source s_all { unix-dgram("/var/run/log");

             unix-dgram("/var/run/logpriv" perm(0600));

         internal();

};

#

# rewrite since syslog message splitting is broken since update to freebsd12...

#

rewrite r_msg {

    set(

        "named", value("PROGRAM") condition(message(".* named [0-9]+ - -.*"))

    );

    subst(".* ([a-zA-Z/\._]+) ([0-9]+) - - (.*)", "$1[$2]: $3", value("MESSAGE"));

};

#

# destinations

#

destination d_test { file("/var/log/fabian_messages" template("$DATE $PROGRAM $HOST $MESSAGE\n")); };

destination d_test2 { file("/var/log/fabian_messages2"); };

destination d_messages { file("/var/log/messages" template("$DATE $HOST $MESSAGE\n")); };

destination d_loghost  { tcp("someiphere" port(514) template("$DATE $HOST $MESSAGE\n")); udp("anotheriphere" port(10525) template("$DATE $HOST $MESSAGE\n")); };

#

#

# logging

#

log { source(s_all); rewrite(r_msg); destination(d_messages); };

log { source(s_all); rewrite(r_msg); destination(d_loghost); };

log { source(s_all); rewrite(r_msg); destination(d_test); };

log { source(s_all); destination(d_test2); };

Am I missing something here, or is syslog-ng somehow behaving unintended?

Fabian