[syslog-ng] Message Macros

Wed Aug 26 17:27:26 UTC 2020

I had always thought that $MESSAGE also contained a header with host and timestamp information.  If you're saying that isn't so, then we must be getting the date included as part of the message from the originating host.  What I'm seeing below, along with the fact that I'm now seeing that some of the log files have the double date and some do not, seems to support that conclusion.

We simply are trying to get a message to look like this:

$ISODATE $HOST $MESSAGE

Looks like we may need to rewrite the messages to get them into a consistent format?

# Source
source pan_splunk_udp {
  network(port(514) so-reuseport(1) so-rcvbuf(268435456) transport("udp") flags(syslog-protocol) log-fetch-limit(256000) persist-name('udp1') );
  network(port(514) so-reuseport(1) so-rcvbuf(268435456) transport("udp") flags(syslog-protocol) log-fetch-limit(256000) persist-name('udp2') );
  network(port(514) so-reuseport(1) so-rcvbuf(268435456) transport("udp") flags(syslog-protocol) log-fetch-limit(256000) persist-name('udp3') );
  network(port(514) so-reuseport(1) so-rcvbuf(268435456) transport("udp") flags(syslog-protocol) log-fetch-limit(256000) persist-name('udp4') );
};

# Destination
destination d_file {
  file("`BASEPATH`/$location/$HOST/$app/${HOST}_$app.log"
  create_dirs(yes) dir-owner("root") dir-group("root") dir-perm(0700)
  owner("root") group("root") perm(0640)
  flags("threaded", "no-multi-line")
  template("${S_ISODATE} ${MESSAGE}\n"));
};

If I remove the template from the destination above, everything works fine.  The date and host is written on each line, however, the date is not in the format we would like since it does not include the year.

...
if {  # app=ise
    filter { message('-ise-'); };
    rewrite {
      set("ise" value("app"));
    };
    destination(d_file_no_location);
  }...

I'm having trouble finding a line that has the double date anywhere but in the Cisco ISE logs.  It may only be affecting those logs.  I think it's pretty clear now that the second date is part of the log message as it is sent by the client.

2020-08-25T15:34:44+00:00 Tue Mar 10 12:54:23 PDT 2020 by xxxxx, cisco-av-pair=lldp-tlv=lldpSystemName=XXX, cisco-av-pair=
lldp-tlv=lldpSystemCapabilitiesMap=00:04:00:04, cisco-av-pair=lldp-tlv=lldpTimeToLive=00:78, cisco-av-pair=lldp-tlv=lldpPortId=01:30, cisco-av...

-Mark

-----Original Message-----
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Peter Kokai (pkokai)
Sent: Wednesday, August 26, 2020 11:15
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [EXTERNAL] Re: [syslog-ng] Message Macros

Hello,

The ${MESSAGE} macro is not supposed to hold dates. If that macro holds a date that is like a parsing issue, or you simply useing flags(no-parse) flags in that source.

Would you mind sharing a configuration and input that produces double date in your case ?
Also if you have the possibility to turn on debug logs (probably not production) -Fedvt option can be helpful as it is going to print exactly what value the MESSAGE macro is assigned.

Example of such log:
```
[2020-08-26T18:13:38.652356] Setting value; name='MESSAGE', value='-- Generated message. --', msg='0x616000008780'
```

--
Kokan

On Wed, Aug 26, 2020 at 02:52:46PM +0000, Faine, Mark R. (MSFC-IS40)[NICS] wrote:
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> 
> Is it possible to break up the $MESSAGE macro into smaller macros that constitute the entire message?  The reason I ask is that we tried this format in our destination:
> 
> template("${S_ISODATE} ${MESSAGE}\n")
> 
> However, we get two dates now, since I'm assuming one is part of the $MESSAGE macro, and we also lost the $HOST in our messages.
> 
> We just want the output to be exactly like it would be without specifying a template but with an ISO date, followed by the hostname/ip, and the rest of the message excluding the date.
> 
> Thanks,
> -Mark
> 
> 

> ______________________________________________________________________________
> Member info: https://urldefense.proofpoint.com/v2/url?u=https-3A__nam05.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Flists.balabit.hu-252Fmailman-252Flistinfo-252Fsyslog-2Dng-26amp-3Bdata-3D02-257C01-257CPeter.Kokai-2540oneidentity.com-257C057ed565507a414368ad08d849cfb674-257C91c369b51c9e439c989c1867ec606603-257C0-257C0-257C637340503766665934-26amp-3Bsdata-3Dp0LXuFKDUFLJATIfEfRaxKwS6KBbQ6sOf41WQ1tg7Io-253D-26amp-3Breserved-3D0&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=E-SLUeSluo7fuNlUvHwG4xb7pJsguawBTr1XH3mfQuY&s=jJleP3qQ-NbpBybwLctaLu_mO_yECC9KTJUYWgGXKbI&e= 
> Documentation: https://urldefense.proofpoint.com/v2/url?u=https-3A__nam05.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.balabit.com-252Fsupport-252Fdocumentation-252F-253Fproduct-253Dsyslog-2Dng-26amp-3Bdata-3D02-257C01-257CPeter.Kokai-2540oneidentity.com-257C057ed565507a414368ad08d849cfb674-257C91c369b51c9e439c989c1867ec606603-257C0-257C0-257C637340503766665934-26amp-3Bsdata-3DkA5Q6fMI-252B2LjsMLTfChGQSScPna1ihSFdaqTZu6-252BB90-253D-26amp-3Breserved-3D0&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=E-SLUeSluo7fuNlUvHwG4xb7pJsguawBTr1XH3mfQuY&s=MMOUcPf1bVwEtbx5VdP-vhYu0BY_mxowvbKQPyR2Gos&e= 
> FAQ: https://urldefense.proofpoint.com/v2/url?u=https-3A__nam05.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.balabit.com-252Fwiki-252Fsyslog-2Dng-2Dfaq-26amp-3Bdata-3D02-257C01-257CPeter.Kokai-2540oneidentity.com-257C057ed565507a414368ad08d849cfb674-257C91c369b51c9e439c989c1867ec606603-257C0-257C0-257C637340503766665934-26amp-3Bsdata-3DNEFQSFZX7v-252B-252B70RgaDNoy5XH1xLJQCunuUAQsDgNX8M-253D-26amp-3Breserved-3D0&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=E-SLUeSluo7fuNlUvHwG4xb7pJsguawBTr1XH3mfQuY&s=AyYbUGL1sVlV_bm4yLoJkCPC9zh7gYrVL4NKI7_lFwg&e= 
> 
______________________________________________________________________________
Member info: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=E-SLUeSluo7fuNlUvHwG4xb7pJsguawBTr1XH3mfQuY&s=PtjMCm2vTlNm6lU-TDWZkf2_u51MT0NU41MAG2xJDPg&e= 
Documentation: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=E-SLUeSluo7fuNlUvHwG4xb7pJsguawBTr1XH3mfQuY&s=D5YkTeYmxCjfsAloJJM7_TRqyWqSntpivuUDZeFdmrk&e= 
FAQ: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwIGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=E-SLUeSluo7fuNlUvHwG4xb7pJsguawBTr1XH3mfQuY&s=Ew3f66BRZvJbjgMrmBg285jSl1UaGZmDkr2f77ymDJI&e= 