Re: [syslog-ng] Zulu timestamp question

21 May 2021

      Hi Daniel,

I think the problem is that these messages do not conform to a known syslog format, so syslog-ng does its best to guess which part of the message means what.
I observed, that if adding changing "2021-05-19T1:14:14Z " to "2021-05-19T01:14:14Z" (so adding a 0 before the hour), it works as expected. However, doing this automatically is quite hard.

If a log message comes with a specific format, we can write the parsing ourselves. There are multiple ways of doing that, I prepared one, which uses the match() filter with regexp and date-parser().
See my example config below:
source s_src {
  network(
    transport("tcp")
    port(12378)
    flags(no-parse)
  );
};

filter f_regexp {
  match("^([^ ]+) ([^ ]+) ([^ ]+) (.*)$" template("${MESSAGE}") flags(store-matches));
};

rewrite r_set_regexp {
  set("$2" value("HOST"));
  set("$3" value("PROGRAM"));
  set("$4" value("MESSAGE"));
};

parser p_date {
  date-parser(template("$1"));
};

destination d_local {
  file("/dev/stdout");
};

log {
  source(s_src);

  filter(f_regexp);
  parser(p_date);
  rewrite(r_set_regexp);

  destination(d_local);
};

The interesing parts are f_set_regexp, r_regexp, p_date and the flags(no-parse) in the source.
In a nutshell:
 1. The flags(no-parse) says, that we want to parse our incoming message manually, we do not need automatic message parsing.
 2. The f_regexp filter matches the log for the pattern given, and stores the matches in $0, $1, $2...
 3. The p_date parser parses the first found entry from f_regexp (that will be the date).
 4. the r_set_regexp rewrite sets the rest of the regexp matches to the correct macros.

This config correctly handled both of your messages for me.
Please give it a try!

Cheers,
Attila

________________________________
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au>
Sent: Thursday, May 20, 2021 1:35 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: Re: [syslog-ng] using VARARGS correctly

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Everyone,

I am having an issue when the Zulu timestamp is between 10 and 23:59. i.e. the logs format differently before 10AM and after 10AM.

I have captured in a tcpdump the syslogs coming in and they both seem the same.
We're at GMT+10 so this event was as 11:14:14 on 19th May.
Msg: 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d

This event was 09:07:47 the next day the 20th May:
Msg: 2021-05-19T23:07:46Z 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d

In the output files, both events go to the 0519.log file, until 10AM or 00:00:00Z the next day.
First event logs as:
May 19 11:14:14 10.18.0.14 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O

Second event logs as:
May 19 23:07:46 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O

I assume some built-in filtering is changing the way these are parsed in syslog-ng?

I have tried to play with raw message filtering but it doesn't take the conf file:
@version:3.5
@include "scl.conf"

# syslog-ng configuration file.
#

options {
   chain_hostnames(no);
   create_dirs (yes);
   dir_perm(0755);
   dns_cache(yes);
   keep_hostname(yes);
   log_fifo_size(2048);
   log_msg_size(8192);
   perm(0644);
   time_reopen (10);
   use_dns(yes);
   use_fqdn(yes);
};

source s_network {
   udp(port(514));
};

source attivo {
   tcp(port(514));
};

### DESTINATIONS
destination d_files_splunk {
   file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes));
};
destination d_files_nti {
   file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes) template(t_nti));
};

### FILTERS
filter nti {
   host("10.18.0.14" type(glob));
};
filter splunk {
   not (filter(nti));
};

### LOG
log {
   source(s_network);
   #filter(splunk);
   destination(d_files_splunk);
};
log {
   source(s_network);
   filter(nti);
   destination(d_files_nti);
};
log {
   source(attivo);
#   filter(splunk);
   destination(d_files_splunk);
};

### TEMPLATES
template t_nti {
        template("${RAWMSG}\n")
};

Any help is appreciated.
Thanks
Daniel Ehrlich
__________________________________________________________________
This email (including any attached files) is confidential and is
for the intended recipient(s) only. If you received this email by
mistake, please, as a courtesy, tell the sender, then delete this
email.
The views and opinions are the originator's and do not necessarily
reflect those of the University of Southern Queensland. Although
all reasonable precautions were taken to ensure that this email
contained no viruses at the time it was sent we accept no
liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider
of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)

Re: [syslog-ng] Zulu timestamp question

Attila Szakacs (aszakacs)