<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div id="appendonsend" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Daniel,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think the problem is that these messages do not conform to a known syslog format, so syslog-ng does its best to guess which part of the message means what.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I observed, that if adding changing "<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">2021-05-19T1:14:14Z<span> " to "<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">2021-05-19T01:14:14Z"
(so adding a 0 before the hour), it works as expected. However, doing this automatically is quite hard.</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><br>
</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">If a log message comes with a specific format, we
can write the parsing ourselves. There are multiple ways of doing that, I prepared one, which uses the match() filter with regexp and date-parser().</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">See my example config below:</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">
<div style="color: rgb(212, 212, 212); background-color: rgb(30, 30, 30); font-family: "Droid Sans Mono", monospace, monospace, "Droid Sans Fallback"; font-weight: normal; font-size: 14px; line-height: 19px;">
<div><span>source s_src {</span></div>
<div><span> network(</span></div>
<div><span> transport(</span><span style="color: rgb(206, 145, 120);">"tcp"</span><span>)</span></div>
<div><span> port(12378)</span></div>
<div><span> flags(no-parse)</span></div>
<div><span> )</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span>}</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span>filter f_regexp {</span></div>
<div><span> match(</span><span style="color: rgb(206, 145, 120);">"^([^ ]+) ([^ ]+) ([^ ]+) (.*)$"</span><span> template(</span><span style="color: rgb(206, 145, 120);">"${MESSAGE}"</span><span>) flags(store-matches))</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span>}</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span>rewrite r_set_regexp {</span></div>
<div><span> set(</span><span style="color: rgb(206, 145, 120);">"$2"</span><span> value(</span><span style="color: rgb(206, 145, 120);">"HOST"</span><span>))</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span> set(</span><span style="color: rgb(206, 145, 120);">"$3"</span><span> value(</span><span style="color: rgb(206, 145, 120);">"PROGRAM"</span><span>))</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span> set(</span><span style="color: rgb(206, 145, 120);">"$4"</span><span> value(</span><span style="color: rgb(206, 145, 120);">"MESSAGE"</span><span>))</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span>}</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span>parser p_date {</span></div>
<div><span> date-parser(template(</span><span style="color: rgb(206, 145, 120);">"$1"</span><span>))</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span>}</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span>destination d_local {</span></div>
<div><span> file(</span><span style="color: rgb(206, 145, 120);">"/dev/stdout"</span><span>)</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span>}</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span>log { </span></div>
<div><span> source(s_src)</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span> filter(f_regexp)</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span> parser(<span style="color: rgb(212, 212, 212); background-color: rgb(30, 30, 30); display: inline !important;">p_date</span>)</span><span style="color: rgb(106, 153, 85);">;</span></div>
<div><span> rewrite(r_set_regexp)</span><span style="color: rgb(106, 153, 85);">;</span></div>
<br>
<div><span> destination(d_local)</span><span style="color: rgb(106, 153, 85);">;</span></div>
<span><span>}</span><span style="color: rgb(106, 153, 85);">;</span></span></div>
<br>
</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">The interesing parts are f_set_regexp, r_regexp,
p_date and the flags(no-parse) in the source.</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">In a nutshell:</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"> 1. The flags(no-parse) says, that we want to parse
our incoming message manually, we do not need automatic message parsing.</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"> 2. The f_regexp filter matches the log for the pattern
given, and stores the matches in $0, $1, $2...</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"> 3. The p_date parser parses the first found entry
from f_regexp (that will be the date).</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"> 4. the r_set_regexp rewrite sets the rest of the
regexp matches to the correct macros.</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><br>
</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">This config correctly handled both of your messages
for me.</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Please give it a try!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Attila</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au><br>
<b>Sent:</b> Thursday, May 20, 2021 1:35 AM<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] using VARARGS correctly</font>
<div> </div>
</div>
<div dir="ltr">
<div style="background-color: rgb(255, 235, 156); width: 100%; border-style: solid; border-color: rgb(156, 101, 0); border-width: 1pt; padding: 2pt; font-size: 10pt; line-height: 12pt; font-family: Calibri; color: black; text-align: left;">
<span style="color: rgb(156, 101, 0); font-weight: bold;">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Everyone,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am having an issue when the Zulu timestamp is between 10 and 23:59. i.e. the logs format differently before 10AM and after 10AM.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have captured in a tcpdump the syslogs coming in and they both seem the same. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">We're at GMT+10 so this event was as </span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">11:14:14
on 19th May.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Msg: 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<span style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<span style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">This event was 09:07:47 the next day the 20th May:</span></div>
<div style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<span style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">Msg: 2021-05-19T23:07:46Z 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d</span></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In the output files, both events go to the 0519.log file, until 10AM or 00:00:00Z the next day.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
First event logs as:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
May 19 11:14:14 10.18.0.14 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Second event logs as:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
May 19 23:07:46 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I assume some built-in filtering is changing the way these are parsed in syslog-ng?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div>
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I have tried to play with raw message filtering but it doesn't take the conf file:</div>
<div style="color: rgb(212, 212, 212); background-color: rgb(30, 30, 30); font-family: "Droid Sans Mono", monospace, monospace, "Droid Sans Fallback"; font-weight: normal; font-size: 14px; line-height: 19px;">
<span style="color: rgb(106, 153, 85);">@version:3.5</span><br>
</div>
<div style="color: rgb(212, 212, 212); background-color: rgb(30, 30, 30); font-family: "Droid Sans Mono", monospace, monospace, "Droid Sans Fallback"; font-weight: normal; font-size: 14px; line-height: 19px;">
<span><span style="color: rgb(106, 153, 85);">
<div>@include "scl.conf"</div>
<div><br>
</div>
<div># syslog-ng configuration file.</div>
<div>#</div>
<div><br>
</div>
<div>options {</div>
<div> chain_hostnames(no);</div>
<div> create_dirs (yes);</div>
<div> dir_perm(0755);</div>
<div> dns_cache(yes);</div>
<div> keep_hostname(yes);</div>
<div> log_fifo_size(2048);</div>
<div> log_msg_size(8192);</div>
<div> perm(0644);</div>
<div> time_reopen (10);</div>
<div> use_dns(yes);</div>
<div> use_fqdn(yes);</div>
<div>};</div>
<div><br>
</div>
<div>source s_network {</div>
<div> udp(port(514));</div>
<div>};</div>
<div><br>
</div>
<div>source attivo {</div>
<div> tcp(port(514));</div>
<div>};</div>
<div><br>
</div>
<div>### DESTINATIONS</div>
<div>destination d_files_splunk {</div>
<div> file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes));</div>
<div>};</div>
<div>destination d_files_nti {</div>
<div> file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes) template(t_nti));</div>
<div>};</div>
<div><br>
</div>
<div>### FILTERS</div>
<div>filter nti {</div>
<div> host("10.18.0.14" type(glob));</div>
<div>};</div>
<div>filter splunk {</div>
<div> not (filter(nti));</div>
<div>};</div>
<div><br>
</div>
<div>### LOG</div>
<div>log {</div>
<div> source(s_network);</div>
<div> #filter(splunk);</div>
<div> destination(d_files_splunk);</div>
<div>};</div>
<div>log {</div>
<div> source(s_network);</div>
<div> filter(nti);</div>
<div> destination(d_files_nti);</div>
<div>};</div>
<div>log {</div>
<div> source(attivo);</div>
<div># filter(splunk);</div>
<div> destination(d_files_splunk);</div>
<div>};</div>
<div><br>
</div>
<div>### TEMPLATES</div>
<div>template t_nti {</div>
<div> template("${RAWMSG}\n")</div>
<span>};</span><br>
</span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="background-color: rgb(255, 255, 255); display: inline !important;">Any help is appreciated.</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="background-color: rgb(255, 255, 255); display: inline !important;">Thanks</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="background-color: rgb(255, 255, 255); display: inline !important;">Daniel Ehrlich</span></div>
</div>
</div>
<div>__________________________________________________________________
<div>This email (including any attached files) is confidential and is
<div>for the intended recipient(s) only. If you received this email by
<div>mistake, please, as a courtesy, tell the sender, then delete this
<div>email.<br>
<div>The views and opinions are the originator's and do not necessarily
<div>reflect those of the University of Southern Queensland. Although
<div>all reasonable precautions were taken to ensure that this email
<div>contained no viruses at the time it was sent we accept no
<div>liability for any losses arising from its receipt.<br>
<div>The University of Southern Queensland is a registered provider
<div>of education with the Australian Government.
<div>(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081) </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>