<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi Jon,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I've checked your example log and it seems that it doesn't conform to syslog protocol (RFC5424 - <a href="https://tools.ietf.org/html/rfc5424#section-6.3.3" id="LPlnk">https://tools.ietf.org/html/rfc5424#section-6.3.3</a> ).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The "]" characters in SDATA values has to be escaped.<br>
I give it a try with this log and it worked with syslog() source:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
<span style="font-family: Consolas, Courier, monospace;"><141>1 2020-08-04T08:10:58.769127-05:00 fqdn.example.com apmd 12374 01490113:5: [F5@12276 hostname="fqdn" errdefs_msgno="01490113:5:" partition_name="Common" session_id="1c95e1e7" Access_Profile="/Common/blah"
Partition="Common" Session_Id="1c95e1e6" <b>Session_Variable_Name="session.machine_info.last.net_adapter.list.[0\].mac_address"</b> Session_Variable_Value="3C:D9:2B:33:9A:8E"] /Common/<blah>:Common:1c95e1e6: session.machine_info.last.net_adapter.list.[0].mac_address
is 3C:D9:2B:33:9A:8E</span><br>
</div>
<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview _EReadonly_1"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
You can debug message parsing of syslog-ng (with -t option or set it with "sbin/syslog-ng-ctl trace -s 1").<br>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Regards,<br>
Gábor</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Wilson, Jonathan <jonathan.wilson@vumc.org><br>
<b>Sent:</b> Monday, August 31, 2020 23:31<br>
<b>To:</b> syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] message being consistently dropped</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal">We are running syslog-ng, open source edition, version 3.22.1, on RedHat Enterprise version 7.8. Among many other data sources, we receive syslog data from an F5 device that acts as a VPN server. One type of message that it sends us is
consistently not making it into the log (though everything else, to my knowledge, does.) I wonder if anyone sees something wrong with the message that would prevent its being parsed/stored.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">The message (reconstructed from a packet capture, identification fields redacted or changed):</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New""><141>1 2020-08-04T08:10:58.769127-05:00 <fqdn here> apmd 12374 01490113:5:
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">[F5@12276 hostname="fqdn" errdefs_msgno="01490113:5:"
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">partition_name="Common" session_id="1c95e1e7" Access_Profile="/Common/blah"
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">Partition="Common" Session_Id="1c95e1e6"
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">Session_Variable_Name="session.machine_info.last.net_adapter.list.[0].mac_address"
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">Session_Variable_Value="3C:D9:2B:33:9A:8E"]</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">/Common/<blah>:Common:1c95e1e6: session.machine_info.last.net_adapter.list.[0].mac_address is 3C:D9:2B:33:9A:8E</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New""> </span></p>
<p class="x_MsoNormal">I inserted the line breaks to help Outlook. Note that although the message appears to be RFC5424-compliant I have discovered that it does not have whatever magic headers let syslog-ng handle it as structured data transparently. As a result,
I cannot use the “syslog” transport but have to use the generic “network” transport and then use flags “syslog-protocol”. This works OK for everything else from this source.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">The message doesn’t appear in our syslog logs at all. It’s consistent.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">The relevant parts of the syslog-ng configuration look like:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">options {</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> ts_format(iso);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> use_dns(yes);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> dns_cache(1000);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> dns_cache_expire(30);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> use_fqdn(yes);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> keep_hostname(no);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> chain-hostnames(no);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> keep_timestamp(yes);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> stats_level(1);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> stats_freq(3600);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> normalize_hostnames(yes);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> log_fifo_size(10000);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> flush_lines(1000);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> log-msg-size(65536);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> trim-large-messages(yes);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> threaded(yes);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> time_reap(10);
</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> create_dirs(yes);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> dir_group(esmapp);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> dir_owner(esmapp);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> group(esmapp);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> owner(esmapp);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> perm(0644);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">};</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New""> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">source s_F5VPN_source {</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> network(</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> transport("tcp")</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> ip(0.0.0.0)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> port(11000)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> keep-timestamp(yes)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> flags(no-multi-line,syslog-protocol)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> max-connections(50)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> log_iw_size(10000)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> log-fetch-limit(20)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> );</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">};</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Courier New""> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt; font-family:"Courier New"">filter f_F5VPN_client { netmask(<blah>) and facility(local1); };</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Courier New""> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">destination d_F5VPN_client {</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> file(</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> "/home/syslog/F5VPN_client.log"</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> ts_format(rfc3164)</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> );</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">};</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Courier New""> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">log {</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> source(s_F5VPN_source);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> filter(f_F5VPN_client);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> destination(d_F5VPN_client);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New""> flags(flow-control);</span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-autospace:none"><span style="font-size:10.0pt; font-family:"Courier New"">};</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"> </p>
<p class="x_MsoNormal">Regards,</p>
<p class="x_MsoNormal">Jon</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b><span style="color:black">Jon Wilson | Principal System Engineer, IT Service Management | Information Technology | Vanderbilt University Medical Center</span></b><span style="color:black"> <br>
<a href="mailto:jonathan.wilson@vumc.org"><span style="color:blue">jonathan.wilson@vumc.org</span></a> | phone: 615-440-7895 | fax: 615-323-2181</span></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</div>
</body>
</html>