<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Hello Mark,<br>
</span>
<div> your setup is really good. If those files are written with the "file" destination without using any templates, than the default format is BSD style syslog and "file" source will also read it without any problem.<br>
</div>
<div><br>
</div>
<div> I can only see two possible pitfalls with this setup:<br>
</div>
<div><br>
</div>
<div> 1. BSD style syslog format (RFC3164) do not handle SDATA (beside other things). So if your logs uses any enrichment, than those will be lost as you stated.<br>
</div>
<div> You can prevent this by changing to RFC5424, by using the "flags(syslog-protocol)" on both the source and destination side.<br>
</div>
<div><br>
</div>
<div> 2. The file source/destination do not use message framing. So if your logs uses any new line (\n) character (i.e. in case of embedded debug, stack trace messages) it can cause problems with message parsing while reading it back from the files.<br>
</div>
<div> - The easiest way to prevent this problem is to deal with the new line characters before writing those logs into the file. - "flags(no-multi-line)"<br>
</div>
<div> - However simply removing them, might be not good for you. A less fortunate solution will depends on your message context, and uses the "multi-line-*" options (https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.25/administration-guide/19#TOPIC-1349359)
of the file source.<br>
</div>
<div> <br>
</div>
<div><br>
</div>
<div>Best regards,<br>
</div>
<div>Laci<br>
</div>
<span></span><br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov><br>
<b>Sent:</b> Monday, July 6, 2020 21:04<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] Convert logs back into syslog-ng logs</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xmsochpdefault, li.x_xmsochpdefault, div.x_xmsochpdefault
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
span.x_xmsohyperlink
{color:#0563C1;
text-decoration:underline}
span.x_xmsohyperlinkfollowed
{color:#954F72;
text-decoration:underline}
span.x_xemailstyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
span.x_EmailStyle24
{font-family:"Calibri",sans-serif;
color:#1F497D}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">syslog-ng is writing the logs originally.<span style="">
</span>They are gzipped older files.<span style=""> </span>If we find a gap in the logs we use these backed up logs on a secondary server to fill the gap.<span style="">
</span>I was hoping to just read them from the file and forward them to the syslog-ng server with the gap.</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">They come from many different sources from VPNs, firewalls, switches, etc. but I think the format is syslog format.</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">They all look like this but of course the IP and MSG varies.</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Jul<span style="">
</span>1 01:09:44 IP MSG</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">These would be gzipped files that are on another server and not being written to any longer.
<span style=""> </span>My concern was that there were originally headers and such that are not necessarily printed in the log file as it is written and they may be important in the processing of the messages.</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Thanks,</span></p>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#1F497D">-Mark</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> syslog-ng <syslog-ng-bounces@lists.balabit.hu>
<b>On Behalf Of </b>Laszlo Szemere (lszemere)<br>
<b>Sent:</b> Monday, July 6, 2020 13:39<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [EXTERNAL] Re: [syslog-ng] Convert logs back into syslog-ng logs</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black">Hello Mark,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> there are many (too many?) options to do this. To avoid "solving a problem that doesn't exist", we should start from your specification. Some helping question:</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> - What is writing those files in the first place? (If the soul purpose of those files is to store messages temporary,
<span style="background:white">Syslog-ng has a built in file buffer solution</span>.)</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> - What is the format of the individual messages in those files?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> - What is the life cycle of those files. (It is always hazardous if two application is writing and reading the same file at the same time without any synchronization.)</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black">To speed things up:</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black">IF !!! If your file is in syslog format, than a simple "file" source will do the job for you. For further information please read our administration guide: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__www.syslog-2Dng.com_technical-2Ddocuments_doc_syslog-2Dng-2Dopen-2Dsource-2Dedition_3.21_administration-2Dguide_18-23TOPIC-2D1180429%26d%3DDwMFAg%26c%3DApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk%26r%3DzMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo%26m%3DsWPK1NMGXp0akTJ1ru-sUVBli50Z1R1VWi2zEt5lWcY%26s%3DhiWjVhnphhXKrqoh5qL182DxTfqNCwEgltKNnA18bqs%26e%3D&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7Ca0f64e3c240b446549ff08d821df6d82%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637296590810369694&sdata=lreO0YsUAVXdusZUxDqaYMzGq8lGRn9mn%2FpehxUqUu8%3D&reserved=0" originalsrc="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.syslog-2Dng.com_technical-2Ddocuments_doc_syslog-2Dng-2Dopen-2Dsource-2Dedition_3.21_administration-2Dguide_18-23TOPIC-2D1180429&d=DwMFAg&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=sWPK1NMGXp0akTJ1ru-sUVBli50Z1R1VWi2zEt5lWcY&s=hiWjVhnphhXKrqoh5qL182DxTfqNCwEgltKNnA18bqs&e=" shash="CylKYW9EmX5Nt7czD7bpI4Thnv7Ci7pYKOlcKmzuZ0m2hV5aUL50oi0gnTDfHm2GhTZ0fwxgFXxNXdbq2FBqdKKd8/DcDYajIOZT/Iinu2ZRV1jP4YbvB5DtKVk+jYkUa1kVyN4lDhDtg5u2WvC/PYo2JkbbFGsPCTNf3rSwn98=">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.21/administration-guide/18#TOPIC-1180429</a></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black">Best regards,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black">Laci</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center"><span style="">
<hr size="3" width="98%" align="center">
</span></div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal" style=""><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:black">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:black"> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu">syslog-ng-bounces@lists.balabit.hu</a>>
on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <<a href="mailto:mark.faine@nasa.gov">mark.faine@nasa.gov</a>><br>
<b>Sent:</b> Monday, July 6, 2020 20:23<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> [syslog-ng] Convert logs back into syslog-ng logs</span><span style="">
</span></p>
<div>
<p class="x_MsoNormal"><span style=""> </span></p>
</div>
</div>
<div>
<div style="border:solid #9C6500 1.0pt; padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="x_MsoNormal" style="line-height:12.0pt; background:#FFEB9C"><b><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt; font-family:"Calibri",sans-serif; color:black"> This email
originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</span></p>
</div>
<p class="x_MsoNormal"><span style=""> </span></p>
<div>
<div>
<p class="x_xmsonormal">I would like to read lines from a file back into<span style="font-size:10.0pt"> log data that syslog-ng can parse. What would be involved in doing this?</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt"> </span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt">Thanks,</span></p>
<p class="x_xmsonormal"><span style="font-size:10.0pt">-Mark</span></p>
<p class="x_xmsonormal"> </p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>