<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div>Dear Sezer,<br>
</div>
<div> <br>
</div>
<div> I am in a little bit of trouble with your email. I can not clearly understand your goals, and don't know anything about your current environment. I will add some notes to your email, hoping that it will trigger some new ideas for you.<br>
</div>
<div><br>
</div>
<div>
<blockquote style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; color: rgb(102, 102, 102);">
<div style="margin: 0px; font-size: 15px; font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; color: rgb(32, 31, 30); background-color: rgb(255, 255, 255)">
Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below:</div>
<div style="margin: 0px; font-size: 15px; font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; color: rgb(32, 31, 30); background-color: rgb(255, 255, 255)">
<span style="margin: 0px; font-size: 12px; line-height: 1.42857; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(115, 115, 115)">${ISODATE} ${HOST} ${FACILITY} ${LEVEL} </span><span style="margin: 0px; font-size: 9pt; line-height: 1.42857; font-family: Menlo; color: rgb(115, 115, 115)">${LEVEL_NUM} </span><span style="margin: 0px; font-size: 12px; line-height: 1.42857; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(115, 115, 115)">${TAGS}
${MESSAGE}</span></div>
</blockquote>
</div>
<div>Syslog-ng is parsing the log messages in the first place. So rather than parsing them again, it is more convenient to use some structured format as an output (like json) if your intention is to forward those log messages for further processing.</div>
<div><br>
</div>
<blockquote style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; color: rgb(102, 102, 102);">
<div>
<div style="margin: 0px; font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; color: rgb(50, 49, 48)">
2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job?</div>
</div>
</blockquote>
<div>If you CAN'T change the output template: I found Python very good at text processing. But if you are more familiar with other programming languages, I would recommend to use those instead of learning a new one.<br>
</div>
<div>If you CAN change the output format: see my notes under question 3.</div>
<div><br>
</div>
<blockquote style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; color: rgb(102, 102, 102);">
<div><span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; background-color: rgb(255, 255, 255); display: inline !important">3) Do you
suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on?</span><br>
</div>
</blockquote>
<div>Parsing means, that you want to extract some information from the raw data you have.</div>
<div>A) There are common applications for this job, I recommend the "SIEM" search term as a starting point. Syslog-ng supports many of those out of the box.</div>
<div>B) If you want to do this processing by yourself: Using a DB is just an another "structured data format", like json or csv, etc.. I recommend to approach this problem from the processing end, not from the syslog-ng output. Try to define what do you want
to extract from the raw data, select the most convenient programming language / application for the task, it will determine the data format automatically.</div>
<div><br>
</div>
<div>Fun note: If you select approach B, and you are familiar with the Python language, you don't have to write those log messages into files at all. Syslog-ng has a "Python destination" (Péter wrote a blog post about it:
<a href="https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting-started">
https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting-started</a>), where your own code can process the log messages on the fly.</div>
<div><br>
</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Laci</div>
<div><br>
</div>
<div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0; margin-bottom:0"></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; margin:0cm 0cm 8pt; font-size:12.8px; line-height:15.6933px">
<font face="Verdana, sans-serif"><span style="font-size:12px">Laszlo Szemere<br>
</span></font><span style="font-family:Verdana,sans-serif; font-size:12px">Software Engineer</span></p>
<div style="color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small">
<b><span style="font-size:9pt; font-family:Arial,sans-serif; color:rgb(251,79,20)">Quest</span></b><span style="font-size:9pt; font-family:Arial,sans-serif; color:rgb(77,77,76)"> </span><span style="font-size:9pt; font-family:Arial,sans-serif; color:rgb(51,51,51)">| </span><span style="font-size:9pt; font-family:Arial,sans-serif"><font color="#6fa8dc">One
Identity</font></span><span style="font-size:9pt; font-family:Arial,sans-serif; color:rgb(77,77,76)"> </span><span style="font-size:9pt; font-family:Arial,sans-serif; color:rgb(51,51,51)">| Balabit</span></div>
<div style="color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small">
<font face="Times New Roman, serif"><span style="font-size:13.3333px"><font color="#1155cc"></font></span></font></div>
<a href="mailto:laszlo.szemere@oneidentity.com" class="OWAAutoLink"></a>
<div style="color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small">
<a href="mailto:laszlo.szemere@oneidentity.com" class="OWAAutoLink"><font face="Times New Roman, serif"><span style="font-size:13.3333px"><font color="#1155cc"></font></span></font></a><font face="Times New Roman, serif"><a href="mailto:laszlo.szemere@oneidentity.com" class="OWAAutoLink"><font color="#1155cc">laszlo.szemere@oneidentity.</font><wbr><font color="#1155cc">com</font></a><font color="#1155cc"></font></font></div>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:12.8px">
</p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(34,34,34); margin:0cm 0cm 8pt; line-height:15.6933px; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(34,34,34); margin:0cm 0cm 8pt; line-height:15.6933px; font-size:11pt; font-family:Calibri,sans-serif">
<b style="font-size:14.6667px"><span lang="EN-US" style="font-size:8pt; line-height:11.4133px; font-family:Verdana,sans-serif; color:rgb(5,170,219)">Address:</span></b><span lang="EN-US" style="font-size:8pt; line-height:11.4133px; font-family:Verdana,sans-serif"> 1117
Budapest, Alíz utca 2. (Balabit-Europe Kft.)</span></p>
<br>
<p style="margin-top: 0px; margin-bottom: 0px;"></p>
</div>
</div>
</div>
<div><br>
</div>
<div>________________________________________<br>
</div>
<div>From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Sezer BAGLAN <sezerbaglan@gmail.com><br>
</div>
<div>Sent: Wednesday, June 12, 2019 10:59<br>
</div>
<div>To: syslog-ng@lists.balabit.hu<br>
</div>
<div>Subject: [syslog-ng] An External Application For Parsing Existing Syslog-ng Log File<br>
</div>
<div><br>
</div>
<div>CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
</div>
<div><br>
</div>
<div>Dear all,<br>
</div>
<div><br>
</div>
<div>Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below:<br>
</div>
<div>${ISODATE} ${HOST} ${FACILITY} ${LEVEL} ${LEVEL_NUM} ${TAGS} ${MESSAGE}<br>
</div>
<div><br>
</div>
<div>1) What is your opinion about the best practices to handle this issue? Please note that I'm not talking about logging messages with filters and parsers. I'm talking about parsing an existing syslog-ng log file.<br>
</div>
<div><br>
</div>
<div>2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job?<br>
</div>
<div><br>
</div>
<div>3) Do you suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on?<br>
</div>
<div><br>
</div>
<div>Thanks in advance.<br>
</div>
<div><br>
</div>
<div>Kind regards,<br>
</div>
<div>Sezer<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</body>
</html>