<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><pre>You will need to tell syslog-ng to encode this value as an integer.
pair("datetime" int64("${S_UNIXTIME}${S_MSEC}"))
Please note the int64 type hint</pre></blockquote>
</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763">I tried it and it also didn't work. </div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763">I figured out the solution by using the datetime function, apparently when working with MongoDB time-series or TTL, it is mandatory that the datetime pair value must be sent of type datetime not a string, int or other types otherwise MongoDB won't accept it. Here is the pair structure worked for me:</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763">pair("createdAt" datetime($S_UNIXTIME))</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763">Thanks.<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:#073763"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 6 Aug 2023 at 21:51, Ibrahim Al Mahfooz <<a href="mailto:ibrahim.nezar@sevennet.net">ibrahim.nezar@sevennet.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)" class="gmail_default">
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">Hello Evan,</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">Thanks for your response, <br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><pre>If you change your datetime to be compsed of the epoc and the
milliseconds it should work.
pair("datetime" "${S_UNIXTIME}${S_MSEC}")</pre></blockquote><div>I made
the change but didn't work, got same log error, also you can see the
datetime is UNIX+MSEC is reflected in the datetime pair but <br></div>
</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">[2023-08-06T21:38:53.717614] Outgoing message to MongoDB destination; message='datetime=1691347133717
host=10.36.0.6 portaction=initial portstatus=assigned
privateip=10.34.102.175 publicip=98.75.22.22 publicportrange=9048~10047
', driver='d_mongodb_ttl#0'<br>[2023-08-06T21:38:53.718564] Failed to
insert into MongoDB; time_reopen='10', reason='\'datetime\' must be
present and contain a valid BSON UTC datetime value',
driver='d_mongodb_ttl#0'</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">From
what I understand from MongoDB, BSON format is mandatory, and MongoDB
expectation is this binary format only, if syslog-ng is not capable of
sending such format, most probably things will not work out.</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">Do you have a working MongoDB Time-Series setup with syslog-ng before?</div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">Best regards</div>
</div><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 6 Aug 2023 at 18:04, Ibrahim Al Mahfooz <<a href="mailto:ibrahim.nezar@sevennet.net" target="_blank">ibrahim.nezar@sevennet.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">Hello,<br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">I have integration working fine between syslog-ng v3.38 and MongoDB v6. I am working to enable MongoDB Time-Series Collections on a testing DB for the purpose of having better efficiency and improved queries over time. <br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">I created the DB and collection, tested it by Mongo Compass client, then I tried to send the data from syslog-ng but it didn't work. MongoDB is throwing the following error: <br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">"17:12:57. 483 | [2023-08-06T17:12:57.355549] Failed to insert into MongoDB; time_reopen='10', reason='\'datetime\' must be present and contain a valid BSON UTC datetime value', driver='d_mongodb_ttl#0" <br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">
<div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">My
understanding is that it is a must to send
the time/date to MongoDB with Time Series enabled DB a BSON format Date. Any suggestions?<br></div>
</div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><u>Below is the mongoDB destination config:</u></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">destination d_mongodb_ttl {<br> mongodb(<br> uri("mongodb://<a href="http://192.168.1.19:27017/syslog-ng_ttl" target="_blank">192.168.1.19:27017/syslog-ng_ttl</a>")<br> collection("logs")<br> value-pairs(<br> scope("selected-macros")<br> exclude("SOURCEIP")<br> exclude("TAGS")<br> pair("datetime" "${S_ISODATE}")<br> pair("privateip" "${privateip}")<br> pair("publicip" "${publicip}")<br> pair("publicportrange" "${publicportrange}")<br> pair("portaction" "${AI}")<br> pair("portstatus" "${AL}")<br> pair("host" "${HOST}"))<br> );<br>};</div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)">Appreciate your help<br></div><div style="font-family:tahoma,sans-serif;color:rgb(7,55,99)"><br></div><br></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">
<div>
<div>
<table style="font-size:small;font-family:Verdana;width:420px" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<table style="font-size:small;font-family:Verdana;width:100%" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<h3 color="#2f2f30" style="margin:0px;font-size:16px;color:rgb(228,31,47)"><span>Ibrahim</span><span> </span><span>Abdulatif</span></h3>
<p color="#2f2f30" style="margin:0px;color:rgb(147,148,152);font-size:12px;line-height:20px"><span>IT and EPC Core Signaling Manager </span></p>
<p color="#2f2f30" style="margin:0px;font-weight:500;color:rgb(47,47,48);font-size:12px;line-height:20px"><span>Sevennet Layers Co., LTD.</span></p>
</td>
<td style="text-align:right"><span style="display:inline-block;text-align:right;width:100%"><img src="https://www.sevennet.net/wp-content/uploads/2021/08/sevennetLogoPartOf-01.png" style="max-width: 230px; display: inline; text-align: right;" width="230"></span></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td height="10"><br></td>
</tr>
<tr>
<td color="#ce0e2d" style="width:100%;border-bottom:1px solid rgb(206,14,45);border-left-style:none;display:block" height="1"></td>
</tr>
<tr>
<td height="10"></td>
</tr>
<tr>
<td>
<table style="font-size:small;font-family:Verdana" cellspacing="0" cellpadding="0">
<tbody>
<tr style="vertical-align:middle" height="25">
<td style="vertical-align:middle" width="30">
<table style="font-size:small;font-family:Verdana" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="vertical-align:bottom">
<span color="#ce0e2d" width="11" style="display:block;background-color:rgb(206,14,45)">
<img src="https://cdn2.hubspot.net/hubfs/53/tools/email-signature-generator/icons/phone-icon-2x.png" color="#ce0e2d" style="display: block; background-color: rgb(206, 14, 45);" width="13">
</span>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0px;color:rgb(47,47,48)">
<a href="tel:+964+(0)+53+511+5514" color="#2f2f30" style="text-decoration:none;color:rgb(47,47,48);font-size:12px" target="_blank">
<span>+964 (0) 53 511 5514</span>
</a>
<span>|</span>
<a href="tel:+964+(0)+770+471+5514" color="#2f2f30" style="text-decoration:none;color:rgb(47,47,48);font-size:12px" target="_blank"><span>+964 (0) 770 471 5514</span></a>
</td>
</tr>
<tr style="vertical-align:middle" height="25">
<td style="vertical-align:middle" width="30">
<table style="font-size:small;font-family:Verdana" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="vertical-align:bottom"><span color="#ce0e2d" width="11" style="display:block;background-color:rgb(206,14,45)"><img src="https://cdn2.hubspot.net/hubfs/53/tools/email-signature-generator/icons/link-icon-2x.png" color="#ce0e2d" style="display: block; background-color: rgb(206, 14, 45);" width="13"></span></td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0px">
<a href="https://www.sevennet.net" color="#2f2f30" style="text-decoration:none;color:rgb(47,47,48);font-size:12px" target="_blank"><span>www.sevennet.net</span></a>
</td>
</tr>
<tr style="vertical-align:middle" height="25">
<td style="vertical-align:middle" width="30">
<table style="font-size:small;font-family:Verdana" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="vertical-align:bottom">
<span color="#ce0e2d" width="11" style="display:block;background-color:rgb(206,14,45)">
<img src="https://cdn2.hubspot.net/hubfs/53/tools/email-signature-generator/icons/address-icon-2x.png" color="#ce0e2d" style="display: block; background-color: rgb(206, 14, 45);" width="13">
</span>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0px">
<span color="#2f2f30" style="font-size:12px;color:rgb(47,47,48)">
<span>2nd Floor, Faruk Investment Group, Salim Street, Sulaymaniyah, Iraq</span>
</span>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div></div>