<p dir="ltr">Well, if you want to search in that amount of data, you will have to pick a search engine. </p>
<p dir="ltr">Elastic seems to be the favourite these days with kibana as its frontend, but these numbers are pretty high. Elastic scales horizontally but I don't know how it performs per node. I would guess it does a few times 10k per node per second. </p>
<p dir="ltr">3tb for each machine per day is 36tb per day? If I assume 300 bytes per message, that becomes 10billion messages per day per node.</p>
<p dir="ltr">Splunk is a commercial search engine for logs, but they recommend a node every 100GB of data stored. Thats probably only true if you want to go back searching for a long while and becomes more efficient if you get to data warehousing and lose the ability to do fast searches on older data. But even then, the size of the cluster and the license for splunk becomes prohibitive (licensing on the gigabyte).</p>
<p dir="ltr">BalaBit also has an indexing engine that has very good per node performance, but as of now it doesn't scale horizontally. It is able to cope with about 8-9billion messages per day on a single node, but you would need a dedicated box for each of your nodes. I can connect you to our internal engineering team, so they can understand your problem and see if we can see a solution to that. What do you think?</p>
<div class="gmail_quote">On Aug 19, 2015 17:49, "Giovanni Mancuso" <<a href="mailto:giovanni.mancuso@par-tec.it">giovanni.mancuso@par-tec.it</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi, <br>
I am writing to ask your advice on a solution I'm thinking.<br>
<br>
I have 12 servers with postfix, amavisd-new and other custom
software that manage the e-mail system and I was working in a web
interface to analysis the logs and correlation with the ability to
search for certain fields (from, to, message-id , date).<br>
<br>
All applications send the logs to a centralized syslog-ng, and I was
trying to understand which type of "destination" is better to use
to ensure the rapid search. I was analyzing the possibility of using
elasticsearch, but I don't know neither it or its performances.<br>
<br>
The quantity of data is very high, about 3TB of data monthly or each
machine, with 2 years of retention.<br>
<br>
What do you think about? Have you any suggestions?<br>
<br>
Thanks<br>
<div>-- <br>
<table width="600" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td style="font-family:Verdana;font-size:14px;color:#ff661b"><strong>Giovanni Mancuso</strong></td>
</tr>
<tr>
<td style="font-family:Verdana;font-size:12px;color:#000;padding-bottom:8px">System Architect</td>
</tr>
<tr>
<td>
<table width="355" cellpadding="0" cellspacing="0" height="5">
<tbody>
<tr>
<td style="background-color:#ff661b;height:5px;font-size:1px;min-width:355px;width:355px;max-height:5px;display:block;line-height:1px"> </td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="font-family:Verdana;font-size:12px;color:#000;padding-top:8px"><strong>T</strong> 06.9826.9600 <strong>M</strong>
+39.340.65.80.739 <strong>F</strong> 06.9826.9680</td>
</tr>
<tr>
<td style="font-family:Verdana;font-size:12px;color:#000">P.zza
S.Benedetto da Norcia, 33 - 00071 Pomezia (RM)</td>
</tr>
<tr>
<td style="font-family:Verdana;font-size:9px"><a href="http://www.par-tec.it" alt="" target="_blank"> <img src="cid:part1.06000302.07040808@par-tec.it" alt="Par-Tec S.p.A." title="Par-Tec S.p.A." style="padding:0;margin:0" border="0"></a> <a href="http://www.par-tec.it" alt="" target="_blank"> <img src="cid:part3.04010603.04020300@par-tec.it" alt="Web
Site" title="Web Site" style="padding:0;margin:0" border="0"></a> <a href="mailto:info@par-tec.it" target="_blank"> <img src="cid:part5.03050609.03090806@par-tec.it" alt="info@par-tec.it" title="info@par-tec.it" style="padding:0;margin:0" border="0"></a> <a href="https://www.facebook.com/ParTecSpA" target="_blank"> <img src="cid:part7.01020204.09020107@par-tec.it" alt="Pagina Facebook" title="Pagina Facebook" style="padding:0;margin:0" border="0"></a> <a href="https://twitter.com/partecspa" target="_blank"> <img src="cid:part9.04030302.07060000@par-tec.it" alt="Profilo Twitter" title="Profilo Twitter" style="padding:0;margin:0" border="0"></a> <a href="https://www.linkedin.com/company/par-tec/" target="_blank"> <img src="cid:part11.05030503.04060304@par-tec.it" alt="Pagina LinkedIn" title="Pagina LinkedIn" style="padding:0;margin:0" border="0"></a> <a href="https://www.youtube.com/user/ParTecSpA" target="_blank"> <img src="cid:part13.05070707.04080808@par-tec.it" alt="Canale YouTube" title="Canale YouTube" style="padding:0;margin:0" border="0"></a> </td>
</tr>
<tr>
<td style="font-family:Verdana;font-size:9px;color:#999">CONFIDENZIALE:
Questo messaggio ed i suoi allegati sono di carattere
confidenziale per i destinatari in indirizzo.<br>
È vietato l'inoltro non autorizzato a destinatari diversi
da quelli indicati nel messaggio originale.<br>
Se ricevuto per errore, l'uso del contenuto è proibito; si
prega di comunicarlo al mittente e cancellarlo
immediatamente.<br>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>