Well, if you want to search in that amount of data, you will have to pick a search engine. Elastic seems to be the favourite these days with kibana as its frontend, but these numbers are pretty high. Elastic scales horizontally but I don't know how it performs per node. I would guess it does a few times 10k per node per second. 3tb for each machine per day is 36tb per day? If I assume 300 bytes per message, that becomes 10billion messages per day per node. Splunk is a commercial search engine for logs, but they recommend a node every 100GB of data stored. Thats probably only true if you want to go back searching for a long while and becomes more efficient if you get to data warehousing and lose the ability to do fast searches on older data. But even then, the size of the cluster and the license for splunk becomes prohibitive (licensing on the gigabyte). BalaBit also has an indexing engine that has very good per node performance, but as of now it doesn't scale horizontally. It is able to cope with about 8-9billion messages per day on a single node, but you would need a dedicated box for each of your nodes. I can connect you to our internal engineering team, so they can understand your problem and see if we can see a solution to that. What do you think? On Aug 19, 2015 17:49, "Giovanni Mancuso" <giovanni.mancuso@par-tec.it> wrote:
Hi, I am writing to ask your advice on a solution I'm thinking.
I have 12 servers with postfix, amavisd-new and other custom software that manage the e-mail system and I was working in a web interface to analysis the logs and correlation with the ability to search for certain fields (from, to, message-id , date).
All applications send the logs to a centralized syslog-ng, and I was trying to understand which type of "destination" is better to use to ensure the rapid search. I was analyzing the possibility of using elasticsearch, but I don't know neither it or its performances.
The quantity of data is very high, about 3TB of data monthly or each machine, with 2 years of retention.
What do you think about? Have you any suggestions?
Thanks -- *Giovanni Mancuso* System Architect *T* 06.9826.9600 *M* +39.340.65.80.739 *F* 06.9826.9680 P.zza S.Benedetto da Norcia, 33 - 00071 Pomezia (RM) [image: Par-Tec S.p.A.] <http://www.par-tec.it> [image: Web Site] <http://www.par-tec.it> [image: info@par-tec.it] <info@par-tec.it> [image: Pagina Facebook] <https://www.facebook.com/ParTecSpA> [image: Profilo Twitter] <https://twitter.com/partecspa> [image: Pagina LinkedIn] <https://www.linkedin.com/company/par-tec/> [image: Canale YouTube] <https://www.youtube.com/user/ParTecSpA> CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale per i destinatari in indirizzo. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore, l'uso del contenuto è proibito; si prega di comunicarlo al mittente e cancellarlo immediatamente.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
