[syslog-ng] Advice on the right destination

[Giovanni Mancuso]

Hi,
I am writing to ask your advice on a solution I'm thinking.

I have 12 servers with postfix, amavisd-new and other custom software
that manage the e-mail system and I was working in a web interface to
analysis the logs and correlation with the ability to search for certain
fields (from, to, message-id , date).

All applications send the logs to a centralized syslog-ng, and I was
trying to understand which type of "destination"  is better to use to
ensure the rapid search. I was analyzing the possibility of using
elasticsearch, but I don't know neither it or its performances.

The quantity of data is very high, about 3TB of data monthly or each
machine, with 2 years of retention.

What do you think about? Have you any suggestions?

Thanks
-- 
*Giovanni Mancuso*
System Architect
 

*T* 06.9826.9600 *M* +39.340.65.80.739 *F* 06.9826.9680
P.zza S.Benedetto da Norcia, 33 - 00071 Pomezia (RM)
Par-Tec S.p.A. <http://www.par-tec.it> Web Site <http://www.par-tec.it>
info at par-tec.it <mailto:info at par-tec.it> Pagina Facebook
<https://www.facebook.com/ParTecSpA> Profilo Twitter
<https://twitter.com/partecspa> Pagina LinkedIn
<https://www.linkedin.com/company/par-tec/> Canale YouTube
<https://www.youtube.com/user/ParTecSpA>
CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere
confidenziale per i destinatari in indirizzo.
È vietato l'inoltro non autorizzato a destinatari diversi da quelli
indicati nel messaggio originale.
Se ricevuto per errore, l'uso del contenuto è proibito; si prega di
comunicarlo al mittente e cancellarlo immediatamente.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo_partec.png
Type: image/png
Size: 1072 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sito.png
Type: image/png
Size: 511 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment-0001.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mail.png
Type: image/png
Size: 377 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fb.png
Type: image/png
Size: 280 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: twitter.png
Type: image/png
Size: 401 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: in.png
Type: image/png
Size: 325 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: yt.png
Type: image/png
Size: 558 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment-0006.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: giovanni_mancuso.vcf
Type: text/x-vcard
Size: 319 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150819/3e8da709/attachment.vcf