[syslog-ng] Advice on the right destination

[Balazs Scheidler]

Well, if you want to search in that amount of data, you will have to pick a
search engine.

Elastic seems to be the favourite these days with kibana as its frontend,
but these numbers are pretty high. Elastic scales horizontally but I don't
know how it performs per node. I would guess it does a few times 10k per
node per second.

3tb for each machine per day is 36tb per day? If I assume 300 bytes per
message, that becomes 10billion messages per day per node.

Splunk is a commercial search engine for logs, but they recommend a node
every 100GB of data stored. Thats probably only true if you want to go back
searching for a long while and becomes more efficient if you get to data
warehousing and lose the ability to do fast searches on older data. But
even then, the size of the cluster and the license for splunk becomes
prohibitive (licensing on the gigabyte).

BalaBit also has an indexing engine that has very good per node
performance, but as of now it doesn't scale horizontally. It is able to
cope with about 8-9billion messages per day on a single node, but you would
need a dedicated box for each of your nodes. I can connect you to our
internal engineering team, so they can understand your problem and see if
we can see a solution to that. What do you think?
On Aug 19, 2015 17:49, "Giovanni Mancuso" <giovanni.mancuso at par-tec.it>
wrote:

> Hi,
> I am writing to ask your advice on a solution I'm thinking.
>
> I have 12 servers with postfix, amavisd-new and other custom software that
> manage the e-mail system and I was working in a web interface to analysis
> the logs and correlation with the ability to search for certain fields
> (from, to, message-id , date).
>
> All applications send the logs to a centralized syslog-ng, and I was
> trying to understand which type of "destination"  is better to use to
> ensure the rapid search. I was analyzing the possibility of using
> elasticsearch, but I don't know neither it or its performances.
>
> The quantity of data is very high, about 3TB of data monthly or each
> machine, with 2 years of retention.
>
> What do you think about? Have you any suggestions?
>
> Thanks
> --
> *Giovanni Mancuso* System Architect   *T* 06.9826.9600 *M*
> +39.340.65.80.739 *F* 06.9826.9680 P.zza S.Benedetto da Norcia, 33 -
> 00071 Pomezia (RM) [image: Par-Tec S.p.A.] <http://www.par-tec.it> [image:
> Web Site] <http://www.par-tec.it> [image: info at par-tec.it]
> <info at par-tec.it> [image: Pagina Facebook]
> <https://www.facebook.com/ParTecSpA> [image: Profilo Twitter]
> <https://twitter.com/partecspa> [image: Pagina LinkedIn]
> <https://www.linkedin.com/company/par-tec/> [image: Canale YouTube]
> <https://www.youtube.com/user/ParTecSpA> CONFIDENZIALE: Questo messaggio
> ed i suoi allegati sono di carattere confidenziale per i destinatari in
> indirizzo.
> È vietato l'inoltro non autorizzato a destinatari diversi da quelli
> indicati nel messaggio originale.
> Se ricevuto per errore, l'uso del contenuto è proibito; si prega di
> comunicarlo al mittente e cancellarlo immediatamente.
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fb.png
Type: image/png
Size: 280 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: twitter.png
Type: image/png
Size: 401 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment-0001.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sito.png
Type: image/png
Size: 511 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo_partec.png
Type: image/png
Size: 1072 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mail.png
Type: image/png
Size: 377 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: in.png
Type: image/png
Size: 325 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: yt.png
Type: image/png
Size: 558 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150831/64f91678/attachment-0006.png