<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Arash,<br>
<br>
You could consider a very simple design where the log retention is
kept as flat files on disk, with the "searchable" time period in
Elasticsearch (or whatever)<br>
<br>
As I see the advantages:<br>
- you can use cheap storage for the retention (since they will not
need to be regularly read back or searched)<br>
- since you are not adding anything to the data (as happens whenever
a log is "parsed" into CEF, json or whatever) this is the most
storage efficient form that still preserves all the information<br>
- flat files compress extremely well (8 - 10 times with standard
gzip)<br>
- syslog-ng can easily create files by date, system, whatever so log
retention, compression, rotation is trivial<br>
- if you need to search something outside your search time window,
the files are readily available to be re-indexed or whatever<br>
- if you need to share logs with anyone (e.g. a vendor for
troubleshooting, an investigation, etc.) gzipped text files are
pretty much universally acceptable<br>
- you don't even need to backup your search tool, since you can
always re-index if necessary<br>
- you can even add basic integrity checking by checksumming the
files (maybe even before and after compression)<br>
<br>
Then the search tool (elasticsearch for example) can store a shorter
time period (maybe 90 days) and roll off older indices.<br>
- makes the search tool faster (fewer logs to search)<br>
- simpler to implement (e.g. no backups required)<br>
- can use faster / more expensive storage since you need less of it<br>
<br>
I have found that even security investigations rarely need anything
beyond 90 days (not *never* but less than 5% of investigations in my
experience).<br>
<br>
This meets PCI (i.e. store a year at least with 3 months immediately
available online)<br>
<br>
<br>
<br>
Not saying you can't (or shouldn't) use the search tool for
retention - I just like to consider them separately (with
correlation / SIEM being a third piece) when designing a logging
& monitoring architecture)<br>
<br>
Just something to think about.<br>
<br>
Jim<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 09/15/2015 01:47 AM, Arash Shams
wrote:<br>
</div>
<blockquote cite="mid:BAY178-W150A2DDE2AECF06E14A038925C0@phx.gbl"
type="cite">
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style>
<div dir="ltr">Hello <br>
ELK solution is not suitable for us to collect all logs in one
server . i prefer to use syslog-ng to collect all logs in Mysql
database. <br>
is there possible to send all servers log to one syslog-ng
server and that collect them to database ? <br>
anyone can give me a good configuration file or some examples in
this case ? <br>
thank you <br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>