Hi Arash,

  You could consider a very simple design where the log retention is kept as flat files on disk, with the "searchable" time period in Elasticsearch (or whatever)

As I see the advantages:
- you can use cheap storage for the retention (since they will not need to be regularly read back or searched)
- since you are not adding anything to the data (as happens whenever a log is "parsed" into CEF, json or whatever) this is the most storage efficient form that still preserves all the information
- flat files compress extremely well (8 - 10 times with standard gzip)
- syslog-ng can easily create files by date, system, whatever so log retention, compression, rotation is trivial
- if you need to search something outside your search time window, the files are readily available to be re-indexed or whatever
- if you need to share logs with anyone (e.g. a vendor for troubleshooting, an investigation, etc.) gzipped text files are pretty much universally acceptable
- you don't even need to backup your search tool, since you can always re-index if necessary
- you can even add basic integrity checking by checksumming the files (maybe even before and after compression)

Then the search tool (elasticsearch for example) can store a shorter time period (maybe 90 days) and roll off older indices.
- makes the search tool faster (fewer logs to search)
- simpler to implement (e.g. no backups required)
- can use faster / more expensive storage since you need less of it

I have found that even security investigations rarely need anything beyond 90 days (not *never* but less than 5% of investigations in my experience).

This meets PCI (i.e. store a year at least with 3 months immediately available online)



Not saying you can't (or shouldn't) use the search tool for retention - I just like to consider them separately (with correlation / SIEM being a third piece) when designing a logging & monitoring architecture)

Just something to think about.

Jim



On 09/15/2015 01:47 AM, Arash Shams wrote:
Hello
ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database.
is there possible to send all servers log to one syslog-ng server and that collect them to database ?
anyone can give me a good configuration file or some examples in this case ?
thank you 


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq