Hi Arash, You could consider a very simple design where the log retention is kept as flat files on disk, with the "searchable" time period in Elasticsearch (or whatever) As I see the advantages: - you can use cheap storage for the retention (since they will not need to be regularly read back or searched) - since you are not adding anything to the data (as happens whenever a log is "parsed" into CEF, json or whatever) this is the most storage efficient form that still preserves all the information - flat files compress extremely well (8 - 10 times with standard gzip) - syslog-ng can easily create files by date, system, whatever so log retention, compression, rotation is trivial - if you need to search something outside your search time window, the files are readily available to be re-indexed or whatever - if you need to share logs with anyone (e.g. a vendor for troubleshooting, an investigation, etc.) gzipped text files are pretty much universally acceptable - you don't even need to backup your search tool, since you can always re-index if necessary - you can even add basic integrity checking by checksumming the files (maybe even before and after compression) Then the search tool (elasticsearch for example) can store a shorter time period (maybe 90 days) and roll off older indices. - makes the search tool faster (fewer logs to search) - simpler to implement (e.g. no backups required) - can use faster / more expensive storage since you need less of it I have found that even security investigations rarely need anything beyond 90 days (not *never* but less than 5% of investigations in my experience). This meets PCI (i.e. store a year at least with 3 months immediately available online) Not saying you can't (or shouldn't) use the search tool for retention - I just like to consider them separately (with correlation / SIEM being a third piece) when designing a logging & monitoring architecture) Just something to think about. Jim On 09/15/2015 01:47 AM, Arash Shams wrote:
Hello ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database. is there possible to send all servers log to one syslog-ng server and that collect them to database ? anyone can give me a good configuration file or some examples in this case ? thank you
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq