[syslog-ng] Syslog ng and Mysql

[Jim Hendrick]

Hi Arash,

   You could consider a very simple design where the log retention is 
kept as flat files on disk, with the "searchable" time period in 
Elasticsearch (or whatever)

As I see the advantages:
- you can use cheap storage for the retention (since they will not need 
to be regularly read back or searched)
- since you are not adding anything to the data (as happens whenever a 
log is "parsed" into CEF, json or whatever) this is the most storage 
efficient form that still preserves all the information
- flat files compress extremely well (8 - 10 times with standard gzip)
- syslog-ng can easily create files by date, system, whatever so log 
retention, compression, rotation is trivial
- if you need to search something outside your search time window, the 
files are readily available to be re-indexed or whatever
- if you need to share logs with anyone (e.g. a vendor for 
troubleshooting, an investigation, etc.) gzipped text files are pretty 
much universally acceptable
- you don't even need to backup your search tool, since you can always 
re-index if necessary
- you can even add basic integrity checking by checksumming the files 
(maybe even before and after compression)

Then the search tool (elasticsearch for example) can store a shorter 
time period (maybe 90 days) and roll off older indices.
- makes the search tool faster (fewer logs to search)
- simpler to implement (e.g. no backups required)
- can use faster / more expensive storage since you need less of it

I have found that even security investigations rarely need anything 
beyond 90 days (not *never* but less than 5% of investigations in my 
experience).

This meets PCI (i.e. store a year at least with 3 months immediately 
available online)



Not saying you can't (or shouldn't) use the search tool for retention - 
I just like to consider them separately (with correlation / SIEM being a 
third piece) when designing a logging & monitoring architecture)

Just something to think about.

Jim



On 09/15/2015 01:47 AM, Arash Shams wrote:
> Hello
> ELK solution is not suitable for us to collect all logs in one server 
> . i prefer to use syslog-ng to collect all logs in Mysql database.
> is there possible to send all servers log to one syslog-ng server and 
> that collect them to database ?
> anyone can give me a good configuration file or some examples in this 
> case ?
> thank you
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150918/13a1995a/attachment.htm