[syslog-ng] Recommended Data base

[Jim Hendrick]

Hmmm - I have done some testing but not settled on the perfect solution
(yet).

- I have multi-line logs being parsed by a program destination and then
writing to mongodb (perl parser)
- I have started testing straight mongo, but getting the right data in
the right fields seems important
- I have done very basic testing with syslog-ng --> redis which I am
planning on then --> elasticsearch
   (I may need to stick Logstash in between redis and elasticsearch)

In general, any relational database (SQL) wins when you can represent
the relationships in the schema.

"nosql" like mongo typically wins with unstructured data, but at a space
penalty (needing to store json format)

If I had to pick one *right now* I would probably use syslog-ng -->
redis --> logstash --> elasticsearch --> kibana
The (R) ELK stack has a lot of support and development, and is pretty
close to a free splunk.

Although I can see using a sharded/replicated mongodb having some basic
advantages, but I have not (yet) found the perfect way to do ad-hoc
queries against the store.

Good luck (and report back!)

Thanks,
Jim


On 08/18/2014 05:06 PM, VMI X wrote:
> Hi,
> Currently we're logging everything to text files for a few LAN clients.
> We're considering using a database instead and have a few questions to
> help us decide:
>
>   * Would a database be a good option to replace existing text files
>     for long term storage considering storage space?
>   * Would mongo OR mysql be better suited for storing system logs? 
>
> I understand answers to these questions can vary depending on specific
> use case but seeking a general recommendation to see what's typically
> being used and what the most stable/supported options would be.
>
>
> -- 
> /Nullius In Verba/
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140818/1bdee678/attachment.htm