[syslog-ng] pulling in "sar" data

[Fabien Wernli]

Hi Jim,

On Sat, Dec 30, 2017 at 11:27:49AM -0500, Jim Hendrick wrote:
>   Has anyone added sar data to syslog-ng? I know of a couple "brute-force"
> ways like running a cron to run the sar command and dump it to a file. But
> before I do that I wanted to check and see if there was a more elegant
> solution.
> 
>  The overall problem would be to run this across 10,000+ servers with
> minimal need to change configurations. (for example - if there was a need
> to change the parameters to sar, or change the frequency of the pull).
> 
>  I guess I was hoping for some add-on that either directly pulled the
> performance data from the kernel or could include a config that would
> specify what data elements would be included.
> 
> FYI - we do use syslog-ng PE but this seems fairly generic so I thought I'd
> ask the list.
> 
>  Given the scale of the problem, pulling "all" data even relatively
> frequently (say every 1-5 minutes) would result in a huge volume increase
> in our logging solution (where we pay by the ingested GB...)
> 
> Thoughts? Advice?

For this kind of data (system/app metrics), which tends to use astronomical
amounts of disk space unprocessed, we use collectd, then
pre-aggregate the data (min/max/avg) then push it to Elasticsearch. Query is
done either using REST or Grafana.

That being said, collection could be done in syslog-ng, much like the pacct
driver is reading from the binary file [1].

This could actually be an interesting idea for GSoC :-)

Cheers

--
[1] https://www.balabit.com/documents/syslog-ng-ose-3.13-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#configuring-source-pacct