[syslog-ng]incoming message stats (fwd)

[Jay Guerette]

Hmmm.... these are pieces of a monitoring system; taking ~8000
performance datapoints per minute from ~200 servers and feeding them
into RRD.

I spool incoming performance data via syslog-ng to a file like this:
05:19:46 10.100.0.29: Processor(Total)\%_Processor_Time=3D1.5157
Paging_File(Total)\%_Usage=3D0.4660

That is the format that you'll notice the C program creates:
syslog(LOG_INFO, "Syslog-ng\\Lines=3D%d Syslog-ng\\Bytes=3D%d", count, byte=
s);

I wrote a Perl daemon/parser  that sits on top of that log and creates
new RRDs or updates exisiting RRDs. So that would translate into
creating or updating:
/rrd/10.100.0.29/Processor(Total)/%_Processor_Time.rrd
/rrd/10.100.0.29/Paging_File(Total)/%_Usage.rrd

This is probably all overkill for what you want; just a couple of
counters. If interested, I could sanitize/comment the code and make it
available.

Something that WORKS, but I have not implemented in production
systems, is the following:
Create a syslog-ng destination like this:

destination rrdtool {
        program("/path/to/rrdtool/bin/rrdtool -"
                template("$MESSAGE\n")
                template_escape(no)
        );
};

This puts rrdtool in listen mode on it's STDIN. Then you syslog log a
message to the rrdtool destination. This message must look like this:

update /path/to/database.rrd N:x

This will tell RRDtool to update the named database with a record
stamped 'N'ow, with the value of 'x'. The database must already exist;
part of the reason I don't use it, my production tool creates RRDs on
the fly for new monitors.

The trick is to make sure that EXACT message format gets to rrdtool.
Some syslog tools, like 'logger' for Linux will throw extra stuff in:

logger update /tmp/test.rrd N:1

resulted in:

root: update /tmp/test.rrd N:1

Which rrdtool ignored.

You would most likely need to create the UDP packet yourself, instead
of relying on an existing syslog tool. I tested it like this:

echo "<14> update /tmp/test.rrd N:1" | nc -w 1 -u 127.0.0.1 514

'<14>' is the syslog code for 'user.info'. I piped the output of echo
to 'nc'; which is a program called netcat (great tool!). netcat just
spit the line to the local syslog-ng via UDP. This worked, the output
was clean and rrdtool updated my test database.

The new 1.2 version of rrdtool can act as a TCP server.
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/doc/rrdtool.en.html#remo=
te_control

Which gives even more options...

But that's enough to chew on for now. Something here will work for you
without too much effort.

If you're not a C programmer of any sort, and want slight
modifications to the code I posted, I'd most likely be happy to help.
Just ask.


On 4/30/05, Nate Campi <nate@campin.net> wrote:
> The C code simply does the work of keeping tabs on the message count and
> sending a syslog message with the stats every 60 seconds. It's up to you
> to somehow get that information out of the logs and update an RRD, and
> then graph that information.
>=20
> All you'd need to get started is here:
> http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/tut/rrdtutorial.en.htm=
l
>=20
> Maybe somebody will send out a detailed HOWTO with scripts (Jay?) but
> for now you're on your own.