[syslog-ng]FYI: Performance

[Russell Adams]

Since my logs are arranged via:

/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY

Scripting a report only took a few minutes to total daily usage, offline.

My numbers are pale in comparison.

I get some 71k lines per day, totaling 12MB/day.

I grovel in your mighty shadow. ;]

How many hosts are you logging?

Russell

On Tue, Nov 30, 2004 at 11:27:11AM -0500, Jay Guerette wrote:
> ( I hope I don't offend anyone with an attachment, it's only 8k. )
> 
> My largest single-server syslog-ng implementation currently handles
> over 13M lines per day, totalling about 1.8Gb per day. I've only
> recently been able to gather this data by creating a process to count
> incoming lines, sum their lengths, and report via syslog at 1 minute
> intervals. See attached graph ( if the attachment survived ).
> 
> I added this configuration before all other entries to make sure it
> sees everything:
> 
> <syslog-ng.conf>
> destination syslog-perf {
>         program(syslog-perf);
> };
> log {
>         source(syslog);
>         destination(syslog-perf);
> };
> </syslog-ng.conf>
> 
> I originally tried this in Perl, then Bash, but neither could keep up
> with the incoming messages. This works like a champ. It compiles on
> Linux. The output format is specific to my syslog-to-rrd
> implementation, but you get the idea. It is suitable for an
> installation that is assured of at least 1 message for each reporting
> interval!
> 
> <syslog-perf.c>
> #include <stdio.h>
> #include <time.h>
> #include <syslog.h>
> 
> #define BUFFER_SIZE 8192
> #define REPORT_INTERVAL 60
> 
> void main(void) {
> 
> char buf[BUFFER_SIZE];
> long count, bytes;
> time_t lastupdate;
> 
> lastupdate = time(NULL);
> while (fgets(buf, BUFFER_SIZE, stdin)) {
>    count++;
>    bytes += (strlen(buf) - 1);
>    if (time(NULL) > (lastupdate + REPORT_INTERVAL)) {
>       openlog("127.0.0.1", LOG_NDELAY, LOG_LOCAL3);
>       syslog(LOG_INFO, "Syslog-ng\\Lines=%d Syslog-ng\\Bytes=%d", count, bytes);
>       closelog();
>       lastupdate += REPORT_INTERVAL;
>       count = 0;
>       bytes = 0;
>    }
> }
> 
> }
> </syslog-perf.c>
>