That was going to be my next question! ;] So what do you use to parse all that data? I'm using Logmuncher which has proven to be quite flexible and meets my needs. Russell On Tue, Nov 30, 2004 at 12:21:26PM -0800, Bill Nash wrote:
[syslog]$ du 15289740 ./11-29
I'm chucking 15 gigs of syslog per day, to the tune of: [11-29]$ cat * | nice wc -l 40784743
On top of that, the entirety of it is thrown to the mercy of a perl based log analyzer (single threaded, no less), which in turn filters and logs to a db, at an average rate of 472 lines per second.
Cheers. =)
- billn
On Tue, 30 Nov 2004, Jay Guerette wrote:
( I hope I don't offend anyone with an attachment, it's only 8k. )
My largest single-server syslog-ng implementation currently handles over 13M lines per day, totalling about 1.8Gb per day. I've only recently been able to gather this data by creating a process to count incoming lines, sum their lengths, and report via syslog at 1 minute intervals. See attached graph ( if the attachment survived ).
I added this configuration before all other entries to make sure it sees everything:
<syslog-ng.conf> destination syslog-perf { program(syslog-perf); }; log { source(syslog); destination(syslog-perf); }; </syslog-ng.conf>
I originally tried this in Perl, then Bash, but neither could keep up with the incoming messages. This works like a champ. It compiles on Linux. The output format is specific to my syslog-to-rrd implementation, but you get the idea. It is suitable for an installation that is assured of at least 1 message for each reporting interval!
<syslog-perf.c> #include <stdio.h> #include <time.h> #include <syslog.h>
#define BUFFER_SIZE 8192 #define REPORT_INTERVAL 60
void main(void) {
char buf[BUFFER_SIZE]; long count, bytes; time_t lastupdate;
lastupdate = time(NULL); while (fgets(buf, BUFFER_SIZE, stdin)) { count++; bytes += (strlen(buf) - 1); if (time(NULL) > (lastupdate + REPORT_INTERVAL)) { openlog("127.0.0.1", LOG_NDELAY, LOG_LOCAL3); syslog(LOG_INFO, "Syslog-ng\\Lines=%d Syslog-ng\\Bytes=%d", count, bytes); closelog(); lastupdate += REPORT_INTERVAL; count = 0; bytes = 0; } }
} </syslog-perf.c>
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html