( I hope I don't offend anyone with an attachment, it's only 8k. ) My largest single-server syslog-ng implementation currently handles over 13M lines per day, totalling about 1.8Gb per day. I've only recently been able to gather this data by creating a process to count incoming lines, sum their lengths, and report via syslog at 1 minute intervals. See attached graph ( if the attachment survived ). I added this configuration before all other entries to make sure it sees everything: <syslog-ng.conf> destination syslog-perf { program(syslog-perf); }; log { source(syslog); destination(syslog-perf); }; </syslog-ng.conf> I originally tried this in Perl, then Bash, but neither could keep up with the incoming messages. This works like a champ. It compiles on Linux. The output format is specific to my syslog-to-rrd implementation, but you get the idea. It is suitable for an installation that is assured of at least 1 message for each reporting interval! <syslog-perf.c> #include <stdio.h> #include <time.h> #include <syslog.h> #define BUFFER_SIZE 8192 #define REPORT_INTERVAL 60 void main(void) { char buf[BUFFER_SIZE]; long count, bytes; time_t lastupdate; lastupdate = time(NULL); while (fgets(buf, BUFFER_SIZE, stdin)) { count++; bytes += (strlen(buf) - 1); if (time(NULL) > (lastupdate + REPORT_INTERVAL)) { openlog("127.0.0.1", LOG_NDELAY, LOG_LOCAL3); syslog(LOG_INFO, "Syslog-ng\\Lines=%d Syslog-ng\\Bytes=%d", count, bytes); closelog(); lastupdate += REPORT_INTERVAL; count = 0; bytes = 0; } } } </syslog-perf.c>
Since my logs are arranged via: /var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY Scripting a report only took a few minutes to total daily usage, offline. My numbers are pale in comparison. I get some 71k lines per day, totaling 12MB/day. I grovel in your mighty shadow. ;] How many hosts are you logging? Russell On Tue, Nov 30, 2004 at 11:27:11AM -0500, Jay Guerette wrote:
( I hope I don't offend anyone with an attachment, it's only 8k. )
My largest single-server syslog-ng implementation currently handles over 13M lines per day, totalling about 1.8Gb per day. I've only recently been able to gather this data by creating a process to count incoming lines, sum their lengths, and report via syslog at 1 minute intervals. See attached graph ( if the attachment survived ).
I added this configuration before all other entries to make sure it sees everything:
<syslog-ng.conf> destination syslog-perf { program(syslog-perf); }; log { source(syslog); destination(syslog-perf); }; </syslog-ng.conf>
I originally tried this in Perl, then Bash, but neither could keep up with the incoming messages. This works like a champ. It compiles on Linux. The output format is specific to my syslog-to-rrd implementation, but you get the idea. It is suitable for an installation that is assured of at least 1 message for each reporting interval!
<syslog-perf.c> #include <stdio.h> #include <time.h> #include <syslog.h>
#define BUFFER_SIZE 8192 #define REPORT_INTERVAL 60
void main(void) {
char buf[BUFFER_SIZE]; long count, bytes; time_t lastupdate;
lastupdate = time(NULL); while (fgets(buf, BUFFER_SIZE, stdin)) { count++; bytes += (strlen(buf) - 1); if (time(NULL) > (lastupdate + REPORT_INTERVAL)) { openlog("127.0.0.1", LOG_NDELAY, LOG_LOCAL3); syslog(LOG_INFO, "Syslog-ng\\Lines=%d Syslog-ng\\Bytes=%d", count, bytes); closelog(); lastupdate += REPORT_INTERVAL; count = 0; bytes = 0; } }
} </syslog-perf.c>
Currently logging 39G a day and just keeping up, with minirsyslog. The problem is the disk io is killing me. :( each byte requires 1 write then 1 read to process and 1 read to compress and finally 1/3 write 1 + 1/3 write and 2 reads I am using lzop compression, gzip just won't keep up. And I avoid perl like the plague. I have one second stage data analyzer script in perl and it takes a good chunk of the resources. Thinking of adding lzo compression directly in to syslog-ng and with a fifo I can reduce my disk io to ~ 1/3 compressed byte per raw byte. Does syslog-ng have a hope here performance wise? -tim
On Wed, 2004-12-01 at 02:50, Timothy Webster wrote:
Currently logging 39G a day and just keeping up, with minirsyslog. The problem is the disk io is killing me. :(
each byte requires 1 write then 1 read to process and 1 read to compress and finally 1/3 write 1 + 1/3 write and 2 reads
I am using lzop compression, gzip just won't keep up. And I avoid perl like the plague. I have one second stage data analyzer script in perl and it takes a good chunk of the resources.
Thinking of adding lzo compression directly in to syslog-ng and with a fifo I can reduce my disk io to ~ 1/3 compressed byte per raw byte.
Does syslog-ng have a hope here performance wise?
Sigh. yes and no. I have a nearly complete reimplementation of syslog-ng with nifty features and probably better performance and the possibility to add more features easier. It's been sitting on my hard disk more or less unmodified for about two years now. -- Bazsi
Jay--- Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day. Balazs -- Unfortunately we can't run non-production stuff to test out that secret code you have there, but would like to see that stuff make it to the stable series in the next year... =) I'm sure people wouldn't mind testing... ? In the lab, at least I can... Which reminds me.. I still need to get some more namedpipe logging information, which seems to get lost somewhere.... - Dave On Wed, 01 Dec 2004 09:24:23 +0100, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2004-12-01 at 02:50, Timothy Webster wrote:
Currently logging 39G a day and just keeping up, with minirsyslog. The problem is the disk io is killing me. :(
each byte requires 1 write then 1 read to process and 1 read to compress and finally 1/3 write 1 + 1/3 write and 2 reads
I am using lzop compression, gzip just won't keep up. And I avoid perl like the plague. I have one second stage data analyzer script in perl and it takes a good chunk of the resources.
Thinking of adding lzo compression directly in to syslog-ng and with a fifo I can reduce my disk io to ~ 1/3 compressed byte per raw byte.
Does syslog-ng have a hope here performance wise?
Sigh. yes and no. I have a nearly complete reimplementation of syslog-ng with nifty features and probably better performance and the possibility to add more features easier. It's been sitting on my hard disk more or less unmodified for about two years now.
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Fri, 2004-12-03 at 00:18, Dave Johnson wrote:
Jay---
Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day.
geee. how many mail messages generate this amount of log messages daily? My guess would be about 60-80 million. Hmm.. your averate message size is quite large (266 bytes) Is this some kind of free email service provider? -- Bazsi
Any worries I had syslog-ng handling growth are pretty much erased. :-) Now I only have to worry about diskio and the load of the parsers... On Thu, 2 Dec 2004 17:18:54 -0600, Dave Johnson <davejjohnson@gmail.com> wrote:
Jay---
Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day.
About 40 millions lines per day from 8 servers here. Centralized syslog-ng, running on simple Dell PowerEdge 350 with IDE disks (SW RAID 0 on Linux). No performance problems, except of searching in logs. Jay Guerette wrote:
Any worries I had syslog-ng handling growth are pretty much erased. :-) Now I only have to worry about diskio and the load of the parsers...
On Thu, 2 Dec 2004 17:18:54 -0600, Dave Johnson <davejjohnson@gmail.com> wrote:
Jay---
Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day.
-- *********************************************************************** Pavel Urban (pavel.urban@ct.cz) IOL system disaster Internet OnLine, www.iol.cz (owned by Czech Telecom, www.ct.cz) *********************************************************************** Vegetables should not operate electronic equipment. Computer Stupidities, http://rinkworks.com/stupid/ ***********************************************************************
On Fri, 3 Dec 2004, Jay Guerette wrote:
Any worries I had syslog-ng handling growth are pretty much erased. :-) Now I only have to worry about diskio and the load of the parsers...
My daily throughput is about half of Dave's. Using a perl live analyzer, sporting almost 800 (well organized) rules, a dual AMD 2800+ runs a load of about .7 at peak, with syslog-ng forking the incoming streams to the analyzer, and to disk. - billn
On Thu, 2 Dec 2004 17:18:54 -0600, Dave Johnson <davejjohnson@gmail.com> wrote:
Jay---
Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day.
For performance reasons we dump raw output to disk and don't use a live analyzer-- destination hosts { file("/slog/$YEAR$MONTH$DAY/$HOST/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; destination useronlyhosts { file("/slog/$YEAR$MONTH$DAY/$HOST/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) template ("$MSG\n") ); }; And there are some valid reasons why we can't at this time (reporting tools are from vendor and we are upgrading platform, logfile format changing/evolving, etc...) ------------- Here are a set of machines that just handle incoming email connections (no content filtering): /machine1/-rw-r--r-- 1 root root 9164 Dec 2 16:58 auth /machine1/-rw-r--r-- 1 root root 125836 Dec 2 23:58 mail /machine1/-rw-r--r-- 1 root root 1464 Dec 2 23:26 syslog /machine1/-rw-r--r-- 1 root root 6186893798 Dec 3 00:00 user /machine2/-rw-r--r-- 1 root root 76570 Dec 2 23:50 auth /machine2/-rw-r--r-- 1 root root 68374 Dec 2 23:58 mail /machine2/-rw-r--r-- 1 root root 2086 Dec 2 23:28 syslog /machine2/-rw-r--r-- 1 root root 6173712608 Dec 3 00:00 user /machine3/-rw-r--r-- 1 root root 76405 Dec 2 23:50 auth /machine3/-rw-r--r-- 1 root root 29456 Dec 2 23:40 mail /machine3/-rw-r--r-- 1 root root 1464 Dec 2 23:30 syslog /machine3/-rw-r--r-- 1 root root 6195319607 Dec 3 00:00 user /machine4/-rw-r--r-- 1 root root 76546 Dec 2 23:50 auth /machine4/-rw-r--r-- 1 root root 29474 Dec 2 23:40 mail /machine4/-rw-r--r-- 1 root root 1464 Dec 2 23:31 syslog /machine4/-rw-r--r-- 1 root root 6183132276 Dec 3 00:00 user * This "user" is actually from a couple named pipe sources for that machine, and syslog-ng hasn't a current mechansim to change facilitiy for sources. ------------ * This is for a medium sized ISP... * These numbers are running on a central Sun V240 (dual 1.2Ghz) server running Sol9. Storage is to an EMC disk array with .5 TB allocated to this server. * Balaz, yeah 266 bytes per syslog line average, for email volume, factor in: - Everyday there are a few million connections blocked (ala rbls) - Content filtering information - Email errors/bouncing/etc... Alright, so after all this is said and done, its only a few email million messages a day... And there are a few cpuhours for this process-- Jul 09 ? 30241:34 /usr/local/sbin/syslog-ng On Fri, 3 Dec 2004 10:06:43 -0800 (PST), Bill Nash <billn@billn.net> wrote:
On Fri, 3 Dec 2004, Jay Guerette wrote:
Any worries I had syslog-ng handling growth are pretty much erased. :-) Now I only have to worry about diskio and the load of the parsers...
My daily throughput is about half of Dave's. Using a perl live analyzer, sporting almost 800 (well organized) rules, a dual AMD 2800+ runs a load of about .7 at peak, with syslog-ng forking the incoming streams to the analyzer, and to disk.
- billn
On Thu, 2 Dec 2004 17:18:54 -0600, Dave Johnson <davejjohnson@gmail.com> wrote:
Jay---
Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day.
syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
[syslog]$ du 15289740 ./11-29 I'm chucking 15 gigs of syslog per day, to the tune of: [11-29]$ cat * | nice wc -l 40784743 On top of that, the entirety of it is thrown to the mercy of a perl based log analyzer (single threaded, no less), which in turn filters and logs to a db, at an average rate of 472 lines per second. Cheers. =) - billn On Tue, 30 Nov 2004, Jay Guerette wrote:
( I hope I don't offend anyone with an attachment, it's only 8k. )
My largest single-server syslog-ng implementation currently handles over 13M lines per day, totalling about 1.8Gb per day. I've only recently been able to gather this data by creating a process to count incoming lines, sum their lengths, and report via syslog at 1 minute intervals. See attached graph ( if the attachment survived ).
I added this configuration before all other entries to make sure it sees everything:
<syslog-ng.conf> destination syslog-perf { program(syslog-perf); }; log { source(syslog); destination(syslog-perf); }; </syslog-ng.conf>
I originally tried this in Perl, then Bash, but neither could keep up with the incoming messages. This works like a champ. It compiles on Linux. The output format is specific to my syslog-to-rrd implementation, but you get the idea. It is suitable for an installation that is assured of at least 1 message for each reporting interval!
<syslog-perf.c> #include <stdio.h> #include <time.h> #include <syslog.h>
#define BUFFER_SIZE 8192 #define REPORT_INTERVAL 60
void main(void) {
char buf[BUFFER_SIZE]; long count, bytes; time_t lastupdate;
lastupdate = time(NULL); while (fgets(buf, BUFFER_SIZE, stdin)) { count++; bytes += (strlen(buf) - 1); if (time(NULL) > (lastupdate + REPORT_INTERVAL)) { openlog("127.0.0.1", LOG_NDELAY, LOG_LOCAL3); syslog(LOG_INFO, "Syslog-ng\\Lines=%d Syslog-ng\\Bytes=%d", count, bytes); closelog(); lastupdate += REPORT_INTERVAL; count = 0; bytes = 0; } }
} </syslog-perf.c>
That was going to be my next question! ;] So what do you use to parse all that data? I'm using Logmuncher which has proven to be quite flexible and meets my needs. Russell On Tue, Nov 30, 2004 at 12:21:26PM -0800, Bill Nash wrote:
[syslog]$ du 15289740 ./11-29
I'm chucking 15 gigs of syslog per day, to the tune of: [11-29]$ cat * | nice wc -l 40784743
On top of that, the entirety of it is thrown to the mercy of a perl based log analyzer (single threaded, no less), which in turn filters and logs to a db, at an average rate of 472 lines per second.
Cheers. =)
- billn
On Tue, 30 Nov 2004, Jay Guerette wrote:
( I hope I don't offend anyone with an attachment, it's only 8k. )
My largest single-server syslog-ng implementation currently handles over 13M lines per day, totalling about 1.8Gb per day. I've only recently been able to gather this data by creating a process to count incoming lines, sum their lengths, and report via syslog at 1 minute intervals. See attached graph ( if the attachment survived ).
I added this configuration before all other entries to make sure it sees everything:
<syslog-ng.conf> destination syslog-perf { program(syslog-perf); }; log { source(syslog); destination(syslog-perf); }; </syslog-ng.conf>
I originally tried this in Perl, then Bash, but neither could keep up with the incoming messages. This works like a champ. It compiles on Linux. The output format is specific to my syslog-to-rrd implementation, but you get the idea. It is suitable for an installation that is assured of at least 1 message for each reporting interval!
<syslog-perf.c> #include <stdio.h> #include <time.h> #include <syslog.h>
#define BUFFER_SIZE 8192 #define REPORT_INTERVAL 60
void main(void) {
char buf[BUFFER_SIZE]; long count, bytes; time_t lastupdate;
lastupdate = time(NULL); while (fgets(buf, BUFFER_SIZE, stdin)) { count++; bytes += (strlen(buf) - 1); if (time(NULL) > (lastupdate + REPORT_INTERVAL)) { openlog("127.0.0.1", LOG_NDELAY, LOG_LOCAL3); syslog(LOG_INFO, "Syslog-ng\\Lines=%d Syslog-ng\\Bytes=%d", count, bytes); closelog(); lastupdate += REPORT_INTERVAL; count = 0; bytes = 0; } }
} </syslog-perf.c>
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Tue, 30 Nov 2004, Russell Adams wrote:
That was going to be my next question! ;]
So what do you use to parse all that data?
I'm using Logmuncher which has proven to be quite flexible and meets my needs.
On Tue, Nov 30, 2004 at 12:21:26PM -0800, Bill Nash wrote:
[syslog]$ du 15289740 ./11-29
I'm chucking 15 gigs of syslog per day, to the tune of: [11-29]$ cat * | nice wc -l 40784743
On top of that, the entirety of it is thrown to the mercy of a perl based log analyzer (single threaded, no less), which in turn filters and logs to a db, at an average rate of 472 lines per second.
I use a custom POE based near real-time analyzer I built myself. Syslog-ng forks the incoming streams, logging to disk, and then logging (unfiltered) to tcp target, which is a socket on my analyzer daemon. It breaks up the incoming packet by facility and severity, runs a set of rules against it, and performs any number of functions on the backend depending on the rule matched (logging to mysql, alarms, etc). It's essentially a perl version of Netcool that uses perl's regexp engine. - billn
participants (7)
-
Balazs Scheidler
-
Bill Nash
-
Dave Johnson
-
Jay Guerette
-
Pavel Urban
-
Russell Adams
-
Timothy Webster