For performance reasons we dump raw output to disk and don't use a live analyzer-- destination hosts { file("/slog/$YEAR$MONTH$DAY/$HOST/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; destination useronlyhosts { file("/slog/$YEAR$MONTH$DAY/$HOST/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) template ("$MSG\n") ); }; And there are some valid reasons why we can't at this time (reporting tools are from vendor and we are upgrading platform, logfile format changing/evolving, etc...) ------------- Here are a set of machines that just handle incoming email connections (no content filtering): /machine1/-rw-r--r-- 1 root root 9164 Dec 2 16:58 auth /machine1/-rw-r--r-- 1 root root 125836 Dec 2 23:58 mail /machine1/-rw-r--r-- 1 root root 1464 Dec 2 23:26 syslog /machine1/-rw-r--r-- 1 root root 6186893798 Dec 3 00:00 user /machine2/-rw-r--r-- 1 root root 76570 Dec 2 23:50 auth /machine2/-rw-r--r-- 1 root root 68374 Dec 2 23:58 mail /machine2/-rw-r--r-- 1 root root 2086 Dec 2 23:28 syslog /machine2/-rw-r--r-- 1 root root 6173712608 Dec 3 00:00 user /machine3/-rw-r--r-- 1 root root 76405 Dec 2 23:50 auth /machine3/-rw-r--r-- 1 root root 29456 Dec 2 23:40 mail /machine3/-rw-r--r-- 1 root root 1464 Dec 2 23:30 syslog /machine3/-rw-r--r-- 1 root root 6195319607 Dec 3 00:00 user /machine4/-rw-r--r-- 1 root root 76546 Dec 2 23:50 auth /machine4/-rw-r--r-- 1 root root 29474 Dec 2 23:40 mail /machine4/-rw-r--r-- 1 root root 1464 Dec 2 23:31 syslog /machine4/-rw-r--r-- 1 root root 6183132276 Dec 3 00:00 user * This "user" is actually from a couple named pipe sources for that machine, and syslog-ng hasn't a current mechansim to change facilitiy for sources. ------------ * This is for a medium sized ISP... * These numbers are running on a central Sun V240 (dual 1.2Ghz) server running Sol9. Storage is to an EMC disk array with .5 TB allocated to this server. * Balaz, yeah 266 bytes per syslog line average, for email volume, factor in: - Everyday there are a few million connections blocked (ala rbls) - Content filtering information - Email errors/bouncing/etc... Alright, so after all this is said and done, its only a few email million messages a day... And there are a few cpuhours for this process-- Jul 09 ? 30241:34 /usr/local/sbin/syslog-ng On Fri, 3 Dec 2004 10:06:43 -0800 (PST), Bill Nash <billn@billn.net> wrote:
On Fri, 3 Dec 2004, Jay Guerette wrote:
Any worries I had syslog-ng handling growth are pretty much erased. :-) Now I only have to worry about diskio and the load of the parsers...
My daily throughput is about half of Dave's. Using a perl live analyzer, sporting almost 800 (well organized) rules, a dual AMD 2800+ runs a load of about .7 at peak, with syslog-ng forking the incoming streams to the analyzer, and to disk.
- billn
On Thu, 2 Dec 2004 17:18:54 -0600, Dave Johnson <davejjohnson@gmail.com> wrote:
Jay---
Yesterday, our email log server here did 47069024518 bytes or 176818253 lines a day.
syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html