[syslog-ng]FYI: Performance

[Dave Johnson]

For performance reasons we dump raw output to disk and don't use a
live analyzer--

destination hosts {
        file("/slog/$YEAR$MONTH$DAY/$HOST/$FACILITY"
        owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) );
        };
destination useronlyhosts {
        file("/slog/$YEAR$MONTH$DAY/$HOST/$FACILITY"
        owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)
        template ("$MSG\n") );
        };

 And there are some valid reasons why we can't at this time
(reporting tools are from vendor and we are upgrading platform,  
 logfile format changing/evolving, etc...)

-------------

Here are a set of machines that just handle incoming email connections 
(no content filtering):
/machine1/-rw-r--r--   1 root     root        9164 Dec  2 16:58 auth
/machine1/-rw-r--r--   1 root     root      125836 Dec  2 23:58 mail
/machine1/-rw-r--r--   1 root     root        1464 Dec  2 23:26 syslog
/machine1/-rw-r--r--   1 root     root     6186893798 Dec  3 00:00 user
/machine2/-rw-r--r--   1 root     root       76570 Dec  2 23:50 auth
/machine2/-rw-r--r--   1 root     root       68374 Dec  2 23:58 mail
/machine2/-rw-r--r--   1 root     root        2086 Dec  2 23:28 syslog
/machine2/-rw-r--r--   1 root     root     6173712608 Dec  3 00:00 user
/machine3/-rw-r--r--   1 root     root       76405 Dec  2 23:50 auth
/machine3/-rw-r--r--   1 root     root       29456 Dec  2 23:40 mail
/machine3/-rw-r--r--   1 root     root        1464 Dec  2 23:30 syslog
/machine3/-rw-r--r--   1 root     root     6195319607 Dec  3 00:00 user
/machine4/-rw-r--r--   1 root     root       76546 Dec  2 23:50 auth
/machine4/-rw-r--r--   1 root     root       29474 Dec  2 23:40 mail
/machine4/-rw-r--r--   1 root     root        1464 Dec  2 23:31 syslog
/machine4/-rw-r--r--   1 root     root     6183132276 Dec  3 00:00 user

* This "user" is actually from a couple named pipe sources for that machine, and
   syslog-ng hasn't a current mechansim to change facilitiy for sources.

------------

* This is for a medium sized ISP...
* These numbers are running on a central Sun V240 (dual 1.2Ghz)
  server running Sol9.  Storage is to an EMC disk array with .5 TB allocated
  to this server.
* Balaz, yeah 266 bytes per syslog line average, for email volume, factor in:
  - Everyday there are a few million connections blocked (ala rbls)
  - Content filtering information
  - Email errors/bouncing/etc...
  Alright, so after all this is said and done, its only a few email million 
  messages a day...

And there are a few cpuhours for this process--
Jul 09 ?       30241:34 /usr/local/sbin/syslog-ng

On Fri, 3 Dec 2004 10:06:43 -0800 (PST), Bill Nash <billn@billn.net> wrote:
> 
> On Fri, 3 Dec 2004, Jay Guerette wrote:
> 
> > Any worries I had syslog-ng handling growth are pretty much erased. :-)
> > Now I only have to worry about diskio and the load of the parsers...
> 
> My daily throughput is about half of Dave's. Using a perl live analyzer,
> sporting almost 800 (well organized) rules, a dual AMD 2800+ runs a load
> of about .7 at peak, with syslog-ng forking the incoming streams to the
> analyzer, and to disk.
> 
> - billn
> 
> 
> 
> > On Thu, 2 Dec 2004 17:18:54 -0600, Dave Johnson <davejjohnson@gmail.com> wrote:
> >> Jay---
> >>
> >>  Yesterday, our email log server here did 47069024518  bytes or
> >> 176818253 lines a day.
> _______________________________________________
> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
>