[syslog-ng] syslog-ng + database performance

[Arya, Manish Kumar]

have you noticed that events are being droped by db?
try to push logs into files n db, and then count
number of events in db and file for certain time :)
in my case files have  8 to 10 times more events than
in db. which means sql is droping many events

--- Didier Conchaudron <didier at conchaudron.net> wrote:

> Hi,
> 
> I'm experiencing problems too but not that way.
> 
> I maintain a loghost for hundreds of servers, and
> every peace of log 
> goes into a
> Postgresql DB. One week of activity is about 90GB of
> data.
> 
> Syslog write into a named pipe, a Perl script read
> it and execute the INSERT
> queries. The loghost itself is a P4 2,8Ghz with 2GB
> of memory and have no
> problems for doing the inserts.
> 
> The problems I have comes from the SELECT done into
> the DB used for reporting.
> I've resolved most of the performances issues with
> specific indexes, 
> especially
> for full text researches.
> 
>  From my point of view, your problem comes the
> Oracle DB tweaking, and probably
> from the listener itself. But my Oracle knowledge is
> poor.
> 
> Cheers,
> 
> Didier
> 
> Quoting "Arya, Manish Kumar" <m.arya at yahoo.com>:
> 
> > Hi Guys,
> >
> >    Thanks for your valuable suggestions for
> syslog-ng
> > UI.
> >
> >    I have seen that most of the UIs are avialable
> > with databases.
> >    I have syslog-ng+oracle setup too. but I am not
> > happy with performance.
> >    we have a central log server with 3000G SAN and
> 15
> > GB RAM. and 20,000 devices are suppose to pump
> logs
> > 24x7 :)
> >    with oracle we faced two serious issues, thats
> why
> > i also started pumping logs in files along with
> db.
> >
> > -inserts, i have using named pipe to insert logs
> in
> > db, but oracle somehow drops inserts, becuase
> "rate of
> > arival of events" is much larger than "rate of
> insert
> > operations". I have noticed that there is about
> 80-90%
> > event drops in db.
> >
> > -select, when we search logs, it was really really
> bad
> > performance it took too long to give results. but
> then
> > we did indexing on hostname and partitioned table
> on
> > time (new range partition is created after every 6
> > hrs)
> > This improved system performance to some extent.
> >
> > can you guys suggest me if mysql or postgre will
> be
> > better to overcome above to problems (but remember
> our
> > db is huge :), so I am not sure if mysql or
> postgre is
> > able to handle such big db)
> >
> > Regards,
> > -Manish
> >
> > --- Jon Stearley <jrstear at sandia.gov> wrote:
> >
> >>
> >> On May 11, 2006, at 12:09 PM, Ken Garland wrote:
> >>
> >> >>
> >>
> file("/logs/log01/indexlog/$YEAR/$MONTH/$DAY/$HOST"
> >> >>   ...
> >> >> -should be able to to parallel search to
> improve
> >> >> search response time.
> >>
> >> If you decide to go with SQL and have $$,
> >> netezza.com will almost
> >> certainly overcome your speed issues (parallel
> >> harware sql!).  Having
> >> gotten utterly bogged down with Mysql on Linux
> >> (stripes, chunks, huge
> >> indexes), I just went back to files because they
> are
> >> simple and
> >> sufficient for my purposes.
> >>
> >> > if you are splitting all logs up into subdirs
> like
> >> that you will
> >> > have quite a fun time doing any parsing.
> >>
> >> If dirs/logs are arranged according to the
> factors
> >> used for subset
> >> selection (year/month/day/host) and the dirs/logs
> >> are listed in a
> >> (periodically updated) file (eg "corpus.docs" in
> >> sisyphus), subset
> >> selection can be done by simply grepping the file
> >> and concatenating
> >> the resulting dirs/logs.  This is one
> implementation
> >> option
> >> underlying the clog.man page I sent earlier.
> >> Further subset
> >> selection by facility and priority could then be
> >> done by grepping the
> >> resulting log content (further dirs/logs
> splitting
> >> by facility/
> >> priority presents multiple bad side effects). 
> $0.02
> >>
> >> -jon
> >>
> >>
> >> _______________________________________________
> >> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> >>
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Frequently asked questions at
> >> http://www.campin.net/syslog-ng/faq.html
> >>
> >>
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around
> > http://mail.yahoo.com
> > _______________________________________________
> > syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> >
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Frequently asked questions at
> http://www.campin.net/syslog-ng/faq.html
> >
> >
> 
> 
> 
>
----------------------------------------------------------------
> This message was sent using IMP, the Internet
> Messaging Program.
> 
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at
> http://www.campin.net/syslog-ng/faq.html
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com