[syslog-ng] syslog-ng + database performance

[Arya, Manish Kumar]

Hi,

>     I have seen that most of the UIs are avialable
> with databases.
>     I have syslog-ng+oracle setup too. but I am not
> happy with performance.

Any numbers, performance figures?
[Manish] database drops 80% of events compared to
files

>     we have a central log server with 3000G SAN and
15
> GB RAM. and 20,000 devices are suppose to pump logs
> 24x7 :)

What's your expected/measured:

o rate of arrival avg/peak in lines/s and socket
connections/s [Manish] avg 300 peak 1000
o data volume/s of this central log server? [Manish]
1~2 GB per day, but this will go to 15 GB when will
will add more deives very soon

What kind of hardware/OS/Oracle version and
Configuration are you 
running this central log server with? 
[Manish] four 1281 MHz SUNW,UltraSPARC-IIIi
Processors, 16 GB RAM
         oracle 10g and solaris 10

>     with oracle we faced two serious issues, thats
why
> i also started pumping logs in files along with db.

It's almost always faster to write log files to flat
files.
[Manish] Yes, thats why I am loging events in files
too along with db for redundancy.

> -inserts, i have using named pipe to insert logs in
> db, but oracle somehow drops inserts, becuase "rate
of
> arival of events" is much larger than "rate of
insert
> operations". I have noticed that there is about
80-90%
> event drops in db.

Parallel Inserts or one single pipe? Do you purge old
data from your DB 
and if so, in which interval?

[Manish] serial inserts, yes but after months, we have
3000 GB SAN.

> -select, when we search logs, it was really really
bad
> performance it took too long to give results. but
then
> we did indexing on hostname and partitioned table on
> time (new range partition is created after every 6
> hrs)
> This improved system performance to some extent.

What's your time frame expectation regarding your
select statements?
[Manish] should return results within 5 sec atmost.
though after doing range partition this has improved
somewhat.

> can you guys suggest me if mysql or postgre will be
> better to overcome above to problems (but remember
our
> db is huge :), so I am not sure if mysql or postgre
is
> able to handle such big db)


Regards,
-Manish

--- Roberto Nibali <ratz at drugphish.ch> wrote:

> Hello,
> 
> >     I have seen that most of the UIs are avialable
> > with databases.
> >     I have syslog-ng+oracle setup too. but I am
> not
> > happy with performance.
> 
> Any numbers, performance figures?
> 
> >     we have a central log server with 3000G SAN
> and 15
> > GB RAM. and 20,000 devices are suppose to pump
> logs
> > 24x7 :)
> 
> What's your expected/measured:
> 
> o rate of arrival avg/peak in lines/s and socket
> connections/s
> o data volume/s
> 
> of this central log server?
> 
> What kind of hardware/OS/Oracle version and
> Configuration are you 
> running this central log server with?
> 
> >     with oracle we faced two serious issues, thats
> why
> > i also started pumping logs in files along with
> db.
> 
> It's almost always faster to write log files to flat
> files.
> 
> > -inserts, i have using named pipe to insert logs
> in
> > db, but oracle somehow drops inserts, becuase
> "rate of
> > arival of events" is much larger than "rate of
> insert
> > operations". I have noticed that there is about
> 80-90%
> > event drops in db.
> 
> Parallel Inserts or one single pipe? Do you purge
> old data from your DB 
> and if so, in which interval?
> 
> > -select, when we search logs, it was really really
> bad
> > performance it took too long to give results. but
> then
> > we did indexing on hostname and partitioned table
> on
> > time (new range partition is created after every 6
> > hrs)
> > This improved system performance to some extent.
> 
> What's your time frame expectation regarding your
> select statements?
> 
> > can you guys suggest me if mysql or postgre will
> be
> > better to overcome above to problems (but remember
> our
> > db is huge :), so I am not sure if mysql or
> postgre is
> > able to handle such big db)
> 
> Cheers,
> Roberto Nibali, ratz
> -- 
> echo 
>
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'
> | dc
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at
> http://www.campin.net/syslog-ng/faq.html
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com