[syslog-ng] syslog-ng + database performance

[Didier Conchaudron]

I've not tested. But as you saying that I will probably do it ;-)

Didier

Quoting "Arya, Manish Kumar" <m.arya at yahoo.com>:

> have you noticed that events are being droped by db?
> try to push logs into files n db, and then count
> number of events in db and file for certain time :)
> in my case files have  8 to 10 times more events than
> in db. which means sql is droping many events
>
> --- Didier Conchaudron <didier at conchaudron.net> wrote:
>
>> Hi,
>>
>> I'm experiencing problems too but not that way.
>>
>> I maintain a loghost for hundreds of servers, and
>> every peace of log
>> goes into a
>> Postgresql DB. One week of activity is about 90GB of
>> data.
>>
>> Syslog write into a named pipe, a Perl script read
>> it and execute the INSERT
>> queries. The loghost itself is a P4 2,8Ghz with 2GB
>> of memory and have no
>> problems for doing the inserts.
>>
>> The problems I have comes from the SELECT done into
>> the DB used for reporting.
>> I've resolved most of the performances issues with
>> specific indexes,
>> especially
>> for full text researches.
>>
>>  From my point of view, your problem comes the
>> Oracle DB tweaking, and probably
>> from the listener itself. But my Oracle knowledge is
>> poor.
>>
>> Cheers,
>>
>> Didier
>>
>> Quoting "Arya, Manish Kumar" <m.arya at yahoo.com>:
>>
>> > Hi Guys,
>> >
>> >    Thanks for your valuable suggestions for
>> syslog-ng
>> > UI.
>> >
>> >    I have seen that most of the UIs are avialable
>> > with databases.
>> >    I have syslog-ng+oracle setup too. but I am not
>> > happy with performance.
>> >    we have a central log server with 3000G SAN and
>> 15
>> > GB RAM. and 20,000 devices are suppose to pump
>> logs
>> > 24x7 :)
>> >    with oracle we faced two serious issues, thats
>> why
>> > i also started pumping logs in files along with
>> db.
>> >
>> > -inserts, i have using named pipe to insert logs
>> in
>> > db, but oracle somehow drops inserts, becuase
>> "rate of
>> > arival of events" is much larger than "rate of
>> insert
>> > operations". I have noticed that there is about
>> 80-90%
>> > event drops in db.
>> >
>> > -select, when we search logs, it was really really
>> bad
>> > performance it took too long to give results. but
>> then
>> > we did indexing on hostname and partitioned table
>> on
>> > time (new range partition is created after every 6
>> > hrs)
>> > This improved system performance to some extent.
>> >
>> > can you guys suggest me if mysql or postgre will
>> be
>> > better to overcome above to problems (but remember
>> our
>> > db is huge :), so I am not sure if mysql or
>> postgre is
>> > able to handle such big db)
>> >
>> > Regards,
>> > -Manish
>> >
>> > --- Jon Stearley <jrstear at sandia.gov> wrote:
>> >
>> >>
>> >> On May 11, 2006, at 12:09 PM, Ken Garland wrote:
>> >>
>> >> >>
>> >>
>> file("/logs/log01/indexlog/$YEAR/$MONTH/$DAY/$HOST"
>> >> >>   ...
>> >> >> -should be able to to parallel search to
>> improve
>> >> >> search response time.
>> >>
>> >> If you decide to go with SQL and have $$,
>> >> netezza.com will almost
>> >> certainly overcome your speed issues (parallel
>> >> harware sql!).  Having
>> >> gotten utterly bogged down with Mysql on Linux
>> >> (stripes, chunks, huge
>> >> indexes), I just went back to files because they
>> are
>> >> simple and
>> >> sufficient for my purposes.
>> >>
>> >> > if you are splitting all logs up into subdirs
>> like
>> >> that you will
>> >> > have quite a fun time doing any parsing.
>> >>
>> >> If dirs/logs are arranged according to the
>> factors
>> >> used for subset
>> >> selection (year/month/day/host) and the dirs/logs
>> >> are listed in a
>> >> (periodically updated) file (eg "corpus.docs" in
>> >> sisyphus), subset
>> >> selection can be done by simply grepping the file
>> >> and concatenating
>> >> the resulting dirs/logs.  This is one
>> implementation
>> >> option
>> >> underlying the clog.man page I sent earlier.
>> >> Further subset
>> >> selection by facility and priority could then be
>> >> done by grepping the
>> >> resulting log content (further dirs/logs
>> splitting
>> >> by facility/
>> >> priority presents multiple bad side effects).
>> $0.02
>> >>
>> >> -jon
>> >>
>> >>
>> >> _______________________________________________
>> >> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
>> >>
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> >> Frequently asked questions at
>> >> http://www.campin.net/syslog-ng/faq.html
>> >>
>> >>
>> >
>> >
>> > __________________________________________________
>> > Do You Yahoo!?
>> > Tired of spam?  Yahoo! Mail has the best spam
>> protection around
>> > http://mail.yahoo.com
>> > _______________________________________________
>> > syslog-ng maillist  -  syslog-ng at lists.balabit.hu
>> >
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Frequently asked questions at
>> http://www.campin.net/syslog-ng/faq.html
>> >
>> >
>>
>>
>>
>>
> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet
>> Messaging Program.
>>
>> _______________________________________________
>> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Frequently asked questions at
>> http://www.campin.net/syslog-ng/faq.html
>>
>>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.