[syslog-ng] Blog post on ELSA

[Bill Anderson]

Given that postgresql outperforms mysql in high writes and high concurrency situations and that more time is spent computing indexes rather that retrieving data from the db, sphinx on pgsql should be quite fast actually. Also on multi-core systems postgresql is significantly faster. The sphinx folks confirm that db communication isn't the bottleneck.

Another Factor to consider is regulations or company policy regarding log storage. MyISAM is terrible when it cones to this, that's the tradeoff. For data integrity you'd be wiser to choose InnoDB or postgresql. Also o note is that postgresql can store the data compressed to begin with thus obviating the need for An archive.

I've tested both directly an for my original needs neither was fast enough though postgresql was much faster than mysql in every case I tested. I would be wary of tossing out assumed numbers about performance. I've seen postgresql copy and even faster of_upload handles well over a gig of data per minute when the input file is much larger than system memory.


Given this, I would say try it and see. 


On Apr 1, 2011, at 7:09, "Martin Holste" <mcholste at gmail.com> wrote:

> Good question.  It would theoretically work with Postgres with a fair
> amount of code modification, using COPY instead of LOAD DATA for the
> imports, and an ODBC connection for Sphinx.  However, it would be a
> lot less time to install MySQL and gain a cursory knowledge of it than
> to modify all of the code the allow for Postgres.  The other issue is
> that Postgres would be much slower in this case because the MySQL
> native connections in Sphinx are much faster than the ODBC connections
> (something like 50k rows/sec versus 6k rows/sec for indexing), and
> MyISAM tables are simpler than transaction-oriented tables, so they
> load data faster.  Lastly, you'd lose the ability to archive logs, as
> it uses the MySQL ARCHIVE storage engine which provides an incredible
> compression ratio because it compresses an entire 32k chunk of table,
> not string by string.
> 
> I would strongly urge you to try MySQL instead of modifying the code.
> It should install very quickly on almost any platform and there won't
> be much tuning or administration required.
> 
> On Fri, Apr 1, 2011 at 2:11 AM, Hendrik Visage <hvjunk at gmail.com> wrote:
>> On Mon, Mar 28, 2011 at 10:26 PM, Martin Holste <mcholste at gmail.com> wrote:
>>> I just put up an entry on my blog (http://ossectools.blogspot.com)
>>> describing the Enterprise Log Search and Archive Project
>>> (http://code.google.com/p/enterprise-log-search-and-archive) I've been
>>> working on which uses Syslog-NG >= 3.1 and pattern-db at its core.
>> 
>> How easily could me make it DB independent? Specifically I have the
>> need for PostgreSQL instead of MySQL (Inhouse suport/knowledge)
>> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>