abandon libdbi and write a new sql driver?
Reading through the mailing list lately I've seen a fair amount of problems reported with libdbi. Given that it also doesnt support prepared statements which is a very important feature, I'm wondering if libdbi should just be dropped entirely. Libdbi seems to be of use to applications that want just very basic sql support, and not for anything involving high performance or reliability. I know its no small undertaking to write a new DBI for all the common databases out there, but if syslog-ng is supposed to be high-performing, libdbi does not fit in with that. I dont know if it'd be best to start a new project to replace libdbi but accomplish the same thing of providing a common API for all databases, or to just write separate modules for syslog-ng 3.2+ for each database out there. Writing modules would certainly be easier as they could be written independently by people who best know each database API. This is all just my opinion, but I for one am all for it. In my production environment, we do logging to a database, and libdbi just didnt have the flexibility or performance we needed, so I ended up writing a separate program which connects to the database to perform the inserts. Would be nice to avoid having to do that. I know I've also heard many recommendations of piping out to a perl script to do the inserts. A perl script should never do its job faster than C code. -Patrick
I know I've also heard many recommendations of piping out to a perl script to do the inserts. A perl script should never do its job faster than C code.
Certainly never faster, but the degree to which it is slower should be weighed against the time commitment (and resultant opportunity cost) of creating a whole new library. That said, I'm all for a new library. Something that could speak HandlerSocket would be a huge win.
On Thu, 2011-01-20 at 13:17 -0700, Patrick H. wrote:
Reading through the mailing list lately I've seen a fair amount of problems reported with libdbi. Given that it also doesnt support prepared statements which is a very important feature, I'm wondering if libdbi should just be dropped entirely. Libdbi seems to be of use to applications that want just very basic sql support, and not for anything involving high performance or reliability.
I know its no small undertaking to write a new DBI for all the common databases out there, but if syslog-ng is supposed to be high-performing, libdbi does not fit in with that. I dont know if it'd be best to start a new project to replace libdbi but accomplish the same thing of providing a common API for all databases, or to just write separate modules for syslog-ng 3.2+ for each database out there. Writing modules would certainly be easier as they could be written independently by people who best know each database API.
This is all just my opinion, but I for one am all for it. In my production environment, we do logging to a database, and libdbi just didnt have the flexibility or performance we needed, so I ended up writing a separate program which connects to the database to perform the inserts. Would be nice to avoid having to do that. I know I've also heard many recommendations of piping out to a perl script to do the inserts. A perl script should never do its job faster than C code.
Honestly I was thinking about forking libdbi/libdbi-drivers as we do have a couple of changes and upstream is not really doing regular releases. And that's the key problem. libdbi is basically stalled as a project and without them releasing bugfixes is causing us and our users pain. But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel? -- Bazsi
Sent: Mon Jan 24 2011 04:12:37 GMT-0700 (Mountain Standard Time) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] abandon libdbi and write a new sql driver?
On Thu, 2011-01-20 at 13:17 -0700, Patrick H. wrote:
Reading through the mailing list lately I've seen a fair amount of problems reported with libdbi. Given that it also doesnt support prepared statements which is a very important feature, I'm wondering if libdbi should just be dropped entirely. Libdbi seems to be of use to applications that want just very basic sql support, and not for anything involving high performance or reliability.
I know its no small undertaking to write a new DBI for all the common databases out there, but if syslog-ng is supposed to be high-performing, libdbi does not fit in with that. I dont know if it'd be best to start a new project to replace libdbi but accomplish the same thing of providing a common API for all databases, or to just write separate modules for syslog-ng 3.2+ for each database out there. Writing modules would certainly be easier as they could be written independently by people who best know each database API.
This is all just my opinion, but I for one am all for it. In my production environment, we do logging to a database, and libdbi just didnt have the flexibility or performance we needed, so I ended up writing a separate program which connects to the database to perform the inserts. Would be nice to avoid having to do that. I know I've also heard many recommendations of piping out to a perl script to do the inserts. A perl script should never do its job faster than C code.
Honestly I was thinking about forking libdbi/libdbi-drivers as we do have a couple of changes and upstream is not really doing regular releases. And that's the key problem. libdbi is basically stalled as a project and without them releasing bugfixes is causing us and our users pain.
But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel?
Well thats certainly an option. I just remember looking at the code once and it wasnt very clean. But that was something like a year ago and I may be remembering some other project I looked at. Primarily I just didnt think libdbi in its current state is in line with what syslog-ng is trying to accomplish. How that gets resolved isnt that important. I know I would love to have a mature common database API I could use in some of my other projects :-P -Patrick
On Mon, Jan 24, 2011 at 12:12:37PM +0100, Balazs Scheidler wrote:
But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel?
Not having prepared statements means every insert requires a parse. That cuts the performance of transactional DBs a lot. Particularly Oracle suffers from this because generating too many statements overflows its memory pools.
-- Bazsi
Matthew.
On Mon, 2011-01-24 at 09:10 -0800, Matthew Hall wrote:
On Mon, Jan 24, 2011 at 12:12:37PM +0100, Balazs Scheidler wrote:
But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel?
Not having prepared statements means every insert requires a parse.
That cuts the performance of transactional DBs a lot.
Particularly Oracle suffers from this because generating too many statements overflows its memory pools.
I see, good to know. Thanks for the information. -- Bazsi
ODBC allows for prepared statements and is supported on all major databases. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler [bazsi@balabit.hu] Sent: Saturday, February 05, 2011 5:30 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] abandon libdbi and write a new sql driver? On Mon, 2011-01-24 at 09:10 -0800, Matthew Hall wrote:
On Mon, Jan 24, 2011 at 12:12:37PM +0100, Balazs Scheidler wrote:
But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel?
Not having prepared statements means every insert requires a parse.
That cuts the performance of transactional DBs a lot.
Particularly Oracle suffers from this because generating too many statements overflows its memory pools.
I see, good to know. Thanks for the information. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi, The problem with ODBC is that it is a _big_ package, with a separate daemon running, and last I've checked database drivers are not necessarily free. (this applies to the unixodbc package). On Sat, 2011-02-05 at 10:08 -0800, Evan Rempel wrote:
ODBC allows for prepared statements and is supported on all major databases. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler [bazsi@balabit.hu] Sent: Saturday, February 05, 2011 5:30 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] abandon libdbi and write a new sql driver?
On Mon, 2011-01-24 at 09:10 -0800, Matthew Hall wrote:
On Mon, Jan 24, 2011 at 12:12:37PM +0100, Balazs Scheidler wrote:
But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel?
Not having prepared statements means every insert requires a parse.
That cuts the performance of transactional DBs a lot.
Particularly Oracle suffers from this because generating too many statements overflows its memory pools.
I see, good to know. Thanks for the information.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
We run ODBC on our systems and there is not another daemon to run. It is a larger "system" to get installed but when in use it is just the application and the odbc driver. The only database that I am aware of that does not have a free odbc driver is Oracle. If there is another general purpose database abstraction layer then by all means explore it. I am only aware of dbi and odbc Evan On 2011-02-22, at 6:30 AM, "Balazs Scheidler" <bazsi@balabit.hu> wrote:
Hi,
The problem with ODBC is that it is a _big_ package, with a separate daemon running, and last I've checked database drivers are not necessarily free. (this applies to the unixodbc package).
On Sat, 2011-02-05 at 10:08 -0800, Evan Rempel wrote:
ODBC allows for prepared statements and is supported on all major databases. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng- bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler [bazsi@balabit.hu] Sent: Saturday, February 05, 2011 5:30 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] abandon libdbi and write a new sql driver?
On Mon, 2011-01-24 at 09:10 -0800, Matthew Hall wrote:
On Mon, Jan 24, 2011 at 12:12:37PM +0100, Balazs Scheidler wrote:
But I don't see anything against libdbi per-se, within syslog-ng a similar API would be needed to cover all database APIs, so why reinvent the wheel?
Not having prepared statements means every insert requires a parse.
That cuts the performance of transactional DBs a lot.
Particularly Oracle suffers from this because generating too many statements overflows its memory pools.
I see, good to know. Thanks for the information.
-- Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2011-01-20 at 13:17 -0700, Patrick H. wrote:
Reading through the mailing list lately I've seen a fair amount of problems reported with libdbi. Given that it also doesnt support prepared statements which is a very important feature, I'm wondering if libdbi should just be dropped entirely. Libdbi seems to be of use to applications that want just very basic sql support, and not for anything involving high performance or reliability.
Just an FYI: I plan to write database specific SQL destination drivers, that _will_ make use of the features the database drivers provide (prepared statements included). I might end up with a thin, syslog-ng specific common layer between these drivers, but that's not going to be a libdbi replacement in any shape or form. And I think that's perfectly fine. The current plan is to have an afmysql and afpgsql destination first, with the rest coming afterwards. I do not have an ETA at the moment, as this is all in my head so far, and there's plenty of higher priority tasks I have to finish first. But I figured I'd let you all know that this is something that I want to work on in the not too distant future. -- |8]
participants (6)
-
Balazs Scheidler
-
Evan Rempel
-
Gergely Nagy
-
Martin Holste
-
Matthew Hall
-
Patrick H.