[OPEN-ILS-DEV] cstore vs storage

[Mike Rylander]

On Mon, Dec 1, 2008 at 5:51 PM, James Fournie
<jfournie at sitka.bclibraries.ca> wrote:
> Hi there,
>
> I am just wondering the differences between open-ils.cstore and
> open-ils.storage. I am just wondering if there are any guidelines for
> using one or the other of these, and if there is a list of the
> differences.
>

Some history will help, I think.

open-ils.storage was the first ORM we built.  It's written in Perl (as
you know) and it started life with two goals:  to provide simple CRUD
and search methods for all object types backed by tables and views;
and to provide a platform for complex queries that needed to be
constructed by hand (for speed or abstraction, but of which the rest
of the system should not need to know the details.

Once upon a time it had about 25,000 generated methods for performing
various different search operations against every field on every
table.  All that's really left now is a simple search method for each
table that accepts a hash of criteria, along with the afformentioned
CRUD (and id-list) methods.  All of the CRUD methods now have analogs
in open-ils.cstore now, but more on that later.

The second (complex queries returning OpenSRF objects) part has lived
longer, though it is constantly being pared down as well.  The bib
search methods, hold targeter implementation, fine generator and fa
few other complex methods see the majority of the use now.  There will
always be a need for some biz-logic/database border methods that do
something more than simple CRUD, and today that is where these border
methods are built.  We want to keep the number of ways the database
can be directly touched to a minimum, and since Perl is much easier
than C to write quickly it will be around for a while.

open-ils.cstore, on the other hand, is written in C.  It was built
mainly in response to performance issues with open-ils.storage,
particularly the CRUD interfaces.  Long story short, simple CRUD and
search methods went from taking a minimum of 35ms for a full round
trip (usually significantly more) to taking 3-5ms.  Thank You C!  For
more complex methods, C is a bit too rigid to encode them directly,
but in those cases the perl overhead is swamped by the query time
anyway, so it's not a big deal in many cases.

What we were lacking all along, though, was a way to construct
query-language agnostic (meaning, flavors of SQL, really) arbitrary
queries.  This is open-ils.cstore's secondary role, which is becoming
fairly advanced in 1.0.1 and beyond.  It's possible now to build a
data structure which mirrors the basic layout of an SQL query and have
OpenSRF objects returned to the client.  This means that many of the
more complex open-ils.storage methods can be built using
open-ils.cstore, and things that were not possible before without
hand-tooling SQL are now relatively easy, and faster.

So, to answer your question, open-ils.storage is where methods that
need to know about the both the database structure and the business
logic requirements of Evergreen live, and while it does have a CRUD
interface (and, in a couple specific cases, should be used for CRUD)
you should generally go to open-ils.cstore for CRUD access.  Also,
take a look at some of the more complex open-ils.cstore.json_query
calls scattered throughout the Perl-based OpenSRF apps in the open-ils
namespace for a glimpse of the cstore-tastic future.

There are some /really/ outdated docs for open-ils.storage on the
wiki, but here are some (more or less) up to date, if now incomplete
for json_query, docs for open-ils.cstore:

http://open-ils.org/dokuwiki/doku.php?id=backend-devel:open-ils_cstore_api


-- 
Mike Rylander
 | VP, Research and Design
 | Equinox Software, Inc. / The Evergreen Experts
 | phone:  1-877-OPEN-ILS (673-6457)
 | email:  miker at esilibrary.com
 | web:  http://www.esilibrary.com