maas-devel team mailing list archive

[Clint Byrum <clint.byrum@xxxxxxxxxxxxx>]

Excerpts from Julian Edwards's message of Thu May 03 23:42:12 -0700 2012:
> On Friday 04 May 2012 18:12:44 Robert Collins wrote:
> > I would discourage, strongly discourage, any direct DB access from
> > pserv: our experience with LP with such access has been universally
> > bad. Let the appserver drive the DB exclusively, and offer appropriate
> > APIs for getting stuff from/to it. I think we glossed over this on
> > IRC; celery talking to postgresql might mean this needs some extra
> > glue for celery, or something.
> 
> For read access only, can you elaborate why this is bad?
> 

I've not been involved with Launchpad, but I have done a few multi-tiered
architectures.

There are a few reasons:

* The database used is an implementation detail. Putting a lightweight
layer of indirection between the DB and the other pieces of the app means
being able to swap out the DB for the cases that matter. With the hyper
scale requirement, this is likely to happen as it becomes clear which
tables just cannot be served through a purely relational model. API's
map intentions rather than implementations.

* API's can be used as layers of control. The postgres and mysql
protocols both make proxying a real chore, and so, its hard to control
the number of threads. pgbouncer seems pretty good, but it then requires
a dedicated proxy just for pgsql, which ties you further into pg. An API
call, however, can be extended to provide needed metrics, and then be
an intelligent choke point or pressure-release for a limited resource
like the database.

* Intelligence in the pipeline. This makes it easier to cache
intelligently, easier to route/shard/etc. The layer of indirection used
to be just in code, but you really need it in the network separation
so that the pieces can be scaled individually and whole parts can be
refactored without touching every place that might access that place.

Put more succinctly, API changes are easier than schema changes.