nova team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

Hi Vish,

Such a large patch has taken me quite some time to digest.  There is a
larger discussion on large patches without any specifications, but
I'll leave that for a later time! :)

I am torn on this one, mostly because I spent a bunch of time
attempting to do the datastore refactoring myself (as did Justin Santa
Barbara), and thus I know the dragons that live in this layer of the
code :)

One of the things that both Justin and I had tried was to keep an
abstraction layer that would allow both NoSQL as well as SQL data
stores to be used.  Unfortunately, it seems that this patch removes
the ability to use ReDIS, among other NoSQL stores.  I think this is a
mistake, and although I like much of the code in this patch, I was
hoping that SQLAlchemy could be hidden behind an abstraction layer
that would play nicely with the non-relational data stores.

As this patch stands, we take a 180 degree turn away from NoSQL data
stores and back into the relatively comfortable norms of the SQL
databases.  While there's nothing particularly wrong with SQL
databases (as you know, I'm a fan of many of them ;) ), I think that
keeping non-relational data store capabilities is pretty critical.

After an email discussion with SQLAlchemy's Michael Bayer about
SQLAlchemy's future with NoSQL data stores.  Although there is an
issue in the SQLAlchemy trac system about this (see here:
http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this
module seeing the light of day is unlikely in the next year or two.

So...what to do?  There are at least four options I can see:

1) Go forward with this patch and add NoSQL stores back at some later
time by ourselves
2) Go forward with this patch and wait until SQLAlchemy properly
supports key value stores
3) Delay this patch until after the Austin release and have a larger
discussion about it here and at the next summit
4) Go back to the drawing board and try again with a less ambitious
set of patches that incrementally changes the way the data stores
work.

I'm personally on the fence.  I'd prefer to at least delay the patch
until after Austin, but I understand there are now at least 4 branches
that depend on this one, which makes things, well, a bit difficult.

-jay

On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya
<vishvananda@xxxxxxxxx> wrote:
> I've proposed a merge of the orm refactor branch that a large part of the
> nasa/anso team has been working on.  I'm hoping everyone can pick it apart
> and we end up with a really clean system that everyone likes.  I've copied
> the description of the change and issues below.  If the mailing list debates
> get too complicated, we should just organize a time to discuss it in IRC.
>
> Proposing merge to get feedback on orm refactoring. I am very interested in
> feedback to all of these changes.
>
> This is a huge set of changes, that touches almost all of the files. I'm
> sure I have broken quite a bit, but better to take the plunge now than to
> postpone this until later. The idea is to allow for pluggable backends
> throughout the code.
>
> Brief Overview
> For compute/volume/network, there are multiple classes
> service - responsible for rpc
>   this currently uses the existing cast and call in rpc.py and a little bit
> of magic
>   to call public methods on the manager class.
>   each service also reports its state into the database every 10 seconds
> manager - responsible for managing respective object classes
>   all the business logic for the classes go here
> db (db_driver) - responsible for abstracting database access
> driver (domain_driver) - responsible for executing actual shell commands and
> implementation
>
> Compute hasn't been fully cleaned up, but to get an idea of how it works,
> take a look
> at volume and network
>
> Known issues/Things to be done:
>
> * nova-api accesses db objects directly
>   It seems cleaner to have only the managers dealing with their respective
> objects. This would
>   mean code for 'run_instances' would move into the manager class and it
> would do the initial
>   setup and cast out to the remote service
>
> * db code uses flat methods to define its interface
>   In my mind this is a little prettier as an abstract base class, but driver
> loading code
>   can load a module or a class. It works, so I'm not sure it needs to be
> changed but feel
>   free to debate it.
>
> * Service classes have no code in them
>   Not sure if this is a problem for people, but the magic of calling the
> manager's methods is
>   done in the base class. We could remove the magic from the base class and
> explicitly
>   wrap methods that we want to make available via rpc if this seems nasty.
>
> * AuthManager Projects/Users/Roles are not integrated into this system.
>   In order for everything to live happily in the backend, we need some type
>   of adaptor for LDAP
>
> * Context is not passed properly across rabbit
>   Context should probably be changed to a simple dictionary so that it can
> be
>   passed properly through the queue
>
> * No authorization checks on access to objects
>   We need to decide on which layer auth checks should happen.
>
> * Some of the methods in ComputeManager need to be moved into other
> layers/managers
> * Compute driver layer should be abstracted more cleanly
> * Flat networking is untested and may need to be reworked
> * Some of the api commands are not working yet
> * Nova Swift Authentication needs to be refactored(Todd is working on this)
>
> _______________________________________________
> Mailing list: https://launchpad.net/~nova
> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~nova
> More help   : https://help.launchpad.net/ListHelp
>
>