nova team mailing list archive

[Rick Clark <rick@xxxxxxxxxxxxx>]

I think the abstraction is good, but I would rather wait until we have a
chance to add redis support.


On 09/10/2010 12:08 PM, Vishvananda Ishaya wrote:
> I don't understand the comment that this doesn't play nicely with
> non-relational data stores.  There is a very clear abstraction layer
> db/api.py that would allow a different backend to be plugged in
> regardless of whether sqlalchemy ever supports it.  I don't think it
> would be too difficult to add db/redis/api.py to this system.
> 
> The one possible gotcha is that there isn't a clear definition of the
> properties that each object needs to have anywhere outside of
> sqlalchemy/models.py and the relations that are needed.  This
> ultimately should be fixed with either documentation or some kind of
> middle tier classes that define the needed properties.
> 
> Delaying until after austin would be a bit troublesome for us since
> we have to move of of redis.  That means a pretty strongly diverged
> branch and any features that anso is working on will probably have to
> be delayed as well.
> 
> Vish
> 
> On Sep 10, 2010, at 9:56 AM, Jay Pipes wrote:
> 
>> Hi Vish,
>> 
>> Such a large patch has taken me quite some time to digest.  There
>> is a larger discussion on large patches without any specifications,
>> but I'll leave that for a later time! :)
>> 
>> I am torn on this one, mostly because I spent a bunch of time 
>> attempting to do the datastore refactoring myself (as did Justin
>> Santa Barbara), and thus I know the dragons that live in this layer
>> of the code :)
>> 
>> One of the things that both Justin and I had tried was to keep an 
>> abstraction layer that would allow both NoSQL as well as SQL data 
>> stores to be used.  Unfortunately, it seems that this patch
>> removes the ability to use ReDIS, among other NoSQL stores.  I
>> think this is a mistake, and although I like much of the code in
>> this patch, I was hoping that SQLAlchemy could be hidden behind an
>> abstraction layer that would play nicely with the non-relational
>> data stores.
>> 
>> As this patch stands, we take a 180 degree turn away from NoSQL
>> data stores and back into the relatively comfortable norms of the
>> SQL databases.  While there's nothing particularly wrong with SQL 
>> databases (as you know, I'm a fan of many of them ;) ), I think
>> that keeping non-relational data store capabilities is pretty
>> critical.
>> 
>> After an email discussion with SQLAlchemy's Michael Bayer about 
>> SQLAlchemy's future with NoSQL data stores.  Although there is an 
>> issue in the SQLAlchemy trac system about this (see here: 
>> http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this 
>> module seeing the light of day is unlikely in the next year or
>> two.
>> 
>> So...what to do?  There are at least four options I can see:
>> 
>> 1) Go forward with this patch and add NoSQL stores back at some
>> later time by ourselves 2) Go forward with this patch and wait
>> until SQLAlchemy properly supports key value stores 3) Delay this
>> patch until after the Austin release and have a larger discussion
>> about it here and at the next summit 4) Go back to the drawing
>> board and try again with a less ambitious set of patches that
>> incrementally changes the way the data stores work.
>> 
>> I'm personally on the fence.  I'd prefer to at least delay the
>> patch until after Austin, but I understand there are now at least 4
>> branches that depend on this one, which makes things, well, a bit
>> difficult.
>> 
>> -jay
>> 
>> On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya 
>> <vishvananda@xxxxxxxxx> wrote:
>>> I've proposed a merge of the orm refactor branch that a large
>>> part of the nasa/anso team has been working on.  I'm hoping
>>> everyone can pick it apart and we end up with a really clean
>>> system that everyone likes.  I've copied the description of the
>>> change and issues below.  If the mailing list debates get too
>>> complicated, we should just organize a time to discuss it in
>>> IRC.
>>> 
>>> Proposing merge to get feedback on orm refactoring. I am very
>>> interested in feedback to all of these changes.
>>> 
>>> This is a huge set of changes, that touches almost all of the
>>> files. I'm sure I have broken quite a bit, but better to take the
>>> plunge now than to postpone this until later. The idea is to
>>> allow for pluggable backends throughout the code.
>>> 
>>> Brief Overview For compute/volume/network, there are multiple
>>> classes service - responsible for rpc this currently uses the
>>> existing cast and call in rpc.py and a little bit of magic to
>>> call public methods on the manager class. each service also
>>> reports its state into the database every 10 seconds manager -
>>> responsible for managing respective object classes all the
>>> business logic for the classes go here db (db_driver) -
>>> responsible for abstracting database access driver
>>> (domain_driver) - responsible for executing actual shell commands
>>> and implementation
>>> 
>>> Compute hasn't been fully cleaned up, but to get an idea of how
>>> it works, take a look at volume and network
>>> 
>>> Known issues/Things to be done:
>>> 
>>> * nova-api accesses db objects directly It seems cleaner to have
>>> only the managers dealing with their respective objects. This
>>> would mean code for 'run_instances' would move into the manager
>>> class and it would do the initial setup and cast out to the
>>> remote service
>>> 
>>> * db code uses flat methods to define its interface In my mind
>>> this is a little prettier as an abstract base class, but driver 
>>> loading code can load a module or a class. It works, so I'm not
>>> sure it needs to be changed but feel free to debate it.
>>> 
>>> * Service classes have no code in them Not sure if this is a
>>> problem for people, but the magic of calling the manager's
>>> methods is done in the base class. We could remove the magic from
>>> the base class and explicitly wrap methods that we want to make
>>> available via rpc if this seems nasty.
>>> 
>>> * AuthManager Projects/Users/Roles are not integrated into this
>>> system. In order for everything to live happily in the backend,
>>> we need some type of adaptor for LDAP
>>> 
>>> * Context is not passed properly across rabbit Context should
>>> probably be changed to a simple dictionary so that it can be 
>>> passed properly through the queue
>>> 
>>> * No authorization checks on access to objects We need to decide
>>> on which layer auth checks should happen.
>>> 
>>> * Some of the methods in ComputeManager need to be moved into
>>> other layers/managers * Compute driver layer should be abstracted
>>> more cleanly * Flat networking is untested and may need to be
>>> reworked * Some of the api commands are not working yet * Nova
>>> Swift Authentication needs to be refactored(Todd is working on
>>> this)
>>> 
>>> _______________________________________________ Mailing list:
>>> https://launchpad.net/~nova Post to     :
>>> nova@xxxxxxxxxxxxxxxxxxx Unsubscribe :
>>> https://launchpad.net/~nova More help   :
>>> https://help.launchpad.net/ListHelp
>>> 
>>> 
> 
> 
> _______________________________________________ Mailing list:
> https://launchpad.net/~nova Post to     : nova@xxxxxxxxxxxxxxxxxxx 
> Unsubscribe : https://launchpad.net/~nova More help   :
> https://help.launchpad.net/ListHelp

Attachment: signature.asc
Description: OpenPGP digital signature