nova team mailing list archive

[Rick Clark <rick@xxxxxxxxxxxxx>]

I'm gong to guess redis' lack of HA.  Did I guess right?

On 09/10/2010 12:19 PM, Jay Pipes wrote:
> On Fri, Sep 10, 2010 at 1:08 PM, Vishvananda Ishaya
> <vishvananda@xxxxxxxxx> wrote:
>> I don't understand the comment that this doesn't play nicely with non-relational data stores.  There is a very clear abstraction layer db/api.py that would allow a different backend to be plugged in regardless of whether sqlalchemy ever supports it.  I don't think it would be too difficult to add db/redis/api.py to this system.
>>
>> The one possible gotcha is that there isn't a clear definition of the properties that each object needs to have anywhere outside of sqlalchemy/models.py and the relations that are needed.  This ultimately should be fixed with either documentation or some kind of middle tier classes that define the needed properties.
>>
>> Delaying until after austin would be a bit troublesome for us since we have to move of of redis.  That means a pretty strongly diverged branch and any features that anso is working on will probably have to be delayed as well.
> 
> Maybe I'm out of some loop, but this is the first I'm hearing of any
> need for you guys to move off of Redis.  Can I ask why this move is
> necessary?  Lack of resources who know how to manage ReDIS stores, or
> something else?  Just curious.
> 
> Also, as far as features that Anso is working on, where can I see
> these features?  We use the blueprints system on Launchpad to manage
> precisely these kind of things.  Having a chunk of coders working on
> their own ideas and features with no insight into what those things
> are spells duplicated effort to me (as this patch duplicated much
> effort put in by a few folks on the datastore stuff..)  It would be
> helpful to know what everyone is working on by looking at the singular
> list of blueprints and using the Launchpad milestone/release system to
> its full effect.
> 
> -jay
> 
>> Vish
>>
>> On Sep 10, 2010, at 9:56 AM, Jay Pipes wrote:
>>
>>> Hi Vish,
>>>
>>> Such a large patch has taken me quite some time to digest.  There is a
>>> larger discussion on large patches without any specifications, but
>>> I'll leave that for a later time! :)
>>>
>>> I am torn on this one, mostly because I spent a bunch of time
>>> attempting to do the datastore refactoring myself (as did Justin Santa
>>> Barbara), and thus I know the dragons that live in this layer of the
>>> code :)
>>>
>>> One of the things that both Justin and I had tried was to keep an
>>> abstraction layer that would allow both NoSQL as well as SQL data
>>> stores to be used.  Unfortunately, it seems that this patch removes
>>> the ability to use ReDIS, among other NoSQL stores.  I think this is a
>>> mistake, and although I like much of the code in this patch, I was
>>> hoping that SQLAlchemy could be hidden behind an abstraction layer
>>> that would play nicely with the non-relational data stores.
>>>
>>> As this patch stands, we take a 180 degree turn away from NoSQL data
>>> stores and back into the relatively comfortable norms of the SQL
>>> databases.  While there's nothing particularly wrong with SQL
>>> databases (as you know, I'm a fan of many of them ;) ), I think that
>>> keeping non-relational data store capabilities is pretty critical.
>>>
>>> After an email discussion with SQLAlchemy's Michael Bayer about
>>> SQLAlchemy's future with NoSQL data stores.  Although there is an
>>> issue in the SQLAlchemy trac system about this (see here:
>>> http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this
>>> module seeing the light of day is unlikely in the next year or two.
>>>
>>> So...what to do?  There are at least four options I can see:
>>>
>>> 1) Go forward with this patch and add NoSQL stores back at some later
>>> time by ourselves
>>> 2) Go forward with this patch and wait until SQLAlchemy properly
>>> supports key value stores
>>> 3) Delay this patch until after the Austin release and have a larger
>>> discussion about it here and at the next summit
>>> 4) Go back to the drawing board and try again with a less ambitious
>>> set of patches that incrementally changes the way the data stores
>>> work.
>>>
>>> I'm personally on the fence.  I'd prefer to at least delay the patch
>>> until after Austin, but I understand there are now at least 4 branches
>>> that depend on this one, which makes things, well, a bit difficult.
>>>
>>> -jay
>>>
>>> On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya
>>> <vishvananda@xxxxxxxxx> wrote:
>>>> I've proposed a merge of the orm refactor branch that a large part of the
>>>> nasa/anso team has been working on.  I'm hoping everyone can pick it apart
>>>> and we end up with a really clean system that everyone likes.  I've copied
>>>> the description of the change and issues below.  If the mailing list debates
>>>> get too complicated, we should just organize a time to discuss it in IRC.
>>>>
>>>> Proposing merge to get feedback on orm refactoring. I am very interested in
>>>> feedback to all of these changes.
>>>>
>>>> This is a huge set of changes, that touches almost all of the files. I'm
>>>> sure I have broken quite a bit, but better to take the plunge now than to
>>>> postpone this until later. The idea is to allow for pluggable backends
>>>> throughout the code.
>>>>
>>>> Brief Overview
>>>> For compute/volume/network, there are multiple classes
>>>> service - responsible for rpc
>>>>   this currently uses the existing cast and call in rpc.py and a little bit
>>>> of magic
>>>>   to call public methods on the manager class.
>>>>   each service also reports its state into the database every 10 seconds
>>>> manager - responsible for managing respective object classes
>>>>   all the business logic for the classes go here
>>>> db (db_driver) - responsible for abstracting database access
>>>> driver (domain_driver) - responsible for executing actual shell commands and
>>>> implementation
>>>>
>>>> Compute hasn't been fully cleaned up, but to get an idea of how it works,
>>>> take a look
>>>> at volume and network
>>>>
>>>> Known issues/Things to be done:
>>>>
>>>> * nova-api accesses db objects directly
>>>>   It seems cleaner to have only the managers dealing with their respective
>>>> objects. This would
>>>>   mean code for 'run_instances' would move into the manager class and it
>>>> would do the initial
>>>>   setup and cast out to the remote service
>>>>
>>>> * db code uses flat methods to define its interface
>>>>   In my mind this is a little prettier as an abstract base class, but driver
>>>> loading code
>>>>   can load a module or a class. It works, so I'm not sure it needs to be
>>>> changed but feel
>>>>   free to debate it.
>>>>
>>>> * Service classes have no code in them
>>>>   Not sure if this is a problem for people, but the magic of calling the
>>>> manager's methods is
>>>>   done in the base class. We could remove the magic from the base class and
>>>> explicitly
>>>>   wrap methods that we want to make available via rpc if this seems nasty.
>>>>
>>>> * AuthManager Projects/Users/Roles are not integrated into this system.
>>>>   In order for everything to live happily in the backend, we need some type
>>>>   of adaptor for LDAP
>>>>
>>>> * Context is not passed properly across rabbit
>>>>   Context should probably be changed to a simple dictionary so that it can
>>>> be
>>>>   passed properly through the queue
>>>>
>>>> * No authorization checks on access to objects
>>>>   We need to decide on which layer auth checks should happen.
>>>>
>>>> * Some of the methods in ComputeManager need to be moved into other
>>>> layers/managers
>>>> * Compute driver layer should be abstracted more cleanly
>>>> * Flat networking is untested and may need to be reworked
>>>> * Some of the api commands are not working yet
>>>> * Nova Swift Authentication needs to be refactored(Todd is working on this)
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~nova
>>>> Post to     : nova@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~nova
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>
>>
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~nova
> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~nova
> More help   : https://help.launchpad.net/ListHelp

Attachment: signature.asc
Description: OpenPGP digital signature