| Thread Previous • Date Previous • Date Next • Thread Next |
On Fri, Sep 10, 2010 at 1:08 PM, Vishvananda Ishaya <vishvananda@xxxxxxxxx> wrote: > I don't understand the comment that this doesn't play nicely with non-relational data stores. There is a very clear abstraction layer db/api.py that would allow a different backend to be plugged in regardless of whether sqlalchemy ever supports it. I don't think it would be too difficult to add db/redis/api.py to this system. > > The one possible gotcha is that there isn't a clear definition of the properties that each object needs to have anywhere outside of sqlalchemy/models.py and the relations that are needed. This ultimately should be fixed with either documentation or some kind of middle tier classes that define the needed properties. > > Delaying until after austin would be a bit troublesome for us since we have to move of of redis. That means a pretty strongly diverged branch and any features that anso is working on will probably have to be delayed as well. Maybe I'm out of some loop, but this is the first I'm hearing of any need for you guys to move off of Redis. Can I ask why this move is necessary? Lack of resources who know how to manage ReDIS stores, or something else? Just curious. Also, as far as features that Anso is working on, where can I see these features? We use the blueprints system on Launchpad to manage precisely these kind of things. Having a chunk of coders working on their own ideas and features with no insight into what those things are spells duplicated effort to me (as this patch duplicated much effort put in by a few folks on the datastore stuff..) It would be helpful to know what everyone is working on by looking at the singular list of blueprints and using the Launchpad milestone/release system to its full effect. -jay > Vish > > On Sep 10, 2010, at 9:56 AM, Jay Pipes wrote: > >> Hi Vish, >> >> Such a large patch has taken me quite some time to digest. There is a >> larger discussion on large patches without any specifications, but >> I'll leave that for a later time! :) >> >> I am torn on this one, mostly because I spent a bunch of time >> attempting to do the datastore refactoring myself (as did Justin Santa >> Barbara), and thus I know the dragons that live in this layer of the >> code :) >> >> One of the things that both Justin and I had tried was to keep an >> abstraction layer that would allow both NoSQL as well as SQL data >> stores to be used. Unfortunately, it seems that this patch removes >> the ability to use ReDIS, among other NoSQL stores. I think this is a >> mistake, and although I like much of the code in this patch, I was >> hoping that SQLAlchemy could be hidden behind an abstraction layer >> that would play nicely with the non-relational data stores. >> >> As this patch stands, we take a 180 degree turn away from NoSQL data >> stores and back into the relatively comfortable norms of the SQL >> databases. While there's nothing particularly wrong with SQL >> databases (as you know, I'm a fan of many of them ;) ), I think that >> keeping non-relational data store capabilities is pretty critical. >> >> After an email discussion with SQLAlchemy's Michael Bayer about >> SQLAlchemy's future with NoSQL data stores. Although there is an >> issue in the SQLAlchemy trac system about this (see here: >> http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this >> module seeing the light of day is unlikely in the next year or two. >> >> So...what to do? There are at least four options I can see: >> >> 1) Go forward with this patch and add NoSQL stores back at some later >> time by ourselves >> 2) Go forward with this patch and wait until SQLAlchemy properly >> supports key value stores >> 3) Delay this patch until after the Austin release and have a larger >> discussion about it here and at the next summit >> 4) Go back to the drawing board and try again with a less ambitious >> set of patches that incrementally changes the way the data stores >> work. >> >> I'm personally on the fence. I'd prefer to at least delay the patch >> until after Austin, but I understand there are now at least 4 branches >> that depend on this one, which makes things, well, a bit difficult. >> >> -jay >> >> On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya >> <vishvananda@xxxxxxxxx> wrote: >>> I've proposed a merge of the orm refactor branch that a large part of the >>> nasa/anso team has been working on. I'm hoping everyone can pick it apart >>> and we end up with a really clean system that everyone likes. I've copied >>> the description of the change and issues below. If the mailing list debates >>> get too complicated, we should just organize a time to discuss it in IRC. >>> >>> Proposing merge to get feedback on orm refactoring. I am very interested in >>> feedback to all of these changes. >>> >>> This is a huge set of changes, that touches almost all of the files. I'm >>> sure I have broken quite a bit, but better to take the plunge now than to >>> postpone this until later. The idea is to allow for pluggable backends >>> throughout the code. >>> >>> Brief Overview >>> For compute/volume/network, there are multiple classes >>> service - responsible for rpc >>> this currently uses the existing cast and call in rpc.py and a little bit >>> of magic >>> to call public methods on the manager class. >>> each service also reports its state into the database every 10 seconds >>> manager - responsible for managing respective object classes >>> all the business logic for the classes go here >>> db (db_driver) - responsible for abstracting database access >>> driver (domain_driver) - responsible for executing actual shell commands and >>> implementation >>> >>> Compute hasn't been fully cleaned up, but to get an idea of how it works, >>> take a look >>> at volume and network >>> >>> Known issues/Things to be done: >>> >>> * nova-api accesses db objects directly >>> It seems cleaner to have only the managers dealing with their respective >>> objects. This would >>> mean code for 'run_instances' would move into the manager class and it >>> would do the initial >>> setup and cast out to the remote service >>> >>> * db code uses flat methods to define its interface >>> In my mind this is a little prettier as an abstract base class, but driver >>> loading code >>> can load a module or a class. It works, so I'm not sure it needs to be >>> changed but feel >>> free to debate it. >>> >>> * Service classes have no code in them >>> Not sure if this is a problem for people, but the magic of calling the >>> manager's methods is >>> done in the base class. We could remove the magic from the base class and >>> explicitly >>> wrap methods that we want to make available via rpc if this seems nasty. >>> >>> * AuthManager Projects/Users/Roles are not integrated into this system. >>> In order for everything to live happily in the backend, we need some type >>> of adaptor for LDAP >>> >>> * Context is not passed properly across rabbit >>> Context should probably be changed to a simple dictionary so that it can >>> be >>> passed properly through the queue >>> >>> * No authorization checks on access to objects >>> We need to decide on which layer auth checks should happen. >>> >>> * Some of the methods in ComputeManager need to be moved into other >>> layers/managers >>> * Compute driver layer should be abstracted more cleanly >>> * Flat networking is untested and may need to be reworked >>> * Some of the api commands are not working yet >>> * Nova Swift Authentication needs to be refactored(Todd is working on this) >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~nova >>> Post to : nova@xxxxxxxxxxxxxxxxxxx >>> Unsubscribe : https://launchpad.net/~nova >>> More help : https://help.launchpad.net/ListHelp >>> >>> > >
| Thread Previous • Date Previous • Date Next • Thread Next |
