nova team mailing list archive

[Ewan Mellor <ewan.mellor@xxxxxxxxxxxxx>]

Are you both (and anyone else interested) going to work on a draft blueprint
before the summit?  I'm keen to see discussion drafts in place before we
all meet in person, otherwise we'll waste a lot of time just getting on the
same page.

Thanks,

Ewan.

On Mon, Oct 04, 2010 at 07:01:57PM +0100, Eric Day wrote:

> ++ on all points. When it was decided to implement a Redis backend for
> the DB API, I tried to explain my concerns about recreating SQLAlchemy
> with a Redis back-end, and then some. We should decide what data
> model we are using (relational, hierarchical, flat key-value, graph,
> ...) and decide on a native API to use. If we wan to stay relational,
> using SQLAlchemy natively and using it's pluggable backend system
> may be sufficient.
> 
> There is also the distributed storage and privacy concerns to discuss
> as well at the summit. For example, each compute/network/volume worker
> could keep it's own store and push any required information up to
> the other layers (scheduler/API). This would ensure a compromised
> worker machine could not touch other workers or their data stores,
> and scaling a central data store is no longer an issue. With these
> concerns in mind, this may also sway which data model and API we want
> to expose to the applications.
> 
> -Eric
> 
> On Mon, Oct 04, 2010 at 01:34:27PM -0400, Jay Pipes wrote:
> > I agree with Soren that big changes like this should occur after some
> > discussion (a thought that I rose on IRC last week).
> > 
> > I look forward to the discussion at the next summit about the ORM in
> > general, and specifically, these concerns about it I have:
> > 
> > 1) Inconsistent (and sometimes incorrect) use of session boundaries in
> > the abstraction layer for sqlalchemy.  Peculiar bugs show up when
> > sessions aren't handled (passed between functions) properly or when
> > transactions are not reliable (NoSQL or when the RDBMS doesn't support
> > transactional statements like SELECT ... FOR UPDATE.
> > 2) Shared code between Glance and Nova (and possibly Swift, but it's
> > less of an issue IMHO).  There is now a copy of the datastore adapter
> > from Nova in Glance (this was done to save time).  We will want to
> > pull this code out into openstack-common ASAP, as I really don't feel
> > like maintaining two copies of the datastore adapter layer ;)
> > 3) The Model class itself.  The biggest issue is that the model class
> > does not live in the API layer; instead, it must be re-implemented for
> > each adapter.  I experienced the joy of trying to re-create the
> > existing SQLAlchemy-based NovaBase model class in the Redis adapter.
> > This was less than enjoyable.  If we choose to use a base model class,
> > it needs to address these fundamental issues *at a bare minimum*:
> >   a) The definition of data fields (versus non-data fields, private
> > fields, or calculated fields)
> >   b) The casting of SQL-like data types (CHAR, INT, BLOB, DATETIME,
> > etc) into data type that Python understands consistently.
> >   c) A model's understanding (or lack of understanding) of its
> > relationships with other types of models.
> >   d) The aforementioned debate about what is appropriate as a primary
> > key identifier for each model (UUID, URI, auto-inc, etc)
> > 
> > -jay
> > 
> > On Sun, Oct 3, 2010 at 5:10 PM, Devin Carlen <devin.carlen@xxxxxxxxx> wrote:
> > > Soren,
> > >
> > > This comment was with regards to using declarative_base(), which was used to quickly implement the data layer and save some time.  The next iteration we've been discussing internally, is to drop the use of declarative_base and using sqlalchemy table definitions instead.
> > >
> > > declarative_base() provides value primarily when you are planning on actually using sqlalchemy models as the middle tier.  We only use them as a stepping stone in the data layer, and then treat everything as dicts currently.  (Whether we should be using dicts everywhere is part of a larger discussion I hope to bring up at the next design summit.)
> > >
> > > By using the table definitions directly and bypassing declarative_base(), we will be able to cut out two superfluous and unnecessary transformations before the data is passed back to the middle tier.
> > >
> > > There are no immediate plans to do this, but you are right - it is worthy of further discussion.
> > >
> > >
> > > Devin
> > >
> > >
> > > This only was meant to indicate that we would be changing from u
> > >
> > > On Oct 2, 2010, at 2:52 AM, Soren Hansen wrote:
> > >
> > >> Looking at the commit message for r316 on trunk, I see this (emphasis mine):
> > >>
> > >>    Also refactored NovaBase and removed several methods since they
> > >>    would have to be changed *when we move away from sqlalchemy models*
> > >>    and begin using sqlalchemy table definitions.
> > >>
> > >> I find it worrisome that such things are stated as though they are
> > >> naturally given or generally agreed upon. I have no recollection of a
> > >> discussion to that effect, let alone a decision.
> > >>
> > >> I certainly hope a big patch to do this doesn't suddenly turn up without
> > >> prior discussion and a related blueprint..
> > >>
> > >> --
> > >> Soren Hansen
> > >> Ubuntu Developer    http://www.ubuntu.com/
> > >> OpenStack Developer http://www.openstack.org/
> > >>
> > >> _______________________________________________
> > >> Mailing list: https://launchpad.net/~nova
> > >> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> > >> Unsubscribe : https://launchpad.net/~nova
> > >> More help   : https://help.launchpad.net/ListHelp
> > >
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~nova
> > > Post to     : nova@xxxxxxxxxxxxxxxxxxx
> > > Unsubscribe : https://launchpad.net/~nova
> > > More help   : https://help.launchpad.net/ListHelp
> > >
> > 
> > _______________________________________________
> > Mailing list: https://launchpad.net/~nova
> > Post to     : nova@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~nova
> > More help   : https://help.launchpad.net/ListHelp
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~nova
> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~nova
> More help   : https://help.launchpad.net/ListHelp