nova team mailing list archive

[Eric Day <eday@xxxxxxxxxxxx>]

++ on all points. When it was decided to implement a Redis backend for
the DB API, I tried to explain my concerns about recreating SQLAlchemy
with a Redis back-end, and then some. We should decide what data
model we are using (relational, hierarchical, flat key-value, graph,
...) and decide on a native API to use. If we wan to stay relational,
using SQLAlchemy natively and using it's pluggable backend system
may be sufficient.

There is also the distributed storage and privacy concerns to discuss
as well at the summit. For example, each compute/network/volume worker
could keep it's own store and push any required information up to
the other layers (scheduler/API). This would ensure a compromised
worker machine could not touch other workers or their data stores,
and scaling a central data store is no longer an issue. With these
concerns in mind, this may also sway which data model and API we want
to expose to the applications.

-Eric

On Mon, Oct 04, 2010 at 01:34:27PM -0400, Jay Pipes wrote:
> I agree with Soren that big changes like this should occur after some
> discussion (a thought that I rose on IRC last week).
> 
> I look forward to the discussion at the next summit about the ORM in
> general, and specifically, these concerns about it I have:
> 
> 1) Inconsistent (and sometimes incorrect) use of session boundaries in
> the abstraction layer for sqlalchemy.  Peculiar bugs show up when
> sessions aren't handled (passed between functions) properly or when
> transactions are not reliable (NoSQL or when the RDBMS doesn't support
> transactional statements like SELECT ... FOR UPDATE.
> 2) Shared code between Glance and Nova (and possibly Swift, but it's
> less of an issue IMHO).  There is now a copy of the datastore adapter
> from Nova in Glance (this was done to save time).  We will want to
> pull this code out into openstack-common ASAP, as I really don't feel
> like maintaining two copies of the datastore adapter layer ;)
> 3) The Model class itself.  The biggest issue is that the model class
> does not live in the API layer; instead, it must be re-implemented for
> each adapter.  I experienced the joy of trying to re-create the
> existing SQLAlchemy-based NovaBase model class in the Redis adapter.
> This was less than enjoyable.  If we choose to use a base model class,
> it needs to address these fundamental issues *at a bare minimum*:
>   a) The definition of data fields (versus non-data fields, private
> fields, or calculated fields)
>   b) The casting of SQL-like data types (CHAR, INT, BLOB, DATETIME,
> etc) into data type that Python understands consistently.
>   c) A model's understanding (or lack of understanding) of its
> relationships with other types of models.
>   d) The aforementioned debate about what is appropriate as a primary
> key identifier for each model (UUID, URI, auto-inc, etc)
> 
> -jay
> 
> On Sun, Oct 3, 2010 at 5:10 PM, Devin Carlen <devin.carlen@xxxxxxxxx> wrote:
> > Soren,
> >
> > This comment was with regards to using declarative_base(), which was used to quickly implement the data layer and save some time.  The next iteration we've been discussing internally, is to drop the use of declarative_base and using sqlalchemy table definitions instead.
> >
> > declarative_base() provides value primarily when you are planning on actually using sqlalchemy models as the middle tier.  We only use them as a stepping stone in the data layer, and then treat everything as dicts currently.  (Whether we should be using dicts everywhere is part of a larger discussion I hope to bring up at the next design summit.)
> >
> > By using the table definitions directly and bypassing declarative_base(), we will be able to cut out two superfluous and unnecessary transformations before the data is passed back to the middle tier.
> >
> > There are no immediate plans to do this, but you are right - it is worthy of further discussion.
> >
> >
> > Devin
> >
> >
> > This only was meant to indicate that we would be changing from u
> >
> > On Oct 2, 2010, at 2:52 AM, Soren Hansen wrote:
> >
> >> Looking at the commit message for r316 on trunk, I see this (emphasis mine):
> >>
> >>    Also refactored NovaBase and removed several methods since they
> >>    would have to be changed *when we move away from sqlalchemy models*
> >>    and begin using sqlalchemy table definitions.
> >>
> >> I find it worrisome that such things are stated as though they are
> >> naturally given or generally agreed upon. I have no recollection of a
> >> discussion to that effect, let alone a decision.
> >>
> >> I certainly hope a big patch to do this doesn't suddenly turn up without
> >> prior discussion and a related blueprint..
> >>
> >> --
> >> Soren Hansen
> >> Ubuntu Developer    http://www.ubuntu.com/
> >> OpenStack Developer http://www.openstack.org/
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~nova
> >> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~nova
> >> More help   : https://help.launchpad.net/ListHelp
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~nova
> > Post to     : nova@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~nova
> > More help   : https://help.launchpad.net/ListHelp
> >
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~nova
> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~nova
> More help   : https://help.launchpad.net/ListHelp