openstack team mailing list archive

[Justin Santa Barbara <justin@xxxxxxxxxxxx>]

Hi Jay,

I couldn't agree more.  I had another bug come up yesterday on another of my
patches (I know - not a good day for me!) where I again broke the OpenStack
API by requiring the metadata attribute.

In this case, it was missed by the unit tests.  I believe I was always
passing metadata, so simply missed the real world case.  Here's the bug
report:
https://bugs.launchpad.net/nova/+bug/724143

This brings up a number of points though:

On testing


   1. The bug reporter apparently knows how to program, but instead of us
   getting a test case which we could immediately use, we got a Ruby test case.
    I think we should do whatever we can so that people that are moderately
   comfortable with code also feel comfortable submitting a failing test case
   that we can use.  I think this means having some version of an API client,
   or even a Reference-Implementation API client, in the source tree, and using
   it.
   2. It's not clear to me how we deal with unit test cases that are failing
   - cases that represent a found bug that is not yet fixed.  Maybe it should
   be submitted on a bugNNN branch, with failing tests, and then whoever works
   on the bug can branch from it?  (Of course, we'd be lucky if all bugs went
   like this, but it does happen.  I often find myself fixing tangential
   issues, and I'd like to know the 'right thing to do' is)
   3. On this particular metadata bug, it was my fault.  I therefore
   submitted a very rapid hotfix which just fixes the issue.  I then coded up
   unit tests that use the OpenStack API, and did in fact hit the issue
   naturally, and then included my fix which resolved it.  I submitted the
   'full patch' as a separate branch.
   4. This 'full patch' branch includes unit tests that bring up the
   OpenStack API and various services in-process, and runs tests just like a
   user.  It would have caught the metadata issue.
   5. The hope is that we can reuse the same tests as smoke tests, by simply
   tweaking them to work against real services instead of bringing up
   in-process stub-services.  These could be (some of) the smoke tests in
   Hudson.
   6. I'd hope that we could have two levels of these smoke tests, one that
   runs on a single configuration (e.g. KVM, OpenISCSI, Glance); another that
   runs a matrix of configurations (and might take an hour or more to run)
   7. Ideally we'd have a torture test that would run overnight and be
   randomized and try to find obscure bugs, even if issues found are not
   necessarily repeatable in the way that non-randomized tests are.


We need an unstable trunk:


   1. In general, it seems that our end-users are using trunk for unreleased
   functionality and treating it as if it were released.  I don't think we
   should be encouraging that, because I know I'll make more mistakes in future
   and some of them will make it past the reviewers' defensive line into trunk;
   it's also simply not realistic to require reviewers to review every
   combination - e.g. how can a reviewer really review my HP SAN patch without
   an HP SAN?  There will be issues in trunk, and if we have to revert them
   rather than just fixing them it will slow us down.  The current situation is
   bad for our users and bad for developers.
   2. One way we could keep everyone happy is by using our test suite to
   auto-merge from an 'unstable trunk' into 'stable trunk', only once code
   passes tests.  Commits would initially merge into 'unstable trunk', and we
   would try to keep that branch moving forwards rather than reverting things
   that go wrong.  Of course, maintaining a good 'stable trunk' relies on
   having good tests, but I think we're getting there.  It's also great
   incentive to write good smoke tests.
   3. Jay: I believe you've done this to great success on the Drizzle
   project?


Justin



On Thu, Feb 24, 2011 at 6:13 AM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:

> Hi all,
>
> I'd like to bring up an alternate reason why it was approved and
> subsequently reverted.
>
> The test cases for the OpenStack API (and much of the EC2 API) assume
> way too many things and mock out too many things. In addition, since
> there are zero smoketests for the OpenStack API, there were no
> functional tests that would have *immediately* highlighted this
> problem (and many other recent EC2 vs OS API problems).
>
> In other words, sure, we should revert the patch to "fix things",
> however the priority should *not* be on refactoring the auth API or
> the way the auth layer in Nova is handled. The priority should be on
> writing a smoketest for the OpenStack API so that we can link it into
> Hudson and these types of issues can be automatically caught.
>
> -jay
>
> On Wed, Feb 23, 2011 at 10:03 PM, Paul Voccio <paul.voccio@xxxxxxxxxxxxx>
> wrote:
> > Justin,
> > I think you hit upon the reason of why I think it was approved and
> reverted.
> > Because it hadn't been talked about in a blueprint or a mail sent to the
> > list (I think I'm up to date on the threads) and a patch landed means
> other
> > alternatives weren't considered before pushing it through to begin with.
> I
> > think we're all open to talking about how to better the auth system and
> make
> > improvements. Dragon has already discussed some alternatives and
> suggestions
> > on the BP page below. I think this is the right way to continue the
> dialog
> > and we all can agree on a good way forward.
> > I'm confident we can figure it out.
> > If I missed a conversation, my apologies.
> > pvo
> > From: Vishvananda Ishaya <vishvananda@xxxxxxxxx>
> > Date: Wed, 23 Feb 2011 18:19:41 -0800
> > To: Justin Santa Barbara <justin@xxxxxxxxxxxx>
> > Cc: <openstack@xxxxxxxxxxxxxxxxxxx>
> > Subject: Re: [Openstack] Should the OpenStack API re-use the EC2
> > credentials?
> >
> > Hey Justin,
> > Does it make any difference that the way the auth is (theoretically)
> > supposed to work with the os api is that the user gets an auth token from
> an
> > external auth server and then uses username / authtoken to actually
> contact
> > the api?  I think it is just faked out right now to use the access_key
> > instead of doing external auth, but I think the reason it works like it
> does
> > is because the plan was to switch to external auth eventually.
> > Vish
> > On Feb 23, 2011, at 5:56 PM, Justin Santa Barbara wrote:
> >
> > I previously fixed OpenStack authentication so it would use the same
> > credentials as EC2.  This bugfix was just reverted, because it caused
> > OpenStack API users to have to enter in different credentials (sorry!),
> but
> > primarily because it hadn't been discussed on the mailing list.  So here
> > goes!
> > Here's a
> > blueprint:
> https://blueprints.launchpad.net/nova/+spec/authentication-consistency
> > Here's an overview of the problem:
> > EC2 uses an (api_key, api_secret) pair.  Post-revert, OpenStack uses the
> > api_key(!) as the password, but a different value entirely as the
> username:
> > (username, api_key).  The bugfix made it so that both APIs used the EC2
> > credentials (api_key, api_secret) .  This did mean that anyone that had
> > saved the 'bad' OpenStack credentials was unable to continue to use those
> > credentials.  I also overlooked exporting the updated credentials in
> novarc
> > (though a merge request was pending).
> > I actually thought originally that this was a straight-up bug, rather
> than a
> > design 'decision', so I should definitely have flagged it better.  Again,
> > sorry to those I impacted.
> > As things stand now, post-revert, this is probably a security flaw,
> because
> > the EC2 API does not treat the api_key as a secret.  The EC2 API can
> > (relatively) safely be run over non-SSL, because it uses signatures
> instead
> > of passing the shared secret directly.
> > This is also not very user-friendly.  Post-revert, an end-user must know
> > whether any particular cloud tool uses the EC2 API or the OpenStack API,
> so
> > that they can enter in the correct pair of credentials.  That doesn't
> seem
> > like a good idea; I think there should be one set of credentials.
> >
> > There is some discussion about the idea of having the api_key be
> > user-friendly.  I don't think it buys us anything, because the api_secret
> is
> > still going to be un-friendly, but I have no objection as long as it is
> does
> > in a way that does not break existing users of the EC2 API.
> > I propose that:
> >  (1) the OpenStack API and EC2 credentials should be the same as each
> other
> > (whatever they are) for the sake of our collective sanity and
> >  (2) we have to change the current configuration anyway for security
> > reasons.
> >  (3) We should not change the EC2 credentials, because we've shipped the
> EC2
> > API and our users have an expectation that we won't break them without
> good
> > reason, so
> >  (4) we must change the credentials for users of the (non-shipped)
> OpenStack
> > API.
> > Estimated user impact: I believe there are two people that will be
> affected,
> > and it will take them ~1 minute each, so total impact ~2 minutes.
> > The longer we delay fixing this, the more people we break and the bigger
> the
> > impact.  It seems that we have no choice but to do a
> > non-backwards-compatible authentication change, but I believe this is OK
> at
> > the moment because the OpenStack API is not yet stable/released - i.e. we
> > can still make fixes without worrying about backwards compatibility
> shims.
> > We're not in "The Old New Thing" land yet :-)
> >
> >
> > As an aside, I am very unhappy about the way this revert was pushed
> through
> > by Rackspace team-members, seemingly without much consideration of
> > alternatives.  Perhaps we should consider changing from needing two
> > core-approves, to needing one Rackspace core-approve and one
> non-Rackspace
> > core-approve.
> >
> > Justin
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
> > _______________________________________________ Mailing list:
> > https://launchpad.net/~openstack Post to : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack More help :
> > https://help.launchpad.net/ListHelp
> >
> > Confidentiality Notice: This e-mail message (including any attached or
> > embedded documents) is intended for the exclusive and confidential use of
> > the
> > individual or entity to which this message is addressed, and unless
> > otherwise
> > expressly indicated, is confidential and privileged information of
> > Rackspace.
> > Any dissemination, distribution or copying of the enclosed material is
> > prohibited.
> > If you receive this transmission in error, please notify us immediately
> by
> > e-mail
> > at abuse@xxxxxxxxxxxxx, and delete the original message.
> > Your cooperation is appreciated.
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
> >
>