← Back to team overview

openstack team mailing list archive

Re: Should the OpenStack API re-use the EC2 credentials?

 

Sounds like we're actually in agreement, just disagreeing about the topology
of the branches!

My preferred topology is actually a series of increasingly stable trunks,
each of which has passed each level of the smoke tests (one platform, multi
platform, torture tests).  The nice thing about this is that then a release
can simply be seen as the next level up in the stability hierarchy, with a
manual pull.  I think Google Chrome has a particularly sophisticated process
which we could learn from:
http://techcrunch.com/2011/01/11/google-chrome-release-cycle-slideshow/

I still think we should start simple with two trunks - stable and unstable.
 But I'll be happy if with per-branch testing also - that just seems a lot
harder to me!
<http://techcrunch.com/2011/01/11/google-chrome-release-cycle-slideshow/>

Justin





On Thu, Feb 24, 2011 at 10:24 AM, Trey Morris <trey.morris@xxxxxxxxxxxxx>wrote:

> Instead of an unstable trunk, i think code should just be better vetted
> before it lands in the trunk. If the difference between trunk and your
> proposed unstable trunk is a set of automated tests, then those tests can
> just as easily be run on a LP branch before it gets into current trunk. We
> just need the tests. No matter how many stability levels of trunk we have,
> without better tests we can't guarantee stability.
>
> On Thu, Feb 24, 2011 at 11:15 AM, Justin Santa Barbara <
> justin@xxxxxxxxxxxx> wrote:
>
>> Hi Jay,
>>
>> I couldn't agree more.  I had another bug come up yesterday on another of
>> my patches (I know - not a good day for me!) where I again broke the
>> OpenStack API by requiring the metadata attribute.
>>
>> In this case, it was missed by the unit tests.  I believe I was always
>> passing metadata, so simply missed the real world case.  Here's the bug
>> report:
>> https://bugs.launchpad.net/nova/+bug/724143
>>
>> This brings up a number of points though:
>>
>> On testing
>>
>>
>>    1. The bug reporter apparently knows how to program, but instead of us
>>    getting a test case which we could immediately use, we got a Ruby test case.
>>     I think we should do whatever we can so that people that are moderately
>>    comfortable with code also feel comfortable submitting a failing test case
>>    that we can use.  I think this means having some version of an API client,
>>    or even a Reference-Implementation API client, in the source tree, and using
>>    it.
>>    2. It's not clear to me how we deal with unit test cases that are
>>    failing - cases that represent a found bug that is not yet fixed.  Maybe it
>>    should be submitted on a bugNNN branch, with failing tests, and then whoever
>>    works on the bug can branch from it?  (Of course, we'd be lucky if all bugs
>>    went like this, but it does happen.  I often find myself fixing tangential
>>    issues, and I'd like to know the 'right thing to do' is)
>>    3. On this particular metadata bug, it was my fault.  I therefore
>>    submitted a very rapid hotfix which just fixes the issue.  I then coded up
>>    unit tests that use the OpenStack API, and did in fact hit the issue
>>    naturally, and then included my fix which resolved it.  I submitted the
>>    'full patch' as a separate branch.
>>    4. This 'full patch' branch includes unit tests that bring up the
>>    OpenStack API and various services in-process, and runs tests just like a
>>    user.  It would have caught the metadata issue.
>>    5. The hope is that we can reuse the same tests as smoke tests, by
>>    simply tweaking them to work against real services instead of bringing up
>>    in-process stub-services.  These could be (some of) the smoke tests in
>>    Hudson.
>>    6. I'd hope that we could have two levels of these smoke tests, one
>>    that runs on a single configuration (e.g. KVM, OpenISCSI, Glance); another
>>    that runs a matrix of configurations (and might take an hour or more to run)
>>    7. Ideally we'd have a torture test that would run overnight and be
>>    randomized and try to find obscure bugs, even if issues found are not
>>    necessarily repeatable in the way that non-randomized tests are.
>>
>>
>> We need an unstable trunk:
>>
>>
>>    1. In general, it seems that our end-users are using trunk for
>>    unreleased functionality and treating it as if it were released.  I don't
>>    think we should be encouraging that, because I know I'll make more mistakes
>>    in future and some of them will make it past the reviewers' defensive line
>>    into trunk; it's also simply not realistic to require reviewers to review
>>    every combination - e.g. how can a reviewer really review my HP SAN patch
>>    without an HP SAN?  There will be issues in trunk, and if we have to revert
>>    them rather than just fixing them it will slow us down.  The current
>>    situation is bad for our users and bad for developers.
>>    2. One way we could keep everyone happy is by using our test suite to
>>    auto-merge from an 'unstable trunk' into 'stable trunk', only once code
>>    passes tests.  Commits would initially merge into 'unstable trunk', and we
>>    would try to keep that branch moving forwards rather than reverting things
>>    that go wrong.  Of course, maintaining a good 'stable trunk' relies on
>>    having good tests, but I think we're getting there.  It's also great
>>    incentive to write good smoke tests.
>>    3. Jay: I believe you've done this to great success on the Drizzle
>>    project?
>>
>>
>> Justin
>>
>>
>>
>> On Thu, Feb 24, 2011 at 6:13 AM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to bring up an alternate reason why it was approved and
>>> subsequently reverted.
>>>
>>> The test cases for the OpenStack API (and much of the EC2 API) assume
>>> way too many things and mock out too many things. In addition, since
>>> there are zero smoketests for the OpenStack API, there were no
>>> functional tests that would have *immediately* highlighted this
>>> problem (and many other recent EC2 vs OS API problems).
>>>
>>> In other words, sure, we should revert the patch to "fix things",
>>> however the priority should *not* be on refactoring the auth API or
>>> the way the auth layer in Nova is handled. The priority should be on
>>> writing a smoketest for the OpenStack API so that we can link it into
>>> Hudson and these types of issues can be automatically caught.
>>>
>>> -jay
>>>
>>> On Wed, Feb 23, 2011 at 10:03 PM, Paul Voccio <paul.voccio@xxxxxxxxxxxxx>
>>> wrote:
>>> > Justin,
>>> > I think you hit upon the reason of why I think it was approved and
>>> reverted.
>>> > Because it hadn't been talked about in a blueprint or a mail sent to
>>> the
>>> > list (I think I'm up to date on the threads) and a patch landed means
>>> other
>>> > alternatives weren't considered before pushing it through to begin
>>> with. I
>>> > think we're all open to talking about how to better the auth system and
>>> make
>>> > improvements. Dragon has already discussed some alternatives and
>>> suggestions
>>> > on the BP page below. I think this is the right way to continue the
>>> dialog
>>> > and we all can agree on a good way forward.
>>> > I'm confident we can figure it out.
>>> > If I missed a conversation, my apologies.
>>> > pvo
>>> > From: Vishvananda Ishaya <vishvananda@xxxxxxxxx>
>>> > Date: Wed, 23 Feb 2011 18:19:41 -0800
>>> > To: Justin Santa Barbara <justin@xxxxxxxxxxxx>
>>> > Cc: <openstack@xxxxxxxxxxxxxxxxxxx>
>>> > Subject: Re: [Openstack] Should the OpenStack API re-use the EC2
>>> > credentials?
>>> >
>>> > Hey Justin,
>>> > Does it make any difference that the way the auth is (theoretically)
>>> > supposed to work with the os api is that the user gets an auth token
>>> from an
>>> > external auth server and then uses username / authtoken to actually
>>> contact
>>> > the api?  I think it is just faked out right now to use the access_key
>>> > instead of doing external auth, but I think the reason it works like it
>>> does
>>> > is because the plan was to switch to external auth eventually.
>>> > Vish
>>> > On Feb 23, 2011, at 5:56 PM, Justin Santa Barbara wrote:
>>> >
>>> > I previously fixed OpenStack authentication so it would use the same
>>> > credentials as EC2.  This bugfix was just reverted, because it caused
>>> > OpenStack API users to have to enter in different credentials (sorry!),
>>> but
>>> > primarily because it hadn't been discussed on the mailing list.  So
>>> here
>>> > goes!
>>> > Here's a
>>> > blueprint:
>>> https://blueprints.launchpad.net/nova/+spec/authentication-consistency
>>> > Here's an overview of the problem:
>>> > EC2 uses an (api_key, api_secret) pair.  Post-revert, OpenStack uses
>>> the
>>> > api_key(!) as the password, but a different value entirely as the
>>> username:
>>> > (username, api_key).  The bugfix made it so that both APIs used the EC2
>>> > credentials (api_key, api_secret) .  This did mean that anyone that had
>>> > saved the 'bad' OpenStack credentials was unable to continue to use
>>> those
>>> > credentials.  I also overlooked exporting the updated credentials in
>>> novarc
>>> > (though a merge request was pending).
>>> > I actually thought originally that this was a straight-up bug, rather
>>> than a
>>> > design 'decision', so I should definitely have flagged it better.
>>>  Again,
>>> > sorry to those I impacted.
>>> > As things stand now, post-revert, this is probably a security flaw,
>>> because
>>> > the EC2 API does not treat the api_key as a secret.  The EC2 API can
>>> > (relatively) safely be run over non-SSL, because it uses signatures
>>> instead
>>> > of passing the shared secret directly.
>>> > This is also not very user-friendly.  Post-revert, an end-user must
>>> know
>>> > whether any particular cloud tool uses the EC2 API or the OpenStack
>>> API, so
>>> > that they can enter in the correct pair of credentials.  That doesn't
>>> seem
>>> > like a good idea; I think there should be one set of credentials.
>>> >
>>> > There is some discussion about the idea of having the api_key be
>>> > user-friendly.  I don't think it buys us anything, because the
>>> api_secret is
>>> > still going to be un-friendly, but I have no objection as long as it is
>>> does
>>> > in a way that does not break existing users of the EC2 API.
>>> > I propose that:
>>> >  (1) the OpenStack API and EC2 credentials should be the same as each
>>> other
>>> > (whatever they are) for the sake of our collective sanity and
>>> >  (2) we have to change the current configuration anyway for security
>>> > reasons.
>>> >  (3) We should not change the EC2 credentials, because we've shipped
>>> the EC2
>>> > API and our users have an expectation that we won't break them without
>>> good
>>> > reason, so
>>> >  (4) we must change the credentials for users of the (non-shipped)
>>> OpenStack
>>> > API.
>>> > Estimated user impact: I believe there are two people that will be
>>> affected,
>>> > and it will take them ~1 minute each, so total impact ~2 minutes.
>>> > The longer we delay fixing this, the more people we break and the
>>> bigger the
>>> > impact.  It seems that we have no choice but to do a
>>> > non-backwards-compatible authentication change, but I believe this is
>>> OK at
>>> > the moment because the OpenStack API is not yet stable/released - i.e.
>>> we
>>> > can still make fixes without worrying about backwards compatibility
>>> shims.
>>> > We're not in "The Old New Thing" land yet :-)
>>> >
>>> >
>>> > As an aside, I am very unhappy about the way this revert was pushed
>>> through
>>> > by Rackspace team-members, seemingly without much consideration of
>>> > alternatives.  Perhaps we should consider changing from needing two
>>> > core-approves, to needing one Rackspace core-approve and one
>>> non-Rackspace
>>> > core-approve.
>>> >
>>> > Justin
>>> >
>>> >
>>> > _______________________________________________
>>> > Mailing list: https://launchpad.net/~openstack
>>> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> > Unsubscribe : https://launchpad.net/~openstack
>>> > More help   : https://help.launchpad.net/ListHelp
>>> >
>>> > _______________________________________________ Mailing list:
>>> > https://launchpad.net/~openstack Post to :
>>> openstack@xxxxxxxxxxxxxxxxxxx
>>> > Unsubscribe : https://launchpad.net/~openstack More help :
>>> > https://help.launchpad.net/ListHelp
>>> >
>>> > Confidentiality Notice: This e-mail message (including any attached or
>>> > embedded documents) is intended for the exclusive and confidential use
>>> of
>>> > the
>>> > individual or entity to which this message is addressed, and unless
>>> > otherwise
>>> > expressly indicated, is confidential and privileged information of
>>> > Rackspace.
>>> > Any dissemination, distribution or copying of the enclosed material is
>>> > prohibited.
>>> > If you receive this transmission in error, please notify us immediately
>>> by
>>> > e-mail
>>> > at abuse@xxxxxxxxxxxxx, and delete the original message.
>>> > Your cooperation is appreciated.
>>> >
>>> > _______________________________________________
>>> > Mailing list: https://launchpad.net/~openstack
>>> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> > Unsubscribe : https://launchpad.net/~openstack
>>> > More help   : https://help.launchpad.net/ListHelp
>>> >
>>> >
>>>
>>
>>
>> _______________________________________________
>>
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is prohibited.
> If you receive this transmission in error, please notify us immediately by e-mail
> at abuse@xxxxxxxxxxxxx, and delete the original message.
> Your cooperation is appreciated.
>
>

Follow ups

References