openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On Thu, Feb 24, 2011 at 3:17 PM, Mark Washenberger
<mark.washenberger@xxxxxxxxxxxxx> wrote:
>> We need an unstable trunk:
>
> I could not possibly disagree more. Trunk is about releasability and stability. As developers we need a stable well-protected trunk so that we can actually work successfully in parallel on our own branches. My ideal for trunk is that when it comes time for tagging a release, the only work is making the tag, not a huge qa regression process.
>
>> I agree. I propose we always keep lp:nova (or lp:<project>) stable,
>> and instead create "trunks" like lp:nova/testing that all the
>> test/regression systems can be run against. This is pretty similar to
>> how we did things with Drizzle, a commit would bounce down the line,
>> finlly landing in lp:<project> when it was verified.
>
> I don't think we need anything beyond traditional development branches. I would like to see us work on tools to make a lot of integration testing available to individual developers in their branches. For the most hardware intensive integration tests, perhaps we should provide an automated system for developers to submit their branches for testing.

This is what we're working on, and what Justin is proposing, Mark.

Basically, in Drizzle-land, people propose a merge into trunk, Hudson
picks up that proposal, pulls the brnach into lp:drizzle/staging,
builds Drizzle on all supported platforms (>12 OS/distro combos), then
runs all automated regression testing against the proposed branch (can
take 3 or more hours).

We're proposing the same kind of automation for OpenStack.

-jay

>> We just need the tests. No matter how many stability levels of trunk
>> we have, without better tests we can't guarantee stability.
>
> Strongly agree.
>
> "Eric Day" <eday@xxxxxxxxxxxx> said:
>
>> I agree. I propose we always keep lp:nova (or lp:<project>) stable,
>> and instead create "trunks" like lp:nova/testing that all the
>> test/regression systems can be run against. This is pretty similar to
>> how we did things with Drizzle, a commit would bounce down the line,
>> finlly landing in lp:<project> when it was verified.
>>
>> -Eric
>>
>> On Thu, Feb 24, 2011 at 10:41:02AM -0800, Justin Santa Barbara wrote:
>>>    Sounds like we're actually in agreement, just disagreeing about the
>>>    topology of the branches!
>>>    My preferred topology is actually a series of increasingly stable trunks,
>>>    each of which has passed each level of the smoke tests (one platform,
>>>    multi platform, torture tests). *The nice thing about this is that then a
>>>    release can simply be seen as the next level up in the stability
>>>    hierarchy, with a manual pull. *I think Google Chrome has a particularly
>>>    sophisticated process which we could learn from:
>>>    http://techcrunch.com/2011/01/11/google-chrome-release-cycle-slideshow/
>>>    I still think we should start simple with two trunks - stable and
>>>    unstable. *But I'll be happy if with per-branch testing also - that just
>>>    seems a lot harder to me!
>>>    Justin
>>>
>>>    On Thu, Feb 24, 2011 at 10:24 AM, Trey Morris <trey.morris@xxxxxxxxxxxxx>
>>>    wrote:
>>>
>>>      Instead of an unstable trunk, i think code should just be better vetted
>>>      before it lands in the trunk. If the difference between trunk and your
>>>      proposed unstable trunk is a set of automated tests, then those tests
>>>      can just as easily be run on a LP branch before it gets into current
>>>      trunk. We just need the tests. No matter how many stability levels of
>>>      trunk we have, without better tests we can't guarantee stability.
>>>
>>>      On Thu, Feb 24, 2011 at 11:15 AM, Justin Santa Barbara
>>>      <justin@xxxxxxxxxxxx> wrote:
>>>
>>>        Hi Jay,
>>>        I couldn't agree more. *I had another bug come up yesterday on another
>>>        of my patches (I know - not a good day for me!) where I again broke
>>>        the OpenStack API by requiring the metadata attribute.
>>>        In this case, it was missed by the unit tests. *I believe I was always
>>>        passing metadata, so simply missed the real world case. *Here's the
>>>        bug report:
>>>        https://bugs.launchpad.net/nova/+bug/724143
>>>        This brings up a number of points though:
>>>        On testing
>>>         1. The bug reporter apparently knows how to program, but instead of
>>>            us getting a test case which we could immediately use, we got a
>>>            Ruby test case. *I think we should do whatever we can so that
>>>            people that are moderately comfortable with code also feel
>>>            comfortable submitting a failing test case that we can use. *I
>>>            think this means having some version of an API client, or even a
>>>            Reference-Implementation API client, in the source tree, and using
>>>            it.
>>>         2. It's not clear to me how we deal with unit test cases that are
>>>            failing - cases that represent a found bug that is not yet fixed.
>>>            *Maybe it should be submitted on a bugNNN branch, with failing
>>>            tests, and then whoever works on the bug can branch from it? *(Of
>>>            course, we'd be lucky if all bugs went like this, but it does
>>>            happen. *I often find myself fixing tangential issues, and I'd
>>>            like to know the 'right thing to do' is)
>>>         3. On this particular metadata bug, it was my fault. *I therefore
>>>            submitted a very rapid hotfix which just fixes the issue. *I then
>>>            coded up unit tests that use the OpenStack API, and did in fact
>>>            hit the issue naturally, and then included my fix which resolved
>>>            it. *I submitted the 'full patch' as a separate branch.
>>>         4. This 'full patch' branch includes unit tests that bring up the
>>>            OpenStack API and various services in-process, and runs tests just
>>>            like a user. *It would have caught the metadata issue.*
>>>         5. The hope is that we can reuse the same tests as smoke tests, by
>>>            simply tweaking them to work against real services instead of
>>>            bringing up in-process stub-services. *These could be (some of)
>>>            the smoke tests in Hudson.
>>>         6. I'd hope that we could have two levels of these smoke tests, one
>>>            that runs on a single configuration (e.g. KVM, OpenISCSI, Glance);
>>>            another that runs a matrix of configurations (and might take an
>>>            hour or more to run)
>>>         7. Ideally we'd have a torture test that would run overnight and be
>>>            randomized and try to find obscure bugs, even if issues found are
>>>            not necessarily repeatable in the way that non-randomized tests
>>>            are.
>>>        We need an unstable trunk:
>>>         1. In general, it seems that our end-users are using trunk for
>>>            unreleased functionality and treating it as if it were released.
>>>            *I don't think we should be encouraging that, because I know I'll
>>>            make more mistakes in future and some of them will make it past
>>>            the reviewers' defensive line into trunk; it's also simply not
>>>            realistic to require reviewers to review every combination - e.g.
>>>            how can a reviewer really review my HP SAN patch without an HP
>>>            SAN? *There will be issues in trunk, and if we have to revert them
>>>            rather than just fixing them it will slow us down. *The current
>>>            situation is bad for our users and bad for developers.*
>>>         2. One way we could keep everyone happy is by using our test suite to
>>>            auto-merge from an 'unstable trunk' into 'stable trunk', only once
>>>            code passes tests. *Commits would initially merge into 'unstable
>>>            trunk', and we would try to keep that branch moving forwards
>>>            rather than reverting things that go wrong. *Of course,
>>>            maintaining a good 'stable trunk' relies on having good tests, but
>>>            I think we're getting there. *It's also great incentive to write
>>>            good smoke tests.
>>>         3. Jay: I believe you've done this to great success on the Drizzle
>>>            project?
>>>        Justin
>>>
>>>        On Thu, Feb 24, 2011 at 6:13 AM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:
>>>
>>>          Hi all,
>>>
>>>          I'd like to bring up an alternate reason why it was approved and
>>>          subsequently reverted.
>>>
>>>          The test cases for the OpenStack API (and much of the EC2 API)
>>>          assume
>>>          way too many things and mock out too many things. In addition, since
>>>          there are zero smoketests for the OpenStack API, there were no
>>>          functional tests that would have *immediately* highlighted this
>>>          problem (and many other recent EC2 vs OS API problems).
>>>
>>>          In other words, sure, we should revert the patch to "fix things",
>>>          however the priority should *not* be on refactoring the auth API or
>>>          the way the auth layer in Nova is handled. The priority should be on
>>>          writing a smoketest for the OpenStack API so that we can link it
>>>          into
>>>          Hudson and these types of issues can be automatically caught.
>>>          -jay
>>>          On Wed, Feb 23, 2011 at 10:03 PM, Paul Voccio
>>>          <paul.voccio@xxxxxxxxxxxxx> wrote:
>>>          > Justin,
>>>          > I think you hit upon the reason of why I think it was approved and
>>>          reverted.
>>>          > Because it hadn't been talked about in a blueprint or a mail sent
>>>          to the
>>>          > list (I think I'm up to date on the threads) and a patch landed
>>>          means other
>>>          > alternatives weren't considered before pushing it through to begin
>>>          with. I
>>>          > think we're all open to talking about how to better the auth
>>>          system and make
>>>          > improvements. Dragon has already discussed some alternatives and
>>>          suggestions
>>>          > on the BP page below. I think this is the right way to continue
>>>          the dialog
>>>          > and we all can agree on a good way forward.
>>>          > I'm confident we can figure it out.
>>>          > If I missed a conversation, my apologies.
>>>          > pvo
>>>          > From: Vishvananda Ishaya <vishvananda@xxxxxxxxx>
>>>          > Date: Wed, 23 Feb 2011 18:19:41 -0800
>>>          > To: Justin Santa Barbara <justin@xxxxxxxxxxxx>
>>>          > Cc: <openstack@xxxxxxxxxxxxxxxxxxx>
>>>          > Subject: Re: [Openstack] Should the OpenStack API re-use the EC2
>>>          > credentials?
>>>          >
>>>          > Hey Justin,
>>>          > Does it make any difference that the way the auth is
>>>          (theoretically)
>>>          > supposed to work with the os api is that the user gets an auth
>>>          token from an
>>>          > external auth server and then uses username / authtoken to
>>>          actually contact
>>>          > the api? *I think it is just faked out right now to use the
>>>          access_key
>>>          > instead of doing external auth, but I think the reason it works
>>>          like it does
>>>          > is because the plan was to switch to external auth eventually.
>>>          > Vish
>>>          > On Feb 23, 2011, at 5:56 PM, Justin Santa Barbara wrote:
>>>          >
>>>          > I previously fixed OpenStack authentication so it would use the
>>>          same
>>>          > credentials as EC2. *This bugfix was just reverted, because it
>>>          caused
>>>          > OpenStack API users to have to enter in different credentials
>>>          (sorry!), but
>>>          > primarily because it hadn't been*discussed*on the mailing list.
>>>          *So here
>>>          > goes!
>>>          > Here's a
>>>          >
>>>          blueprint:*https://blueprints.launchpad.net/nova/+spec/authentication-consistency
>>>          > Here's an overview of the problem:
>>>          > EC2 uses an (api_key, api_secret) pair. *Post-revert, OpenStack
>>>          uses the
>>>          > api_key(!) as the password, but a different value entirely as the
>>>          username:
>>>          > (username, api_key). *The bugfix made it so that both APIs used
>>>          the EC2
>>>          > credentials (api_key, api_secret) . *This did mean that anyone
>>>          that had
>>>          > saved the 'bad' OpenStack credentials was unable to continue to
>>>          use those
>>>          > credentials. *I also overlooked exporting the updated credentials
>>>          in novarc
>>>          > (though a merge request was pending).
>>>          > I actually thought originally that this was a straight-up bug,
>>>          rather than a
>>>          > design 'decision', so I should definitely have flagged it better.
>>>          *Again,
>>>          > sorry to those I impacted.
>>>          > As things stand now, post-revert, this is probably a security
>>>          flaw, because
>>>          > the EC2 API does not treat the api_key as a secret. *The EC2 API
>>>          can
>>>          > (relatively) safely be run over non-SSL, because it uses
>>>          signatures instead
>>>          > of passing the shared secret directly.
>>>          > This is also not very user-friendly. *Post-revert, an end-user
>>>          must know
>>>          > whether any particular cloud tool uses the EC2 API or the
>>>          OpenStack API, so
>>>          > that they can enter in the correct pair of credentials. *That
>>>          doesn't seem
>>>          > like a good idea; I think there should be one set of credentials.
>>>          >
>>>          > There is some discussion about the idea of having the api_key be
>>>          > user-friendly. *I don't think it buys us anything, because the
>>>          api_secret is
>>>          > still going to be un-friendly, but I have no objection as long as
>>>          it is does
>>>          > in a way that does not break existing users of the EC2 API.
>>>          > I propose that:
>>>          > *(1) the OpenStack API and EC2 credentials should be the same as
>>>          each other
>>>          > (whatever they are) for the sake of our collective sanity and
>>>          > *(2) we have to change the current configuration anyway for
>>>          security
>>>          > reasons.
>>>          > *(3) We should not change the EC2 credentials, because we've
>>>          shipped the EC2
>>>          > API and our users have an expectation that we won't break them
>>>          without good
>>>          > reason, so
>>>          > *(4) we must change the credentials for users of the (non-shipped)
>>>          OpenStack
>>>          > API.
>>>          > Estimated user impact: I believe there are two people that will be
>>>          affected,
>>>          > and it will take them ~1 minute each, so total impact ~2 minutes.
>>>          > The longer we delay fixing this, the more people we break and the
>>>          bigger the
>>>          > impact. *It seems that we have no choice but to do a
>>>          > non-backwards-compatible authentication change, but I believe this
>>>          is OK at
>>>          > the moment because the OpenStack API is not yet stable/released -
>>>          i.e. we
>>>          > can still make fixes without worrying about backwards
>>>          compatibility shims.
>>>          > We're not in "The Old New Thing" land yet :-)
>>>          >
>>>          >
>>>          > As an aside, I am very unhappy about the way this revert was
>>>          pushed through
>>>          > by Rackspace team-members, seemingly without much consideration of
>>>          > alternatives. *Perhaps we should consider changing from needing
>>>          two
>>>          > core-approves, to needing one Rackspace core-approve and one
>>>          non-Rackspace
>>>          > core-approve.
>>>          >
>>>          > Justin
>>>          >
>>>          >
>>>          > _______________________________________________
>>>          > Mailing list: https://launchpad.net/~openstack
>>>          > Post to ****: openstack@xxxxxxxxxxxxxxxxxxx
>>>          > Unsubscribe : https://launchpad.net/~openstack
>>>          > More help **: https://help.launchpad.net/ListHelp
>>>          >
>>>          > _______________________________________________ Mailing list:
>>>          > https://launchpad.net/~openstack Post to :
>>>          openstack@xxxxxxxxxxxxxxxxxxx
>>>          > Unsubscribe : https://launchpad.net/~openstack More help :
>>>          > https://help.launchpad.net/ListHelp
>>>          >
>>>          > Confidentiality Notice: This e-mail message (including any
>>>          attached or
>>>          > embedded documents) is intended for the exclusive and confidential
>>>          use of
>>>          > the
>>>          > individual or entity to which this message is addressed, and
>>>          unless
>>>          > otherwise
>>>          > expressly indicated, is confidential and privileged information of
>>>          > Rackspace.
>>>          > Any dissemination, distribution or copying of the enclosed
>>>          material is
>>>          > prohibited.
>>>          > If you receive this transmission in error, please notify us
>>>          immediately by
>>>          > e-mail
>>>          > at abuse@xxxxxxxxxxxxx, and delete the original message.
>>>          > Your cooperation is appreciated.
>>>          >
>>>          > _______________________________________________
>>>          > Mailing list: https://launchpad.net/~openstack
>>>          > Post to * * : openstack@xxxxxxxxxxxxxxxxxxx
>>>          > Unsubscribe : https://launchpad.net/~openstack
>>>          > More help * : https://help.launchpad.net/ListHelp
>>>          >
>>>          >
>>>
>>>        _______________________________________________
>>>        Mailing list: https://launchpad.net/~openstack
>>>        Post to * * : openstack@xxxxxxxxxxxxxxxxxxx
>>>        Unsubscribe : https://launchpad.net/~openstack
>>>        More help * : https://help.launchpad.net/ListHelp
>>>
>>>  Confidentiality Notice: This e-mail message (including any attached or
>>>  embedded documents) is intended for the exclusive and confidential use of the
>>>  individual or entity to which this message is addressed, and unless otherwise
>>>  expressly indicated, is confidential and privileged information of Rackspace.
>>>  Any dissemination, distribution or copying of the enclosed material is
>>> prohibited.
>>>  If you receive this transmission in error, please notify us immediately by
>>> e-mail
>>>  at abuse@xxxxxxxxxxxxx, and delete the original message.
>>>  Your cooperation is appreciated.
>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>