openstack team mailing list archive

[Trey Morris <trey.morris@xxxxxxxxxxxxx>]

I'm fairly certain that in this particular case, without support from the
hypervisor and api lieutenants, and a testing guru or two, I would still be
trying to fit all the pieces together. Merging with broken pieces in this
case was necessary.

Some more thoughts to fuel discussion...

1) cooperation:
Though time consuming and resulting in larger merges, fiat cooperation
between necessary groups to make changes without breaking individual
pieces.

2) shims:
Sandy would say we could use shims here, wait for the other parts to be
merged, then remove the shims in such a way that nothing is ever broken. I
think a procedure like this works in the vast majority of cases, and makes
for smaller merges with trunk. Certainly we don't want shims in place any
more than we want skipped tests, so how do we get the pieces updated? Who is
going to do the updating?

3) skipped tests:
Very visible. Everyone sees them. But very bad habits can be formed around
using them.

4) dropped support:
We've decided to add feature X to Nova. Developer A determines adding X
requires nontrivial changes to hypervisors Q W E and R (and yes, soren,
their tests). Developer A, being familiar with Q, updates Q (and tests) to
work with Nova+X; however, W E and R are still broken. If requests for
option 1 above fail and cooperation doesn't seem to happen and things drag
out, we should remove the shims, merge, and drop support for W E and R from
Nova until updated code (with tests) for W E and R is proposed for merging
into trunk.




On Wed, Jul 27, 2011 at 3:11 AM, Vishvananda Ishaya
<vishvananda@xxxxxxxxx>wrote:

> Hey Everyone,
>
> We've branched for the Proposed D3 Milestone, and there are a few critical
> bugs that still need to land.
>
> https://bugs.launchpad.net/nova/+bug/816236
>
> This one in particular doesn't have a fixed proposed:
>
> https://bugs.launchpad.net/nova/+bug/816236
>
> The rest have fixes in the queue except for one that tr3buchet promises to
> propose in the morning. I think that we can have all of these in by the end
> of tomorrow if we put some attention on them.  I think it is best to delay
> the d3 miilestone release until all of these have landed.
>
> You will notice that one of the bugs is about fixing some skipped tests.
>  Soren mentioned in the release meeting yesterday that he thought it was a
> horrible idea to allow the branch in that initially skipped the tests.  I'd
> like to explain how that happened, the positive and negative results, and my
> opinion on whether it should happen again.
>
> There was a branch called multi_nic that was being worked on for quite some
> time that made changes to a large part of the code base.  This branch was
> difficult to maintain as changes to trunk often happen very quickly, and
> other groups and blueprints were depending on it.  The author of the branch
> was having trouble finding the necessary help for some of the hypervisors.
>
> I decided that that the best approach was to push it in with some skipped
> tests right after the d2 milestone.  I was thinking it was likely that the
> lesser used hypervisors might break and that it would get more people
> helping fix tests and breakages if we got it into trunk and we could have
> everything fixed and stable again by d3.
>
> Positives: we got a lot of fixes and dependent branches in, including ha
> networking and a network refactor for quantum
>
> Negatives: the skipped tests didn't really get fixed.  We have less
> confidence in the stability of some parts of the code because of this. Also
> it is still possible that there are issues in other hypervisors that we
> haven't faced yet.
>
> So was it worth it?  Hard to say.  My personal feeling is that we would be
> much farther behind in the networking code if we had waited to merge, but I
> don't know for sure.
>
> Should we do it again? I would say no. Allowing skipping tests is a
> dangerous precedent and I don't feel it should be necessary. We got a little
> stuck on that branch and needed something, but I hope to find a cleaner way
> to get it unstuck in the future.
>
> We do need a way to do large changes like this more efficiently. My thought
> is that we need to be willing to devote more resources to them.  There were
> a lot of people dependent on multi_nic, so rather than have everyone waiting
> for one person to "finish" it, a team of people should be assigned to get it
> up to snuff.  That means we have to do better at working together between
> teams and sharing some of our expertise when needed.  I was able to fix a
> lot of the tests fairly quickly, and I'm sure my help would have been just
> as useful prior to the merge.
>
> If anyone else has ideas for doing these things better, I'd love to hear
> your thoughts.
>
> Vish
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>