openstack team mailing list archive

[Vishvananda Ishaya <vishvananda@xxxxxxxxx>]

Hey Everyone,

We've branched for the Proposed D3 Milestone, and there are a few critical bugs that still need to land.

https://bugs.launchpad.net/nova/+bug/816236

This one in particular doesn't have a fixed proposed:

https://bugs.launchpad.net/nova/+bug/816236

The rest have fixes in the queue except for one that tr3buchet promises to propose in the morning. I think that we can have all of these in by the end of tomorrow if we put some attention on them.  I think it is best to delay the d3 miilestone release until all of these have landed.

You will notice that one of the bugs is about fixing some skipped tests.  Soren mentioned in the release meeting yesterday that he thought it was a horrible idea to allow the branch in that initially skipped the tests.  I'd like to explain how that happened, the positive and negative results, and my opinion on whether it should happen again.

There was a branch called multi_nic that was being worked on for quite some time that made changes to a large part of the code base.  This branch was difficult to maintain as changes to trunk often happen very quickly, and other groups and blueprints were depending on it.  The author of the branch was having trouble finding the necessary help for some of the hypervisors.

I decided that that the best approach was to push it in with some skipped tests right after the d2 milestone.  I was thinking it was likely that the lesser used hypervisors might break and that it would get more people helping fix tests and breakages if we got it into trunk and we could have everything fixed and stable again by d3.

Positives: we got a lot of fixes and dependent branches in, including ha networking and a network refactor for quantum

Negatives: the skipped tests didn't really get fixed.  We have less confidence in the stability of some parts of the code because of this. Also it is still possible that there are issues in other hypervisors that we haven't faced yet.

So was it worth it?  Hard to say.  My personal feeling is that we would be much farther behind in the networking code if we had waited to merge, but I don't know for sure.

Should we do it again? I would say no. Allowing skipping tests is a dangerous precedent and I don't feel it should be necessary. We got a little stuck on that branch and needed something, but I hope to find a cleaner way to get it unstuck in the future.

We do need a way to do large changes like this more efficiently. My thought is that we need to be willing to devote more resources to them.  There were a lot of people dependent on multi_nic, so rather than have everyone waiting for one person to "finish" it, a team of people should be assigned to get it up to snuff.  That means we have to do better at working together between teams and sharing some of our expertise when needed.  I was able to fix a lot of the tests fairly quickly, and I'm sure my help would have been just as useful prior to the merge.

If anyone else has ideas for doing these things better, I'd love to hear your thoughts.

Vish