launchpad-dev team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

On Wednesday 09 June 2010 13:33:59 Aaron Bentley wrote:
> Yes, it has.  It was live and enabled from at least May 20 to May 31.
> Please see my emails from that period.  But during that period, build
> attempts would  fail due to the closed ports or incompatibility between
> bzr-builder and buildds.

Fair enough - I think we're basically talking about the same thing but in 
different ways here.  Anyway ...

> > How were Paul's changes QA tested?
> 
> Paul ran the full test suite.  He didn't anticipate that his change
> would require testing on dogfood.  Since staging doesn't have builders,
> he didn't have a convenient way to QA.

Ok so I guess we've got to the bottom of why it failed.

When we make build-related changes in Soyuz we always put some jobs through 
the dogfood build farm since the test suite is totally inadequate for picking 
up this sort of failure.

I have a few recommendations based on this analysis:

 1. I mentioned this to Tim earlier, but I'd appreciate you folks' help on 
making the test suite better.  It currently has some xmlrpc tests using a mock 
slave, but it's really quite hard to use the real thing which is why we've 
never done it and relied on dogfood QA instead.  Some fresh ideas would be 
good!

 2. Never, ever skip QA on a buildfarm-related change (even if we did have 
full test suite coverage).

 3. I'm going to compile a wiki page of basic QA tasks that must pass when 
testing build farm jobs. Your input on this would be invaluable.

We're also going to be putting a builder and a buildd-manager on staging soon.  
Watch this space.

Cheers
J