← Back to team overview

openstack-qa-team team mailing list archive

Re: Tempest Gating

 

Jim, we discussed this at the QA meeting and would appreciate your going ahead with this. We are going to raise the current problems at the nova meeting today and see if we can get more attention to the breakages whose tests in tempest have been commented out. We hope that (3), because it directly ties a failure to a particular commit, will make it easier for core team developers to fix problems quickly when they
arise.

We would really like to run full tempest on all projects. Running tempest in parallel is blocked due to a problem with the nose multiprocessing plugin. No one on the qa team seems to know how to fix that.
If you have any ideas I think Jay is the closest to that issue.

 -David

On 9/5/2012 9:03 PM, James E. Blair wrote:
Hi,

In last weeks QA meeting, there was some talk about making improvements
around devstack-gate and tempest.

As I understand it, one of the problems is that things are changing in
other projects that are causing tests in the full (non-smoke) tempest
suite to fail, which prevents changes being merged to tempest.  This is
unique to tempest because of the asymmetrical gate currently in place,
where tempest is gated on the full suite but other projects are gated
only on the smoke tests.

There was talk about solving this by adding a second, non-gating, run of
tempest to run those jobs.  I'd like to keep the number of
devstack/tempest runs low, because of the heavy resource consumption --
especially when you consider that we've committed to doing cross-version
testing, which will start to exponentially increase the number of
devstack runs we do.

Here's what I'd like to propose instead:

1) Make the tempest gate symmetrical -- cross-project gating works best
when all the projects gate on the same tests so that one project can't
break another.

2) Add more tests to the set of tests that get run in the
devstack/tempest gate (currently the smoke tests).  It sounds like
several tests in the full suite really are breaking because of changes
in other projects -- those should probably be considered for promotion.

3) Modify devstack-gate to run tempest a second time (on the same host,
after completing the smoke tests), to run the rest of the suite, but
ignore the results of that for gating purposes.  This way the results
will be available for every change, without affecting gating or
consuming more virtual machines.  In other words, have devstack gate:

   run devstack
   run smoke tests
   save return code
   run full suite - smoke tests
   exit with saved return code

How does that sound?

-Jim




Follow ups

References