openstack-qa-team team mailing list archive

[corvus@xxxxxxxxxxxx (James E. Blair)]

Hi,

In last weeks QA meeting, there was some talk about making improvements
around devstack-gate and tempest.

As I understand it, one of the problems is that things are changing in
other projects that are causing tests in the full (non-smoke) tempest
suite to fail, which prevents changes being merged to tempest.  This is
unique to tempest because of the asymmetrical gate currently in place,
where tempest is gated on the full suite but other projects are gated
only on the smoke tests.

There was talk about solving this by adding a second, non-gating, run of
tempest to run those jobs.  I'd like to keep the number of
devstack/tempest runs low, because of the heavy resource consumption --
especially when you consider that we've committed to doing cross-version
testing, which will start to exponentially increase the number of
devstack runs we do.

Here's what I'd like to propose instead:

1) Make the tempest gate symmetrical -- cross-project gating works best
when all the projects gate on the same tests so that one project can't
break another.

2) Add more tests to the set of tests that get run in the
devstack/tempest gate (currently the smoke tests).  It sounds like
several tests in the full suite really are breaking because of changes
in other projects -- those should probably be considered for promotion.

3) Modify devstack-gate to run tempest a second time (on the same host,
after completing the smoke tests), to run the rest of the suite, but
ignore the results of that for gating purposes.  This way the results
will be available for every change, without affecting gating or
consuming more virtual machines.  In other words, have devstack gate:

  run devstack
  run smoke tests
  save return code
  run full suite - smoke tests
  exit with saved return code

How does that sound?

-Jim