openstack-qa-team team mailing list archive

[Daryl Walleck <daryl.walleck@xxxxxxxxxxxxx>]

We also have a different problem with running tests in parallel now. None of the newly designed basic/advanced ops test can be run in parallel given their dependency between tests. The only way I can think of to proceed would be to rework the nose plugin to run test classes in parallel, but tests within the classes in serial. I'll poke around this weekend to see what's possible.

Daryl

On Sep 6, 2012, at 1:15 PM, "David Kranz" <david.kranz@xxxxxxxxxx> wrote:

> Jim, we discussed this at the QA meeting and would appreciate your going ahead with this.
> We are going to raise the current problems at the nova meeting today and see if we can get more attention to the
> breakages whose tests in tempest  have been commented out.  We hope that (3), because it directly ties a failure to a
> particular commit, will make it easier for core team developers to fix problems quickly when they
> arise.
> 
> We would really like to run full tempest on all projects. Running tempest in parallel is blocked due to
> a problem with the nose multiprocessing plugin. No one on the qa team seems to know how to fix that.
> If you have any ideas I think Jay is the closest to that issue.
> 
> -David
> 
> On 9/5/2012 9:03 PM, James E. Blair wrote:
>> Hi,
>> 
>> In last weeks QA meeting, there was some talk about making improvements
>> around devstack-gate and tempest.
>> 
>> As I understand it, one of the problems is that things are changing in
>> other projects that are causing tests in the full (non-smoke) tempest
>> suite to fail, which prevents changes being merged to tempest.  This is
>> unique to tempest because of the asymmetrical gate currently in place,
>> where tempest is gated on the full suite but other projects are gated
>> only on the smoke tests.
>> 
>> There was talk about solving this by adding a second, non-gating, run of
>> tempest to run those jobs.  I'd like to keep the number of
>> devstack/tempest runs low, because of the heavy resource consumption --
>> especially when you consider that we've committed to doing cross-version
>> testing, which will start to exponentially increase the number of
>> devstack runs we do.
>> 
>> Here's what I'd like to propose instead:
>> 
>> 1) Make the tempest gate symmetrical -- cross-project gating works best
>> when all the projects gate on the same tests so that one project can't
>> break another.
>> 
>> 2) Add more tests to the set of tests that get run in the
>> devstack/tempest gate (currently the smoke tests).  It sounds like
>> several tests in the full suite really are breaking because of changes
>> in other projects -- those should probably be considered for promotion.
>> 
>> 3) Modify devstack-gate to run tempest a second time (on the same host,
>> after completing the smoke tests), to run the rest of the suite, but
>> ignore the results of that for gating purposes.  This way the results
>> will be available for every change, without affecting gating or
>> consuming more virtual machines.  In other words, have devstack gate:
>> 
>>   run devstack
>>   run smoke tests
>>   save return code
>>   run full suite - smoke tests
>>   exit with saved return code
>> 
>> How does that sound?
>> 
>> -Jim
>> 
> 
> 
> -- 
> Mailing list: https://launchpad.net/~openstack-qa-team
> Post to     : openstack-qa-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack-qa-team
> More help   : https://help.launchpad.net/ListHelp