openstack-qa-team team mailing list archive

[Daryl Walleck <daryl.walleck@xxxxxxxxxxxxx>]

So it sounds like we have two problems to solve: an optimization of resources, and the configuration to enable parallel execution. As opposed to making one large merge prop for enable this for the entire suite at once, I'm gong to begin submitting patches for groupings of tests individually so progress can be gauged. This only requires adding class vars, and will not affect the tests in any way when running serially.

Daryl

On May 31, 2012, at 9:47 AM, David Kranz wrote:

> On 5/30/2012 3:56 PM, Jay Pipes wrote:
>> > With ubuntu and several other full Linux OSes, the best I've seen
>>> is 15 minute smoke tests and 40-ish minute full regression runs, which
>>> in my case is within my pain threshold. In terms of actions, we've paid
>>> the majority of the time price for testing scenarios. The additional
>>> tests I'm adding (such as the SSH code) don't add significant time to
>>> tests, but keep adding value. The same can be said of checking for
>>> events that should be generated on server actions, and other side
>>> effects that take minimal time to run. As we start to move in that
>>> direction, test classes will become more streamlined/optimized. The test
>>> class I added with the initial SSH tests is a good example of this, and
>>> I'd be more than glad to share some other examples of how we can do more
>>> with less. Eventually these types of optimizations can grow into our
>>> coding standards.
>>>> 
>>>>> What's everyone else's feelings on smoke test pain point times? I think
>>>>> getting a sense of that will make any further decisions a bit more clear.
>>>> 
>>>> Regardless of what length of time we say, I still think we need to get
>>>> parallel execution working properly (with the existing --processes
>>>> option in nosetest's multiprocessing plugin).
>>> 
>>> Agreed. I think like we talked about at the conference, I think this
>>> discussion and the overall discussion on optimization would probably be
>>> better handled in a Skype meeting where we can all discuss this in real
>>> time. I'm free every day this week save Friday.
>> 
>> Cool, I will be at the meeting tomorrow, but otherwise not really available much this week (have my folks in town). Next week, though, I'm all yours. :) Let's set up a time to Skype when we're at the IRC meeting tomorrow.
>> 
>> Best,
>> -jay
>> 
> Though I agree with the sentiment here, I think we also need to pay more attention to the relationship between tempest and unit tests in the projects. There are currently >60 calls to create_server in the tempest tests.  I didn't check to see if any were inside loops. This is just going to be slow, and get worse. Perhaps we should try to avoid creating servers when possible. The main difference between the nova unit tests and tempest seems to be that tempest really uses the hypervisor and is more anal about combinations and return codes and api details. I think there are some tempest tests with multiple variants of parameters but where the difference in  code path is already covered by the unit tests, or should be.
> 
> Perhaps tempest should only have one case for such families of calls but make sure that the unit tests are an adequate substitute.
> 
> -David
> 
> 
> -- 
> Mailing list: https://launchpad.net/~openstack-qa-team
> Post to     : openstack-qa-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack-qa-team
> More help   : https://help.launchpad.net/ListHelp