openstack-qa-team team mailing list archive

[Daryl Walleck <daryl.walleck@xxxxxxxxxxxxx>]

Hi David,

The per test fixtures are there for two reasons. One - stability. If we were to share one server among the tests, any of the previous tests tainted it or left it in a bad state, the rest of the suite would fail for unclear reasons. The second is for parallel execution. We cannot perform all those actions on a single server at once. 

I certainly agree that we should optimize Tempest as best we can. Tempest is an black-box, integration test suite. In my view, its purpose is to throughly, from end to end, test the integration of OpenStack components. For quick testings, we have unit and component level integration tests.  If those suites are lacking, then more effort should be thrown behind those efforts as well. I'm open to making all optimizations where we can, and I think there's still quite a few things we can do. However, what I look for from Tempest is confidence in making a production deploy of OpenStack, and out there are corners I would rather not cut. I personally would not be comfortable delegating tests for basic tasks such as resize, rebuild to run daily.  What do you think acceptable run times for a smoke grouping and full regression run should be?

Daryl


On Jun 1, 2012, at 4:03 PM, David Kranz wrote:

> I am a little confused about this. Most test classes define tearDownClass that frees resources allocated in setUpClass. But two of the classes deviate from this.
> 
> ServerActionsTest uses setUp and tearDown and creates a new server in setUp. I think this means that a new server is created before running each test method. This test is very slow, taking 9 minutes with three hefty compute nodes in a cluster. Many of the methods could reuse the same server and the negative tests don't need to create one at all. Unfortunately I think a lot of that time is spent just doing resize. I think we should consider making this test be nightly-build only.
> 
> ServerDetailsNegativeTest has methods that create lots of servers and has a tearDown method that deletes them after each test method. That seems unnecessary.
> This test is very slow, taking 3 minutes on a cluster with three hefty compute nodes. And that is with 15 tests being skipped pending bug fixes.
> It also has a tearDownClass method that deletes all running servers whether this test created them or not. That seems pretty bad. Why is it doing this?
> 
> Does any one have any comment about this?
> 
> -David
> 
> -- 
> Mailing list: https://launchpad.net/~openstack-qa-team
> Post to     : openstack-qa-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack-qa-team
> More help   : https://help.launchpad.net/ListHelp