openstack team mailing list archive

[Tim Simpson <tim.simpson@xxxxxxxxxxxxx>]

It would be advantageous for the tests to only use one client, whatever it turns out to be. I think it would be most practical to use a bleeding edge version of Nova client and add any features necessary to it in its own repo directly or by somehow mixing it in with the test code.

As for Dtest- I guess great minds think alike? It looks like both projects started at the same time (Proboscis began back in February in an internal Rackspace repo) so its unfortunate we're only just now discovering each other's work. Dtest looks very interesting and the ability to run tests in parallel is something I've been thought about doing in Proboscis, but unsure of how to proceed since Proboscis orders things before sending the list to Nose (for the purposes of backwards compatibility). It may be possible however for Proboscis to feed Dtest's execution routines.

At first glance the ordering functionality in dtest is similar to Proboscis but after reviewing some of the code in backfire I believe there are features of Proboscis not present there. Proboscis in general is TestNG in Python, and anytime I've broken away from how TestNG handles these things (usually on accident) I've found it to be a mistake-TestNG really did think out how to do this pretty well.

We should talk sometime, I think each project could gain a lot from it.

________________________________________
From: Matt Dietz
Sent: Monday, September 12, 2011 12:22 PM
To: Tim Simpson; Jay Pipes
Cc: openstack@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Openstack] Integration tests

Regarding novaclient, I was debating this myself last night. Our backfire
suite currently uses it. However, I can see scenarios where nova has
advanced but novaclient has yet to, and because all three projects are
independently developed, we're stuck temporarily. I can see utility in
tests specifically for Novaclient, and ones that use a built in client (as
the suite that Soren provided currently does.)

Also, Backfire uses a module Kevin Mitchell wrote named Dtest that
actually provides a lot of that same functionality you mention, Tim. It
also gives you the ability to spawn the tests individually in greenthreads
so they execute in parallel. We got push back when we initially tried to
merge our tests into Nova, partially because it uses a 3rd party library
for executing the tests. We can try merging these things together, if
that's what everyone wants.

On 9/12/11 11:39 AM, "Tim Simpson" <tim.simpson@xxxxxxxxxxxxx> wrote:

>I'm with the Reddwarf (Database as a Service) team at Rackspace. We
>employ a large set of integration / functional tests which run in a
>Vagrant controlled VM.
>
>Our tests use python-novaclient as well as a similar client for Reddwarf
>we've written ourselves. The motivation is to eat our own dog food- if
>the novaclient is the official rich client for Python, why shouldn't the
>test make use of it?
>
>We also use a library called Proboscis so we can order our tests without
>prefixing numbers to them, automatically skip related tests when
>something in a dependency fails, and avoid global variables even as we
>pass state between related tests. I think the openstack-integration-tests
>would definitely benefit from it.
>
>I'd love to have a conversation about getting the traits outlined above
>adopted into this standardized testing solution. :)
>
>Output of our tests running: http://pastie.org/2521835
>Proboscis documentation: http://packages.python.org/proboscis/
>________________________________________
>From: openstack-bounces+tim.simpson=rackspace.com@xxxxxxxxxxxxxxxxxxx
>[openstack-bounces+tim.simpson=rackspace.com@xxxxxxxxxxxxxxxxxxx] on
>behalf of Jay Pipes [jaypipes@xxxxxxxxx]
>Sent: Monday, September 12, 2011 8:20 AM
>To: Matt Dietz
>Cc: openstack@xxxxxxxxxxxxxxxxxxx
>Subject: Re: [Openstack] Integration tests
>
>Wow, another integration test framework emerges out of the woodwork :)
>
>Looking forward to getting all of these into a single place... and
>clearing up the lack of communication between teams on this subject!
>There's Kong, Stacktester, and Backfire. Any others out there we
>should know about?
>
>Cheers,
>jay
>
>On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz <matt.dietz@xxxxxxxxxxxxx>
>wrote:
>> Suite looks great, Soren!
>> Wanted to mention that we actually developed our own suite of tests,
>>which
>> you can find at https://github.com/ohthree/backfire I'm planning to
>> reconcile the difference so we can stop the independent efforts, but
>>it's
>> going to take time. Something else I'd like to see in the suite, that we
>> currently have in backfire via a custom test module, is the ability to
>> parallelize the tests for execution.
>> From: Soren Hansen <soren@xxxxxxxxxxx>
>> Date: Sat, 10 Sep 2011 00:21:10 +0200
>> To: <openstack@xxxxxxxxxxxxxxxxxxx>
>> Subject: Re: [Openstack] Integration tests
>>
>> That link shouldn't have included the +bug bit. Copy/paste fail. :(
>>
>> Sent from my phone. Pardon my brevity.
>>
>> _______________________________________________ Mailing list:
>> https://launchpad.net/~openstack Post to : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack More help :
>> https://help.launchpad.net/ListHelp This email may include confidential
>> information. If you received it in error, please delete it.
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>_______________________________________________
>Mailing list: https://launchpad.net/~openstack
>Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>Unsubscribe : https://launchpad.net/~openstack
>More help   : https://help.launchpad.net/ListHelp

This email may include confidential information. If you received it in error, please delete it.