openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On Tue, Sep 27, 2011 at 3:48 AM, Dean Horrigan <Dean_Horrigan@xxxxxxxx> wrote:
> I was hoping we could try arrange to get together at next weeks
> OpenStack Design Summit to discuss how can we work together to share:
>
> 1. Test Cases (manual and automated)
> 2. Test Resources
> 3. Test Environments
> 4. Test Automation
>
> This would help us get a better quality product and with the level of
> right collaboration reduce the cost of testing and the time to market.
> So this side of the conference would be a workshop with a small number
> of representatives from key players. I think opening this to a wide
> audience would be difficult to manage and end up being unproductive.

As you know, I'm up for this :) There is already a session accepted on
combining the existing functional/integration test suites:

http://summit.openstack.org/sessions/view/40

I propose we discuss your point #1 above in that session.

> I would also like to discuss possible tools/idea’s etc. that we think
> would benefit the community.
>
> >From a Dell perspective we have been working on a regression analysis
> framework. This tool would eventually hook into the code and identify
> which test cases need to be run for a specific build/drop, based on the
> changes since the last build. Therefore ensuring that the only tests run
> are those that need to be run. This will be done through a mapping
> system between code, features and tests.
>
> I think the above tool would be very beneficial to other test teams
> working on the OpenStack products. So if other companies were to
> contribute similar tools and suggestions then there is the potential
> that we can all benefit greatly from this.

I certainly want to make sure that we are de-duplicating the work that
is being done by the various groups interested in testing and QA. That
was the impetus behind the OpenStack QA team and project on Launchpad
:) Where can we see this regression analysis framework? Is it up on
GitHub?

As for test automation and test environments (points 4 and 3 above),
we already have a Jenkins system for the automation piece:

https://jenkins.openstack.org/job/openstack-deploy-rax/
https://jenkins.openstack.org/job/openstack-test-rax/

along with a number of scripts and glue code for firing off
deployments into bare-metal clusters available here:

https://github.com/openstack/openstack-ci
https://github.com/openstack/openstack-puppet

We plan on introducing the ability for these scripts to pull in Chef
recipes instead of the Puppet modules it currently pulls from the
GitHub repo above.

Right now, the openstack-deploy-rax deploys an OpenStack system onto a
4-machine cluster, that is "wiped and reset" after each test run with
a snapshot image that can be quickly kexec'd. This is to avoid the
lengthy build times associated with having to fully PXE boot with
something like cobbler or Crowbar.

That said, we would not be opposed to having additional Jenkins
builders that tested the full deployment environment *including* the
deployment framework (Cobbler, Crowbar, Ensemble, whatever). But, we'd
need to secure hardware resources (bare-metal) in order to do this
testing properly.

> So what I really want to find out is who would be interested in getting
> together next week, so that I can try and arrange a time and place.

I think from our team specifically, I would count on myself, Jim Blair
and Monty Taylor. I'm also pretty sure at least 4 folks from NTT
(Nati-san, Shida-san, Tushar Patil, Rohit Karajgi) will plan on
attending.

Looking forward to the discussions!
-jay