openstack team mailing list archive

[Jesse Andrews <anotherjesse@xxxxxxxxx>]

Catching up as well...

On Thu, Feb 17, 2011 at 2:57 PM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:
> The Anso team has told me that they run the smoketests regularly, but
> I would assume they only run these smoketests against a test cluster
> that mimicks the Nebula environment. Devin and termie can correct me
> if I'm wrong on that.

We've taken a few approaches as the code has evolved (automatic
deployment and testing was always crucial)

While preparing for the Nebula beta release our system was as follows.

Automatically deploying a fresh cluster every morning using a shell
script that would:

  * pxe/preseed -> deploy and configure base ubuntu os
  * puppet -> deploy and configure nova
  * run a few shell commands on the head node to setup users/whatnot

Then we would run the smoketests against the cluster, recording errors
and exceptions (to sentry using nova-sentry).

Then through the day users and QA would test against the cluster,
reporting bugs.  If required anyone could re-deploy (pxe -> setup) by
running a single shell command.  (it would take about 30 minutes from
reboot to ready to run)

Running a cloud is more than just nova.  We've had critical issues
with configuration (intel 10G cards + LRO/MRO + linux network bridging
brought network performance for guests down to 20Kbps) that were
completely external to Nova, but crucial for running an openstack
deployment.

Have a set of "golden systems":   OS + configurations (network model +
api + hypervisors) + relevant smoketests
that run with every trunk merge.  Having the system that runs these
tests open source so others can setup labs to run tests for their
equipment/configurations would be preferable as well.

I also agree that "machines are cheaper than devs" and think a good
phase two would be able to run this against arbitrary branches.