openstack team mailing list archive
Mailing list archive
Re: Steps that can help stabilize Nova's trunk
Catching up as well...
On Thu, Feb 17, 2011 at 2:57 PM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:
> The Anso team has told me that they run the smoketests regularly, but
> I would assume they only run these smoketests against a test cluster
> that mimicks the Nebula environment. Devin and termie can correct me
> if I'm wrong on that.
We've taken a few approaches as the code has evolved (automatic
deployment and testing was always crucial)
While preparing for the Nebula beta release our system was as follows.
Automatically deploying a fresh cluster every morning using a shell
script that would:
* pxe/preseed -> deploy and configure base ubuntu os
* puppet -> deploy and configure nova
* run a few shell commands on the head node to setup users/whatnot
Then we would run the smoketests against the cluster, recording errors
and exceptions (to sentry using nova-sentry).
Then through the day users and QA would test against the cluster,
reporting bugs. If required anyone could re-deploy (pxe -> setup) by
running a single shell command. (it would take about 30 minutes from
reboot to ready to run)
Running a cloud is more than just nova. We've had critical issues
with configuration (intel 10G cards + LRO/MRO + linux network bridging
brought network performance for guests down to 20Kbps) that were
completely external to Nova, but crucial for running an openstack
Have a set of "golden systems": OS + configurations (network model +
api + hypervisors) + relevant smoketests
that run with every trunk merge. Having the system that runs these
tests open source so others can setup labs to run tests for their
equipment/configurations would be preferable as well.
I also agree that "machines are cheaper than devs" and think a good
phase two would be able to run this against arbitrary branches.