← Back to team overview

openstack team mailing list archive

Re: [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack


On 02/27/2012 09:22 AM, Duncan McGreggor wrote:
It's been an invaluable source for not only information, but also
planning for the cloud work here at DreamHost.

This is great to hear! I'm happy to hear other people are benefiting from this as much as Ubuntu

To the point of this email, though, I have a question for you that I
wasn't able to parse an explicit answer to from your post: For the
packages that are built in the PPA linked above, are they only built
after all the components of OpenStack have been confirmed working as a
whole? Or are they built just after individual testing?

The jenkins trunk build jobs are the ones responsible for uploading to the PPA. It basically stuffs the newly built package in the local repository and uploads it to the PPA (if it fails to build, it doesnt upload). The nova trunk build also triggers the deployment + testing. So, to answer, those PPA packages are *pre* deployment + smoke testing. The idea with that is that we can use the PPA externally to debug specific issues that might turn up during testing without blocking the deployment/smoke testing queue. We're limited by the number of machines we have and can not currently run deployment testing in parallel.

My question comes from this concern: if we're building out a product
based on this PPA, (before Precise is delivered) we want to make sure
that when we bring up new systems by installing the packages from the
PPA, all of those work together properly. If the latest code from
keystone, for example, hasn't been building due to testing errors, we
want to make sure that the presence of the older keystone package in
the PPA won't be causing issues with the newer builds of the rest of

So you might have noticed the version of Keystone we are testing is getting a bit dusty. We decided to freeze the version of Keystone we're testing just before the KSL/redux branch was merged. There are still some features that need to land before we can modify our Juju charms to deploy the new version (specifically SQL persistence for the service catalog). I'm hoping these will land before e4 (tomorrow) so we can begin testing the new branch this week.

To clarify: in your blog post, you explicitly mention the validation
process per component, starting with the upstream git repos. In the
deploy phase, you verify that the system as a whole (all of OpenStack)
works as expected. But what happens when one or more of those
components don't work? Are packages rolled back in the PPA until the
PPA only provides packages that will result in, once installed, a
complete working system?

So far, we haven't rolled anything back in the PPA or the local repository. I believe Keystone is the only package we've had to freeze while things stabilize upstream. Fortunately, we've found the CI (even with the limited test coverage we currently run) is picking up new bugs almost instantaneously. In most cases we're able to either get a fix into gerrit same day or find a proposed fix that we can cherry pick into our debian/patches/ repository and carry temporarily in our lp:~openstack-ubuntu-testing packages branches until its been fixed proper upstream.

Please understand the PPA is for testing purposes only, and we're not making any promise that what installs from there is stable/usable/working. We're using the work around that PPA and this CI to essentially gate what goes into the Ubuntu archive. This cycle, Chuck has been uploading a weekly snapshot of Openstack (usually every Friday). With the CI in place, we can essentially verify that those packages build clean and install okay, and provide something usable given the last weeks Gerrit churn.

Keep in mind this is still a developing process. I expect this will evolve over the next few months. We've just begun trigger pre-commit testing to the Diablo stable branch against Oneiric. Still working out some kinks, but we'll hopefully be going into Folsom+Precise with coverage of both trunk and stable updates to Essex/Precise LTS.

Also note that, at this point in the Ubuntu development cycle, it can often take *a long time* before an upload to a PPA is built and ready for use. Its probably safe to assume that what you're using from the PPA is what we were testing yesterday (check the version of the package for a timestamp)

Keep up the great work, guys -- you have fans out here in the wild,
wild world of OpenStack :-)



Follow ups