← Back to team overview

openstack team mailing list archive

Re: [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack


Adam, you've outdone yourself :-) That was a phenomenal reply with
tons of good information.

You've provided our team with all the info we need now -- thanks!


On Mon, Feb 27, 2012 at 10:36 PM, Adam Gandelman <adamg@xxxxxxxxxxxxx> wrote:
> On 02/27/2012 09:22 AM, Duncan McGreggor wrote:
>> It's been an invaluable source for not only information, but also
>> planning for the cloud work here at DreamHost.
> This is great to hear!  I'm happy to hear other people are benefiting from
> this as much as Ubuntu
>> To the point of this email, though, I have a question for you that I
>> wasn't able to parse an explicit answer to from your post: For the
>> packages that are built in the PPA linked above, are they only built
>> after all the components of OpenStack have been confirmed working as a
>> whole? Or are they built just after individual testing?
> The jenkins trunk build jobs are the ones responsible for uploading to the
> PPA.  It basically stuffs the newly built package in the local repository
> and uploads it to the PPA (if it fails to build, it doesnt upload).  The
> nova trunk build also triggers the deployment + testing.   So, to answer,
> those PPA packages are *pre* deployment + smoke testing.  The idea with that
> is that we can use the PPA externally to debug specific issues that might
> turn up during testing without blocking the deployment/smoke testing queue.
>  We're limited by the number of machines we have and can not currently run
> deployment testing in parallel.
>> My question comes from this concern: if we're building out a product
>> based on this PPA, (before Precise is delivered) we want to make sure
>> that when we bring up new systems by installing the packages from the
>> PPA, all of those work together properly. If the latest code from
>> keystone, for example, hasn't been building due to testing errors, we
>> want to make sure that the presence of the older keystone package in
>> the PPA won't be causing issues with the newer builds of the rest of
>> OpenStack.
> So you might have noticed the version of Keystone we are testing is getting
> a bit dusty.  We decided to freeze the version of Keystone we're testing
> just before the KSL/redux branch was merged.   There are still some features
> that need to land before we can modify our Juju charms to deploy the new
> version (specifically SQL persistence for the service catalog).  I'm hoping
> these will land before e4 (tomorrow) so we can begin testing the new branch
> this week.
>> To clarify: in your blog post, you explicitly mention the validation
>> process per component, starting with the upstream git repos. In the
>> deploy phase, you verify that the system as a whole (all of OpenStack)
>> works as expected. But what happens when one or more of those
>> components don't work? Are packages rolled back in the PPA until the
>> PPA only provides packages that will result in, once installed, a
>> complete working system?
> So far, we haven't rolled anything back in the PPA or the local repository.
>  I believe Keystone is the only package we've had to freeze while things
> stabilize upstream.  Fortunately, we've found the CI (even with the limited
> test coverage we currently run) is picking up new bugs almost
> instantaneously.  In most cases we're able to either get a fix into gerrit
> same day or find a proposed fix that we can cherry pick into our
> debian/patches/ repository and carry temporarily in our
> lp:~openstack-ubuntu-testing packages branches until its been fixed proper
> upstream.
> Please understand the PPA is for testing purposes only, and we're not making
> any promise that what installs from there is stable/usable/working.  We're
> using the work around that PPA and this CI to essentially gate what goes
> into the Ubuntu archive.  This cycle, Chuck has been uploading a weekly
> snapshot of Openstack (usually every Friday).  With the CI in place, we can
> essentially verify that those packages build clean and install okay, and
> provide something usable given the last weeks Gerrit churn.
> Keep in mind this is still a developing process.  I expect this will evolve
> over the next few months.  We've just begun trigger pre-commit testing to
> the Diablo stable branch against Oneiric.  Still working out some kinks, but
> we'll hopefully be going into Folsom+Precise with coverage of both trunk and
> stable updates to Essex/Precise LTS.
> Also note that, at this point in the Ubuntu development cycle, it can often
> take *a long time* before an upload to a PPA is built and ready for use.
>  Its probably safe to assume that what you're using from the PPA is what we
> were testing yesterday (check the version of the package for a timestamp)
>> Keep up the great work, guys -- you have fans out here in the wild,
>> wild world of OpenStack :-)
> :)
> Cheers,
> Adam
> --
> Ubuntu-cloud mailing list
> Ubuntu-cloud@xxxxxxxxxxxxxxxx
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud