openstack team mailing list archive

["Daniel P. Berrange" <berrange@xxxxxxxxxx>]

On Thu, Jun 28, 2012 at 08:13:28AM -0700, Monty Taylor wrote:
> On 06/28/2012 07:32 AM, Daniel P. Berrange wrote:
> > This leaves me with the following questions...
> > 
> >  1. Why was the recorded failure from SmokeStack not considered
> >     to be a blocker for the merge of the commit by Gerrit or
> >     Jenkins or any of the reviewers ?
> >
> >  2. Why did SmokeStack not get re-triggered for the later patch
> >     set revisions, before it was merged ?
> 
> The answer to 1 and 2 is largely the same - SmokeStack is a community
> contributed resources and is not managed by the CI team. Dan Prince does
> a great job with it, but it's not a resource that we have the ability to
> fix should it start messing up, so we have not granted it the
> permissions to file blocking votes.
> 
> The tests that smokestack runs could all be written such that they are
> run by jenkins. The repos that run the jenkins tests are all in git and
> managed by openstack's gerrit. If there are testing profiles that it
> runs that we as a community value and want to see part of the gate,
> anyone is welcome to port them.

Ok, this makes sense to me now. I had assumed SmokeStack was a core
part of the infrastructure.

> >  3. Why did Jenkins not ensure that the tests were run on a libvirt
> >     enabled host ?
> 
> This is a different, and slightly more complex. We run tests in
> virtualenvs so that the process used to test the code can be
> consistently duplicated by all of the developers in the project. This is
> the reason that we no longer do ubuntu package creation as part of the
> gate - turns out that's really hard for a developer running on OSX to do
> locally on their laptop - and if Jenkins reports an blocking error in a
> patch, we want a developer to be able to reproduce the problem locally
> so that they can have a chance at fixing it.
> 
> Problem arise in paradise though. libvirt being one of them. It's not
> possible to install libvirt into a virtualenv, because it's a swig-based
> module built as part of the libvirt source itself. One of the solutions
> to this is to allow the testing virtual environments to use packages
> installed at the system level. We suggested this a little while ago, but
> this was rejected by the nova team who valued the benefit of having a
> restricted test run so that we know we've got all of the depends
> properly specified.
> 
> To that end, after chatting with Brian Waldon, I put this up as a
> possible next try:
> 
> https://review.openstack.org/#/c/8949/
> 
> Which adds an additional testing environment that has system software
> enabled and also installs additional "optional" things. With that
> environment, we should be able to run a jenkins gate that tests things
> with full libvirt, and also tests the mysql upgrade paths, without
> screwing our fine friends who run OSX.
> 
> Fundamentally though - we're at a point of trying to have our cake and
> eat it too. Either we want comprehensive testing of all of the unit
> tests, or we want to be careful about not making the test environment to
> hard for a developer to exactly mimic.
> 
> I'm obviously on the side of having us have gating tests that some devs
> might not be able to do on their laptops - such as  running the libvirt
> tests properly. We're working on cloud software - worst case scenario if
> there's an intractable problem, as dev can always spin up an ubuntu
> image somewhere.

I think I agree with you, since in practice I believe the additional
requirements on developers are not unreasonable in general. Taking the
livirt example (though it applies to other examples like MySQL too)...

If a developer is submitting changes that touches a part of OpenStack
unrelated to the virt drivers, then there's low liklihood that they'll
cause libvirt test failures. This means they won't need to have libvirt
available themselves in the common case, and thus there's no new onerous
requirement placed on them. A similar scenario occurs if they are touching
non-libvirt drivers (eg XenAPI, VMWare).

Only if they touch the libvirt driver itself, or things that it is derived
/from/ (eg the base virt driver code), then are they likely to need to run
the tests locally to toubleshoot. In such a case, it is not unreasonable
that they be prepared to setup local installs for troubleshooting purposes.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|