openstack team mailing list archive

["Daniel P. Berrange" <berrange@xxxxxxxxxx>]

Today we face a situation where Nova GIT master fails to pass all
the libvirt test cases. This regression was accidentally introduced
by the following changeset

   https://review.openstack.org/#/c/8778/

If you look at the history of that, the first SmokeStack test run
fails with some (presumably) transient errors, and added negative
karma to the change against patchset 2. If it were not for this
transient failure, it should have shown the regression in the
libvirt test case. The libvirt test case in question was one that
is skipped, unless libvirt is actually present on the host running
the tests. SmokeStack had made sure the tests would run on such a
host.

There were then further patchsets uploaded, and patchset 4 was
approved for merge. Jenkins ran its gate jobs and these all passed
successfully. I am told that Jenkins will actually run the unittests
that are included in Nova, so I would have expected it to see the
flawed libvirt test case, but it didn't. I presume therefore, that
Jenkins is not running on a libvirt enabled host.

The end result was that the broken changeset was merged to master,
which in turns means any other developers submitting changes
touching the libvirt area will get broken tests reported that
were not actually their own fault.

This leaves me with the following questions...

 1. Why was the recorded failure from SmokeStack not considered
    to be a blocker for the merge of the commit by Gerrit or
    Jenkins or any of the reviewers ?

 2. Why did SmokeStack not get re-triggered for the later patch
    set revisions, before it was merged ?

 3. Why did Jenkins not ensure that the tests were run on a libvirt
    enabled host ?


Obviously this was all made worse by the transient problems we've had
with the tests suite infrastructure these past 2 days, but regardless
it seems like we have a gap in our merge approval procedures here.

IMHO, either SmokeStack needs to be made compulsory, or Jenkins needs
to ensure tests are run on suitable hosts like SmokeStack does, or
both.

Regards,
Daniel

[1] 
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|