canonical-ubuntu-qa team mailing list archive

[Dave Jones <2058040@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

autopkgtest is "clever" about when it resets the testbed betweens tests,
and this is a Bad Thing (tm) when it breaks assumptions about isolation.
Specifically, in lib/adt_testbed.py under Testbed.reset is found the
following code:

    def reset(self, deps_new, with_recommends):
        '''Reset the testbed, if possible and necessary'''

        adtlog.debug('testbed reset: modified=%s, deps_installed=%s(r: %s), deps_new=%s(r: %s)' %
                     (self.modified, self.deps_installed, self.recommends_installed,
                      deps_new, with_recommends))
        if 'revert' in self.caps and (
                self.modified or self.recommends_installed != with_recommends or
                [d for d in self.deps_installed if d not in deps_new]):
            adtlog.debug('testbed reset')
            pl = self.command('revert', (), 1)
            self._opened(pl)
        self.modified = False

Paraphrasing, if the test-bed is capable of reversion ('revert' in
self.caps), and the current test's dependencies (self.deps_installed)
include packages that are *not* present in the next test's dependencies
(deps_new), or if the recommends list has changed, then *and only then*
is the test bed sent the "revert" command which will reset it completely
(in the case of virt-qemu this restarts the VM from scratch). Otherwise
it doesn't bother.

If one makes the (fairly reasonable?) assumption that tests operate in
isolation to one another this can lead to some ... surprising results.
Consider the following d/t/control:

  Test-Command: touch /etc/foo
  Depends: build-essential
  Restrictions: isolation-machine, needs-root

  Test-Command: test -e /etc/foo
  Depends: build-essential
  Restrictions: isolation-machine, needs-root

If the tests are truly isolated, this *should* fail ... but it doesn't.
Reverse the order of the tests, and of course it does fail. Or add
another package to the Depends of the first test (so it's a superset of
the second test) and it fails.

Having spent quite some time failing to debug something, because I'd
made the faulty assumption that tests were actually isolated, this level
of "clever" is annoying and should either be removed or at least
documented, preferably in the man-page so it jumps out at me next time
I've forgotten this!

** Affects: autopkgtest (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/2058040

Title:
  autopkgtest doesn't isolate testbeds (sometimes!)

Status in autopkgtest package in Ubuntu:
  New

Bug description:
  autopkgtest is "clever" about when it resets the testbed betweens
  tests, and this is a Bad Thing (tm) when it breaks assumptions about
  isolation. Specifically, in lib/adt_testbed.py under Testbed.reset is
  found the following code:

      def reset(self, deps_new, with_recommends):
          '''Reset the testbed, if possible and necessary'''

          adtlog.debug('testbed reset: modified=%s, deps_installed=%s(r: %s), deps_new=%s(r: %s)' %
                       (self.modified, self.deps_installed, self.recommends_installed,
                        deps_new, with_recommends))
          if 'revert' in self.caps and (
                  self.modified or self.recommends_installed != with_recommends or
                  [d for d in self.deps_installed if d not in deps_new]):
              adtlog.debug('testbed reset')
              pl = self.command('revert', (), 1)
              self._opened(pl)
          self.modified = False

  Paraphrasing, if the test-bed is capable of reversion ('revert' in
  self.caps), and the current test's dependencies (self.deps_installed)
  include packages that are *not* present in the next test's
  dependencies (deps_new), or if the recommends list has changed, then
  *and only then* is the test bed sent the "revert" command which will
  reset it completely (in the case of virt-qemu this restarts the VM
  from scratch). Otherwise it doesn't bother.

  If one makes the (fairly reasonable?) assumption that tests operate in
  isolation to one another this can lead to some ... surprising results.
  Consider the following d/t/control:

    Test-Command: touch /etc/foo
    Depends: build-essential
    Restrictions: isolation-machine, needs-root

    Test-Command: test -e /etc/foo
    Depends: build-essential
    Restrictions: isolation-machine, needs-root

  If the tests are truly isolated, this *should* fail ... but it
  doesn't. Reverse the order of the tests, and of course it does fail.
  Or add another package to the Depends of the first test (so it's a
  superset of the second test) and it fails.

  Having spent quite some time failing to debug something, because I'd
  made the faulty assumption that tests were actually isolated, this
  level of "clever" is annoying and should either be removed or at least
  documented, preferably in the man-page so it jumps out at me next time
  I've forgotten this!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2058040/+subscriptions