cloud-init team mailing list archive

[Daniel Watkins <daniel.watkins@xxxxxxxxxxxxx>]

Hello folks,

I've just proposed a PR which will switch cloud-init's test runner from
nose to pytest: https://github.com/canonical/cloud-init/pull/211

I go into more detail below, but the TL;DR is that nose is deprecated,
pytest is a good replacement, and we plan on landing this next week,
after 20.1 has been cut.

Why do we want to move away from nose?  nose has been in maintenance
mode for years (per its own docs[0]) and it eventually will break[1].
Doing this now means that we don't have to treat the migration as a fire
drill.  This is the primary motivator for making a change.

Why do we want to move to pytest?  There are a few reasons:

* pytest is popular.  GitHub metrics are perhaps not the best
  metric, but pytest is "used by" 150k projects (vs nose with 73.9k, or
  nose2 with 2.1k).  This means that there's a decent chance that a
  Python developer coming to the cloud-init project will already have
  experience using pytest.
* pytest is actively developed.  Over the past month, "19 authors have
  pushed 179 commits to master" in the pytest project, (vs. 1 author and
  2 commits for nose2).  This means that it's less likely that we will
  have to perform such a migration in future.  It also means that any
  bugs we find are more likely to be addressed.
* pytest is stable.  Members of the cloud-init core team also maintain a
  project which uses pytest that has to work all the way back in
  Ubuntu 14.04 (which has pytest 2.5.1).  We have had to work around the
  absence of _features_ in older pytest versions, but have had no
  problems with core functionality.
* pytest is featureful.  In addition to being an effective test runner,
  it also provides rich assertion output, dependency injection (via
  fixtures) and, most importantly, test parameterisation[2].  These all
  provide means to reduce test boilerplate, and more easily expand tests
  to cover multiple scenarios.
* As alluded to above, the cloud-init core team uses pytest for other
  projects, so we already have a depth of familiarity and expertise with
  what pytest can do and how best to use it.  This will reduce the cost
  of the migration to us, as well as meaning we are well-placed to
  support the community.

In terms of a call for testing, I don't use the `run-container` or
`tox-venv` scripts so I haven't tested them (and I don't believe they
are exercised by our automated testing).  If you use these, please take
a minute to test that the new branch won't break you once it lands.
Beyond that, please do test out the branch and leave any thoughts you
have on the pull request.

As you can see in the pull request diff[3], no test code changes were
required for the migration, so I don't believe that any in-flight PRs
will be affected by this change.  With that said, so that we have a full
upstream dev cycle to account for any unexpected issues that arise, we
plan to land this branch _after_ 20.1 is cut (which is happening on
Tuesday[3]).

As always, if you have any questions, comments or concerns,  please
reply to this mail or join #cloud-init on Freenode IRC and bring them up
there.


Thanks!

Dan


[0] https://nose.readthedocs.io/en/latest/
[1] As an example of the potential fragility of nose: the codebase only
    supports Python 2, but when installed with Python 3, it runs `2to3`
    over itself, and then installs the result.
[2] https://docs.pytest.org/en/latest/parametrize.html
[3] https://github.com/canonical/cloud-init/pull/211/files
[4] https://lists.launchpad.net/cloud-init/msg00244.html

Attachment: signature.asc
Description: PGP signature