launchpad-reviewers team mailing list archive

[Colin Watson <mp+427615@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad-buildd:self-service-rebuild-images into launchpad-buildd:master.

Commit message:
Describe waiting for image rebuilds in more detail

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/427615

In https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/427478, Jürgen asked "How do you know the image building has finished?"  We don't have a straightforward view of this, unfortunately, but here's some more detail.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad-buildd:self-service-rebuild-images into launchpad-buildd:master.

diff --git a/docs/how-to/deployment.rst b/docs/how-to/deployment.rst
index 8e9d83e..ebfb40a 100644
--- a/docs/how-to/deployment.rst
+++ b/docs/how-to/deployment.rst
@@ -85,10 +85,26 @@ Releasing to production
    staging management environment (``stg-vbuilder@launchpad-bastion-ps5``)
    to start rebuilding images.
 
-#. Once image builds complete, builders will get the new image after they
-   finish their next build (or are disabled) and go through being reset.
-   `Build farm administrators
+#. Once the new image is rebuilt, which normally takes on the order of 15-60
+   minutes depending on the architecture, builders will get the new image
+   after they finish their next build (or are disabled) and go through being
+   reset.  As a result, ``manage-builders -v`` should start showing the new
+   version over time.
+
+#. Wait for the new version to appear for at least one builder in each
+   region and architecture (other than ``riscv64``, which is managed
+   separately).  If this doesn't happen after 90 minutes, then ask IS for
+   assistance in investigating; they can start by checking ``juju status``
+   in ``prod-launchpad-vbuilders@is-bastion-ps5.internal``.
+
+#. Once the updated version is visible for at least one builder in each
+   region and architecture (other than ``riscv64``), `build farm
+   administrators
    <https://launchpad.net/~launchpad-buildd-admins/+members>`_ can use
-   ``manage-builders --virt --idle --reset`` to reset idle builders.
+   ``manage-builders --virt --idle --reset`` to reset idle builders, thereby
+   causing builders that haven't taken any builds recently to catch up.
+
+#. Ask Colin Watson or William Grant to reflash ``riscv64`` builders to the
+   new version (currently a separate manual process).
 
 #. Close any bugs fixed by the new release.