launchpad-reviewers team mailing list archive

[Ines Almeida <mp+464936@xxxxxxxxxxxxxxxxxx>]

Ines Almeida has proposed merging ~ines-almeida/launchpad-mojo-specs/+git/private:vbuilder-rebuild-images-manifest-update into ~launchpad/launchpad-mojo-specs/+git/private:vbuilder.

Commit message:
Update `rebuild-images` script to be more resilient when images are missing
    
The script now skips the instances with the missing images, and log it in a message at the end of the run.
This is important because we might have known issues with building some images.


Requested reviews:
  Canonical Launchpad Engineering (launchpad)

For more details, see:
https://code.launchpad.net/~ines-almeida/launchpad-mojo-specs/+git/private/+merge/464936

I will also do some changes to the deployment documentation in the `launchpad-buildd` repository as well, to add more context.

As it was, one image missing in qastaging would lead to not being able to deploy to any of the other builders. This will make deployment clearer by letting us know which instances had issues out of all of them, and unblocking deployments when there are known issues with a certain image.
-- 
Your team Launchpad code reviewers is subscribed to branch ~launchpad/launchpad-mojo-specs/+git/private:vbuilder.

diff --git a/vbuilder/rebuild-images b/vbuilder/rebuild-images
index b4f8348..b28fc02 100755
--- a/vbuilder/rebuild-images
+++ b/vbuilder/rebuild-images
@@ -70,6 +70,7 @@ def main():
     name_prefix = name_prefix_by_stage[stage]
     targets = targets_by_stage[stage]
     juju_services = utils.juju_services()
+    failed_target_instances = []
     for region, arch, series in targets:
         application = f"glance-simplestreams-sync-{region}-{arch}"
         unit = get_leader_unit(juju_services, application)
@@ -77,8 +78,17 @@ def main():
             "juju", "ssh", unit, "sudo", "/usr/local/bin/rebuild-latest-image",
             f"{name_prefix}/ubuntu-{series}-daily-{arch}-",
             ]
-        utils.run(None, rebuild_cmd)
+        try:
+            utils.run(None, rebuild_cmd)
+        except Exception as e:
+            failed_target_instances.append(application)
+            print(f"Command in `{application}` failed with error: {str(e)}")
 
+    print("\n\nRun completed.")
+    if failed_target_instances:
+        print("Rebuilding images failed in the following instances:")
+        for instance in failed_target_instances:
+            print(f" - {instance}")
 
 if __name__ == "__main__":
     main()