launchpad-reviewers team mailing list archive

[Ines Almeida <mp+464936@xxxxxxxxxxxxxxxxxx>]

Ideally what should happen when deploying is that we either: (1) should only upgrade all the builders if all images exist (hard to implement), or we upgrade all of the images that can be upgraded and investigate and fix the one(s) that failed (what this MP does).

Regardless of the reason why images are missing, the current behavior is far from ideal - we neither refrain from deploying, nor do we deploy all but the 1 failing. Moving forward with the deployment and listing the images that failed with a big THESE ONES FAILED warning, is much better here.

---

Regarding the why images were missing in qastaging: some `glance-simplestreams-sync` units (that build the images) were deactivated in qastaging due to network issues from last year. I had a chat with Colin about it. A lot of our builders in qastaging are not in a clear state, and we need to address it.

But note that currently it is NOT POSSIBLE to deploy buildd to qastaging for the builders that are active and working (it is only possible now because I cowboyed this update there), and it wouldn't be possible to know which arch/regions are failing because it fails in the first iteration of the deployment.

This change lists the failing units, so that we can address them all, and still move forward in our projects.

---

I updated the error message to be more pressing that something needs to be addressed (just made it all Caps with the word `FAILURE` on it), and updated the naming of `instanced` to `applications` to make it coherent.

---

As mentioned before in the description, after updating this, we will also need to update the `buildd` deployment docs to mention addressing this error message if it appears.

Diff comments:

> diff --git a/vbuilder/rebuild-images b/vbuilder/rebuild-images
> index b4f8348..b28fc02 100755
> --- a/vbuilder/rebuild-images
> +++ b/vbuilder/rebuild-images
> @@ -77,8 +78,17 @@ def main():
>              "juju", "ssh", unit, "sudo", "/usr/local/bin/rebuild-latest-image",
>              f"{name_prefix}/ubuntu-{series}-daily-{arch}-",
>              ]
> -        utils.run(None, rebuild_cmd)
> +        try:
> +            utils.run(None, rebuild_cmd)
> +        except Exception as e:

The exception raised is actually a generic `Exception`.

```
Exception: No active image starting with 'launchpad-buildd-qastaging/ubuntu-focal-daily-arm64-gpu-'
```

> +            failed_target_instances.append(application)
> +            print(f"Command in `{application}` failed with error: {str(e)}")
>  
> +    print("\n\nRun completed.")
> +    if failed_target_instances:
> +        print("Rebuilding images failed in the following instances:")
> +        for instance in failed_target_instances:
> +            print(f" - {instance}")
>  
>  if __name__ == "__main__":
>      main()


-- 
https://code.launchpad.net/~ines-almeida/launchpad-mojo-specs/+git/private/+merge/464936
Your team Launchpad code reviewers is subscribed to branch ~launchpad/launchpad-mojo-specs/+git/private:vbuilder.