canonical-ubuntu-qa team mailing list archive

[Tim Andersson <mp+477574@xxxxxxxxxxxxxxxxxx>]

Tim Andersson has proposed merging ~andersson123/autopkgtest-cloud:worker-nova-client-robustness into autopkgtest-cloud:master.

Requested reviews:
  Canonical's Ubuntu QA (canonical-ubuntu-qa)

For more details, see:
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/477574

Don't exit the worker script when the hypervisor is feeling poorly and not responding to novaclient requests!!!
-- 
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:worker-nova-client-robustness into autopkgtest-cloud:master.

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
index e5ab016..a5eb744 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
@@ -34,6 +34,7 @@ import systemd.journal
 from influxdb import InfluxDBClient
 from influxdb.exceptions import InfluxDBClientError
 from keystoneauth1 import session
+from keystoneauth1.exceptions.http import InternalServerError
 from keystoneauth1.identity import v2, v3
 from osc_lib import utils as osc_utils
 
@@ -674,7 +675,21 @@ def kill_openstack_server(test_uuid: str):
         session=sess,
         region_name=os.environ["OS_REGION_NAME"],
     )
-    for instance in nova.servers.list():
+    nova_servers = None
+    nova_retry_seconds = 10
+    nova_retry_attempts = 10
+    while not nova_servers:
+        for attempt in range(nova_retry_attempts):
+            try:
+                nova_servers = nova.servers.list()
+                break
+            except InternalServerError:
+                logging.info(
+                    f"Failed to list nova servers, retrying in {nova_retry_seconds} seconds, attempt {attempt+1}/{nova_retry_attempts+1}"
+                )
+        else:
+            return
+    for instance in nova_servers:
         if test_uuid in instance.name:
             try:
                 instance.delete()