canonical-ubuntu-qa team mailing list archive

[Brian Murray <mp+466336@xxxxxxxxxxxxxxxxxx>]

Brian Murray has proposed merging autopkgtest-cloud:retryKeystoneConnectFailures into autopkgtest-cloud:master.

Commit message:
worker: retry tests which fail trying to contact the keystone service

Requested reviews:
  Canonical's Ubuntu QA (canonical-ubuntu-qa)

For more details, see:
https://code.launchpad.net/~ubuntu-release/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/466336

In https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/amd64/r/r-cran-survey/20240522_110540_37db1@/log.gz I saw the following traceback:

769s Traceback (most recent call last):
769s   File "/usr/lib/python3/dist-packages/novaclient/shell.py", line 820, in main
769s     OpenStackComputeShell().main(argv)
769s   File "/usr/lib/python3/dist-packages/novaclient/shell.py", line 742, in main
769s     args.func(self.cs, args)
769s   File "/usr/lib/python3/dist-packages/novaclient/v2/shell.py", line 971, in do_boot
769s     server = cs.servers.create(*boot_args, **boot_kwargs)
769s   File "/usr/lib/python3/dist-packages/novaclient/v2/servers.py", line 1477, in create
769s     return self._boot(response_key, *boot_args, **boot_kwargs)
769s   File "/usr/lib/python3/dist-packages/novaclient/v2/servers.py", line 836, in _boot
769s     return self._create('/servers', body, response_key,
769s   File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
769s     resp, body = self.api.client.post(url, body=body)
769s   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post
769s     return self.request(url, 'POST', **kwargs)
769s   File "/usr/lib/python3/dist-packages/novaclient/client.py", line 69, in request
769s     resp, body = super(SessionClient, self).request(url,
769s   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 545, in request
769s     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
769s   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request
769s     return self.session.request(url, method, **kwargs)
769s   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 913, in request
769s     resp = send(**kwargs)
769s   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1020, in _send_request
769s     raise exceptions.ConnectFailure(msg)
769s keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://nova.ps5.canonical.com:8774/v2.1/servers: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
769s ERROR (ConnectFailure): Unable to establish connection to https://nova.ps5.canonical.com:8774/v2.1/servers: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

We should consider that a temp fail.
-- 
Your team Canonical's Ubuntu QA is requested to review the proposed merge of autopkgtest-cloud:retryKeystoneConnectFailures into autopkgtest-cloud:master.

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
index 0173fe5..f378182 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
@@ -119,6 +119,7 @@ TEMPORARY_TEST_FAIL_STRINGS = [
     ": error cleaning up:",
     " has modification time ",  # clock skew, LP: #1880839
     "OSError: [Errno 28] No space left on device",
+    "keystoneauth1.exceptions.connection.ConnectFailure",
 ]
 
 # If we repeatedly time out when installing, there's probably a problem with