canonical-ubuntu-qa team mailing list archive

[Skia <2067628@xxxxxxxxxxxxxxxxxx>]

About 2., and the `instance.delete(wait=True)` idea, I've dug a bit in the code of the openstack clients, and I've found that:
* The `--wait` option of the `openstack server delete` command is implemented here: https://opendev.org/openstack/python-openstackclient/src/commit/dd6ac285d5b35f87947f9c00f5558b55e9787747/openstackclient/compute/v2/server.py#L2135  
* This calls the `wait_for_delete` function defined here: https://opendev.org/openstack/osc-lib/src/commit/f9bcdecf1abd0c38fd60cfd7d99a00b113bd6d57/osc_lib/utils/__init__.py#L661
* This is a simple polling loop, there's nothing more.
* I need to run some experiments, but it looks doable to make that same call to `wait_for_delete` in the worker code.
* In a second time, maybe this is something that could be implemented somehow directly in `novaclient` here: https://opendev.org/openstack/python-novaclient/src/commit/2bd135c13793da48af5fbad5208b00aaba7e39dc/novaclient/v2/servers.py#L66, to add that convenient `wait=True` parameter, but this would first require opening a bug somewhere to discuss this with the maintainers.

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to Auto Package Testing.
https://bugs.launchpad.net/bugs/2067628

Title:
  openstack core count may be inaccurate

Status in Auto Package Testing:
  In Progress

Bug description:
  Its has been reported (thanks ginggs) that failures can happen in
  autopkgtest-cloud with error:

    Quota exceeded for cores: Requested 2, but already used 514 of 515
  cores (HTTP 403)

  Full log at [1]. I checked number of running VMs shortly after, and I
  counted less than 400 cores in use (taking into account that
  autopkgtest-big instances have 4 cores).

  This may be a side effect of dropping the flock [2]. It may be that
  the instance deletion is asynchronous, and cores are freed only after
  the delete operation is complete.

  We should do something like:

  1. Figure out a way to query openstack for the current quota usage, and check how it matches the number of running VMs.
  2. Check if in the worker we can do something like instance.delete(wait=True) so that we want for the VM to be deleted before proceeding. I made that option up, but given that the CLI tool has a wait parameter, delete() is likely to also have something like that.
  3. Check whether this improves the comparison of point (1.)

  [1] https://autopkgtest.ubuntu.com/results/autopkgtest-xenial/xenial/i386/u/ubuntu-advantage-tools/20240528_152118_2d212@/log.gz
  [2] https://salsa.debian.org/ubuntu-ci-team/autopkgtest/-/commit/49f5760dddcdf7b3f70c177f3000391d1db0dbdd

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/2067628/+subscriptions