canonical-ubuntu-qa team mailing list archive

[Paride Legovini <2067628@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Its has been reported (thanks ginggs) that failures can happen in
autopkgtest-cloud with error:

  Quota exceeded for cores: Requested 2, but already used 514 of 515
cores (HTTP 403)

Full log at [1]. I checked number of running VMs shortly after, and I
counted less than 400 cores in use (taking into account that
autopkgtest-big instances have 4 cores).

This may be a side effect of dropping the flock [2]. It may be that the
instance deletion is asynchronous, and cores are freed only after the
delete operation is complete.

We should do something like:

1. Figure out a way to query openstack for the current quota usage, and check how it matches the number of running VMs.
2. Check if in the worker we can do something like instance.delete(wait=True) so that we want for the VM to be deleted before proceeding. I made that option up, but given that the CLI tool has a wait parameter, delete() is likely to also have something like that.
3. Check whether this improves the comparison of point (1.)

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-xenial/xenial/i386/u/ubuntu-advantage-tools/20240528_152118_2d212@/log.gz
[2] https://salsa.debian.org/ubuntu-ci-team/autopkgtest/-/commit/49f5760dddcdf7b3f70c177f3000391d1db0dbdd

** Affects: auto-package-testing
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to Auto Package Testing.
https://bugs.launchpad.net/bugs/2067628

Title:
  openstack core count may be inaccurate

Status in Auto Package Testing:
  New

Bug description:
  Its has been reported (thanks ginggs) that failures can happen in
  autopkgtest-cloud with error:

    Quota exceeded for cores: Requested 2, but already used 514 of 515
  cores (HTTP 403)

  Full log at [1]. I checked number of running VMs shortly after, and I
  counted less than 400 cores in use (taking into account that
  autopkgtest-big instances have 4 cores).

  This may be a side effect of dropping the flock [2]. It may be that
  the instance deletion is asynchronous, and cores are freed only after
  the delete operation is complete.

  We should do something like:

  1. Figure out a way to query openstack for the current quota usage, and check how it matches the number of running VMs.
  2. Check if in the worker we can do something like instance.delete(wait=True) so that we want for the VM to be deleted before proceeding. I made that option up, but given that the CLI tool has a wait parameter, delete() is likely to also have something like that.
  3. Check whether this improves the comparison of point (1.)

  [1] https://autopkgtest.ubuntu.com/results/autopkgtest-xenial/xenial/i386/u/ubuntu-advantage-tools/20240528_152118_2d212@/log.gz
  [2] https://salsa.debian.org/ubuntu-ci-team/autopkgtest/-/commit/49f5760dddcdf7b3f70c177f3000391d1db0dbdd

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/2067628/+subscriptions