yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1854212] [NEW] For ironic nova-computes that have no nodes associated aggregate operations are inconsistent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Vladyslav Drok <vdrok@xxxxxxxxxxxx>
Date: Wed, 27 Nov 2019 19:25:01 -0000
Reply-to: Bug 1854212 <1854212@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Consider a case of mixed hypervisor setup, there are computes doing QEMU
instances, and there are computes providing possibility to boot ironic
nodes. There are no ironic nodes commissioned at the moment, but QEMU
computes are setup and ready to go. Here is how host list, service list
and hypervisor list commands output look like (bmt hosts are computes
with ironic virt driver):

root@ctl01:~# openstack compute service list --service nova-compute
+----+--------------+--------+----------+---------+-------+----------------------------+
| ID | Binary       | Host   | Zone     | Status  | State | Updated At                 |
+----+--------------+--------+----------+---------+-------+----------------------------+
| 55 | nova-compute | bmt03  | foo_zone | enabled | up    | 2019-11-27T12:08:54.000000 |
| 58 | nova-compute | bmt02  | nova     | enabled | up    | 2019-11-27T12:08:32.000000 |
| 61 | nova-compute | bmt01  | nova     | enabled | up    | 2019-11-27T12:09:06.000000 |
| 64 | nova-compute | cmp003 | nova     | enabled | up    | 2019-11-27T12:08:24.000000 |
| 67 | nova-compute | cmp001 | nova     | enabled | up    | 2019-11-27T12:08:14.000000 |
| 70 | nova-compute | cmp002 | nova     | enabled | up    | 2019-11-27T12:08:21.000000 |
+----+--------------+--------+----------+---------+-------+----------------------------+
root@ctl01:~# openstack host list | grep compute
| bmt03     | compute     | foo_zone |
| bmt02     | compute     | nova     |
| bmt01     | compute     | nova     |
| cmp003    | compute     | nova     |
| cmp001    | compute     | nova     |
| cmp002    | compute     | nova     |
root@ctl01:~# openstack hypervisor list
+----+--------------------------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname                  | Hypervisor Type | Host IP      | State |
+----+--------------------------------------+-----------------+--------------+-------+
|  7 | cmp003.bm-cicd-queens-ovs-maas.local | QEMU            | 10.167.11.17 | up    |
| 10 | cmp001.bm-cicd-queens-ovs-maas.local | QEMU            | 10.167.11.15 | up    |
| 13 | cmp002.bm-cicd-queens-ovs-maas.local | QEMU            | 10.167.11.16 | up    |

Then the following test is run:
tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_remove_host_as_user
[id-7a53af20-137a-4e44-a4ae-e19260e626d9,negative]

The test tries to add a host to an aggregate and then remove it, addtion
seems to be ok but removal fails with the following:

Cannot remove host bmt03 in aggregate 52 (HTTP 404) (Request-ID: req-
25acad79-b820-4889-a2a3-71b4a7913a86)

In nova api something like the following can be seen:

2019-11-27 12:11:42,263.263 2762 INFO nova.api.openstack.wsgi [req-
80f752d1-0ce6-446a-ba92-cf815bfedd78 b31b4e9171154b1d9d1fec5af0ed5880
6a33e1d31b53496885e19d7cba4390dd - default default] HTTP exception
thrown: Host 'bmt01' is not mapped to any cell

The problem seems to be in an inconsistent behaviour during adding the
host to aggregate and removing it from aggregate, as during addition it
allows host mapping to be missing, but does not allow it during host
removal.

** Affects: nova
     Importance: Undecided
     Assignee: Vladyslav Drok (vdrok)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => Vladyslav Drok (vdrok)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1854212

Title:
  For ironic nova-computes that have no nodes associated aggregate
  operations are inconsistent

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Consider a case of mixed hypervisor setup, there are computes doing
  QEMU instances, and there are computes providing possibility to boot
  ironic nodes. There are no ironic nodes commissioned at the moment,
  but QEMU computes are setup and ready to go. Here is how host list,
  service list and hypervisor list commands output look like (bmt hosts
  are computes with ironic virt driver):

  root@ctl01:~# openstack compute service list --service nova-compute
  +----+--------------+--------+----------+---------+-------+----------------------------+
  | ID | Binary       | Host   | Zone     | Status  | State | Updated At                 |
  +----+--------------+--------+----------+---------+-------+----------------------------+
  | 55 | nova-compute | bmt03  | foo_zone | enabled | up    | 2019-11-27T12:08:54.000000 |
  | 58 | nova-compute | bmt02  | nova     | enabled | up    | 2019-11-27T12:08:32.000000 |
  | 61 | nova-compute | bmt01  | nova     | enabled | up    | 2019-11-27T12:09:06.000000 |
  | 64 | nova-compute | cmp003 | nova     | enabled | up    | 2019-11-27T12:08:24.000000 |
  | 67 | nova-compute | cmp001 | nova     | enabled | up    | 2019-11-27T12:08:14.000000 |
  | 70 | nova-compute | cmp002 | nova     | enabled | up    | 2019-11-27T12:08:21.000000 |
  +----+--------------+--------+----------+---------+-------+----------------------------+
  root@ctl01:~# openstack host list | grep compute
  | bmt03     | compute     | foo_zone |
  | bmt02     | compute     | nova     |
  | bmt01     | compute     | nova     |
  | cmp003    | compute     | nova     |
  | cmp001    | compute     | nova     |
  | cmp002    | compute     | nova     |
  root@ctl01:~# openstack hypervisor list
  +----+--------------------------------------+-----------------+--------------+-------+
  | ID | Hypervisor Hostname                  | Hypervisor Type | Host IP      | State |
  +----+--------------------------------------+-----------------+--------------+-------+
  |  7 | cmp003.bm-cicd-queens-ovs-maas.local | QEMU            | 10.167.11.17 | up    |
  | 10 | cmp001.bm-cicd-queens-ovs-maas.local | QEMU            | 10.167.11.15 | up    |
  | 13 | cmp002.bm-cicd-queens-ovs-maas.local | QEMU            | 10.167.11.16 | up    |

  Then the following test is run:
  tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_remove_host_as_user
  [id-7a53af20-137a-4e44-a4ae-e19260e626d9,negative]

  The test tries to add a host to an aggregate and then remove it,
  addtion seems to be ok but removal fails with the following:

  Cannot remove host bmt03 in aggregate 52 (HTTP 404) (Request-ID: req-
  25acad79-b820-4889-a2a3-71b4a7913a86)

  In nova api something like the following can be seen:

  2019-11-27 12:11:42,263.263 2762 INFO nova.api.openstack.wsgi [req-
  80f752d1-0ce6-446a-ba92-cf815bfedd78 b31b4e9171154b1d9d1fec5af0ed5880
  6a33e1d31b53496885e19d7cba4390dd - default default] HTTP exception
  thrown: Host 'bmt01' is not mapped to any cell

  The problem seems to be in an inconsistent behaviour during adding the
  host to aggregate and removing it from aggregate, as during addition
  it allows host mapping to be missing, but does not allow it during
  host removal.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1854212/+subscriptions
Follow ups

[Bug 1854212] Re: For ironic nova-computes that have no nodes associated aggregate operations are inconsistent
From: OpenStack Infra, 2020-01-20