yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1661258

Title:
  Deleted ironic node has an inventory in nova_api database

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Running latest devstack, ironic and nova, I get the following error
  when I request an instance:

  | fault                                | {"message": "Node 6cc8803d-4e77-4948-b653-663d8d5e52b7 could not be found. (HTTP 404)", "code": 500, "details": "  File \"/opt/stack/nova/nova/compute/manager.py\", line 1780, in _do_build_and_run_instance |
  |                                      |     filter_properties)                                                                                                                                                                                        |
  |                                      |   File \"/opt/stack/nova/nova/compute/manager.py\", line 2016, in _build_and_run_instance                                                                                                                     |
  |                                      |     instance_uuid=instance.uuid, reason=six.text_type(e))                                                                                                                                                     |
  |                                      | ", "created": "2017-02-02T13:42:01Z"}                                                                                                                                                                         |

  On ironic side, this node was indeed deleted, it is also deleted from
  nova.compute_nodes table:

  | created_at          | updated_at          | deleted_at          | id | service_id | vcpus | memory_mb | local_gb | vcpus_used | memory_mb_used | local_gb_used | hypervisor_type | hypervisor_version | cpu_info | disk_available_least | free_ram_mb | free_disk_gb | current_workload | running_vms | hypervisor_hostname                  | deleted | host_ip        | supported_instances              | pci_stats                                                                                                                                                                         | metrics | extra_resources | stats                  | numa_topology | host   | ram_allocation_ratio | cpu_allocation_ratio | uuid                                 | disk_allocation_ratio |
  ...................................................
  | 2017-02-02 12:20:27 | 2017-02-02 13:20:15 | 2017-02-02 13:21:15 |  2 |       NULL |     1 |      1536 |       10 |          0 |              0 |             0 | ironic          |                  1 |          |                   10 |        1536 |           10 |                0 |           0 | 6cc8803d-4e77-4948-b653-663d8d5e52b7 |       2 | 192.168.122.22 | [["x86_64", "baremetal", "hvm"]] | {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": []}, "nova_object.namespace": "nova"} | []      | NULL            | {"cpu_arch": "x86_64"} | NULL          | ubuntu |                    1 |                    0 | 035be695-0797-44b3-930b-42349e40579e |                     0 |

  But in nova_api.inventories it's still there:

  | created_at          | updated_at | id | resource_provider_id | resource_class_id | total | reserved | min_unit | max_unit | step_size | allocation_ratio |
  ..........................
  | 2017-02-02 13:20:14 | NULL       | 13 |                    2 |                 0 |     1 |        0 |        1 |        1 |         1 |               16 |
  | 2017-02-02 13:20:14 | NULL       | 14 |                    2 |                 1 |  1536 |        0 |        1 |     1536 |         1 |                1 |
  | 2017-02-02 13:20:14 | NULL       | 15 |                    2 |                 2 |    10 |        0 |        1 |       10 |         1 |                1 |

  nova_api.resource_providers bit:
  | created_at          | updated_at          | id | uuid                                 | name                                 | generation | can_host |
  .........................
  | 2017-02-02 12:20:27 | 2017-02-02 13:20:14 |  2 | 035be695-0797-44b3-930b-42349e40579e | 6cc8803d-4e77-4948-b653-663d8d5e52b7 |          7 |        0 |

  Waiting for resource tracker run did not help, node's been deleted for
  ~30 minutes already and the inventory is still there.

  Code versions:
  Devstack commit debc695ddfc8b7b2aeb53c01c624e15f69ed9fa2 Updated from generate-devstack-plugins-list.
  Nova commit 5dad7eaef7f8562425cce6b233aed610ca2d3148 Merge "doc: update the man page entry for nova-manage db sync"
  Ironic commit 5071b99835143ebcae876432e2982fd27faece10 Merge "Remove deprecated heartbeat policy check"

  If it is anyhow relevant, I also run two nova-computes on the same
  host, I've set host=test for the second one, other than that all
  configs are the same. I was trying to reproduce another cell-related
  issue, and was creating/deleting ironic nodes, so that they map to the
  second nova-compute by the hash_ring.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1661258/+subscriptions