yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Public bug reported:

I found this when trying to recreate bug 1716706.

https://bugs.launchpad.net/nova/+bug/1716706/comments/4

Basically I can get conductor to fail the quota recheck and go to set
the instance into ERROR state but it fails to find the instance since we
don't have the cell context targeted:

Sep 13 17:58:26 devstack-queens nova-conductor[3129]: WARNING nova.scheduler.utils [None req-90a115b2-5838-4be2-afe2-a3b755015e19 demo demo] [instance: 888925b0-164a-4d4a-bb6c-c0426f904e95] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances
Sep 13 17:58:26 devstack-queens nova-conductor[3129]: ERROR root [None req-90a115b2-5838-4be2-afe2-a3b755015e19 demo demo] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/conductor/manager.py", line 1003, in schedule_and_build_instances\n orig_num_req=len(build_requests))\n', ' File "/opt/stack/nova/nova/compute/utils.py", line 764, in check_num_instances_quota\n allowed=total_alloweds)\n', 'TooManyInstances: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances\n']: InstanceNotFound: Instance 888925b0-164a-4d4a-bb6c-c0426f904e95 could not be found.
Sep 13 17:58:26 devstack-queens nova-conductor[3129]: ERROR oslo_messaging.rpc.server [None req-90a115b2-5838-4be2-afe2-a3b755015e19 demo demo] Exception during message handling: InstanceNotFound: Instance 888925b0-164a-4d4a-bb6c-c0426f904e95 could not be found.
Sep 13 17:58:26 devstack-queens nova-conductor[3129]: ERROR oslo_messaging.rpc.server InstanceNotFound: Instance 888925b0-164a-4d4a-bb6c-c0426f904e95 could not be found.

Because we don't target the cell when updating the instance.

https://github.com/openstack/nova/blob/cfdec41eeec5fab220702efefdaafc45559aeb14/nova/conductor/manager.py#L1168

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: nova/pike
     Importance: Undecided
         Status: New


** Tags: quotas

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1717000

Title:
  InstanceNotFound prevents putting over-quota instance into ERROR state

Status in OpenStack Compute (nova):
  New
Status in OpenStack Compute (nova) pike series:
  New

Bug description:
  I found this when trying to recreate bug 1716706.

  https://bugs.launchpad.net/nova/+bug/1716706/comments/4

  Basically I can get conductor to fail the quota recheck and go to set
  the instance into ERROR state but it fails to find the instance since
  we don't have the cell context targeted:

  Sep 13 17:58:26 devstack-queens nova-conductor[3129]: WARNING nova.scheduler.utils [None req-90a115b2-5838-4be2-afe2-a3b755015e19 demo demo] [instance: 888925b0-164a-4d4a-bb6c-c0426f904e95] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances
  Sep 13 17:58:26 devstack-queens nova-conductor[3129]: ERROR root [None req-90a115b2-5838-4be2-afe2-a3b755015e19 demo demo] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/conductor/manager.py", line 1003, in schedule_and_build_instances\n orig_num_req=len(build_requests))\n', ' File "/opt/stack/nova/nova/compute/utils.py", line 764, in check_num_instances_quota\n allowed=total_alloweds)\n', 'TooManyInstances: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances\n']: InstanceNotFound: Instance 888925b0-164a-4d4a-bb6c-c0426f904e95 could not be found.
  Sep 13 17:58:26 devstack-queens nova-conductor[3129]: ERROR oslo_messaging.rpc.server [None req-90a115b2-5838-4be2-afe2-a3b755015e19 demo demo] Exception during message handling: InstanceNotFound: Instance 888925b0-164a-4d4a-bb6c-c0426f904e95 could not be found.
  Sep 13 17:58:26 devstack-queens nova-conductor[3129]: ERROR oslo_messaging.rpc.server InstanceNotFound: Instance 888925b0-164a-4d4a-bb6c-c0426f904e95 could not be found.

  Because we don't target the cell when updating the instance.

  https://github.com/openstack/nova/blob/cfdec41eeec5fab220702efefdaafc45559aeb14/nova/conductor/manager.py#L1168

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1717000/+subscriptions