← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #77477

[Bug 1819910] [NEW] test_server_connectivity_live_migration intermittently fails with NoValidHost due to PUT /allocations failure

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Seen here:

http://logs.openstack.org/94/637594/1/gate/tempest-
slow/90def65/controller/logs/screen-n-super-
cond.txt.gz#_Mar_13_01_08_16_120854

Mar 13 01:08:16.120854 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: WARNING nova.scheduler.utils [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
Mar 13 01:08:16.121077 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: Traceback (most recent call last):
Mar 13 01:08:16.121335 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
Mar 13 01:08:16.121545 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     return func(*args, **kwargs)
Mar 13 01:08:16.121819 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 154, in select_destinations
Mar 13 01:08:16.122027 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     allocation_request_version, return_alternates)
Mar 13 01:08:16.122232 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
Mar 13 01:08:16.122518 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     allocation_request_version, return_alternates)
Mar 13 01:08:16.122719 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 244, in _schedule
Mar 13 01:08:16.122924 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     claimed_instance_uuids)
Mar 13 01:08:16.123128 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 281, in _ensure_sufficient_hosts
Mar 13 01:08:16.123322 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     raise exception.NoValidHost(reason=reason)
Mar 13 01:08:16.123623 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: NoValidHost: No valid host was found. There are not enough hosts available.
Mar 13 01:08:16.123829 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Mar 13 01:08:16.124035 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: WARNING nova.scheduler.utils [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] [instance: c2fd36f9-0b41-439c-9700-17aa0abe13c2] Setting instance to ACTIVE state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

And it looks like that is actually due to this:

http://logs.openstack.org/94/637594/1/gate/tempest-
slow/90def65/controller/logs/screen-n-super-
cond.txt.gz#_Mar_13_01_08_15_842358

Mar 13 01:08:15.842358 ubuntu-xenial-inap-mtl01-0003740112 nova-
conductor[25115]: WARNING nova.scheduler.client.report [None
req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-
TestNetworkAdvancedServerOps-840936673 tempest-
TestNetworkAdvancedServerOps-840936673] Failed to save allocation for
c2fd36f9-0b41-439c-9700-17aa0abe13c2. Got HTTP 400: {"errors":
[{"status": 400, "request_id": "req-2ba69e9f-63ac-
4f71-9dd0-2d5b97fcfbe8", "detail": "The server could not comply with the
request since it is either malformed or otherwise incorrect.\n\n JSON
does not validate: {} does not have enough properties  Failed validating
'minProperties' in
schema['properties']['allocations']['items']['properties']['resources']:
{'additionalProperties': False,      'minProperties': 1,
'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
'type': 'integer'}},      'type': 'object'}  On
instance['allocations'][0]['resources']:     {}  ", "title": "Bad
Request"}]}: DestinationHypervisorTooOld: The instance requires a newer
hypervisor version than has been provided.

The DestinationHypervisorTooOld error there is misleading, it looks like
the real failure was an allocation claim in placement:

http://logs.openstack.org/94/637594/1/gate/tempest-
slow/90def65/controller/logs/screen-placement-
api.txt.gz#_Mar_13_01_08_15_839694

Mar 13 01:08:15.839694 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: DEBUG nova.api.openstack.placement.wsgi_wrapper [req-a9345308-4e20-4976-ac6d-a1c529d14b16 req-2ba69e9f-63ac-4f71-9dd0-2d5b97fcfbe8 service placement] Placement API returning an error response: JSON does not validate: {} does not have enough properties
Mar 13 01:08:15.839924 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: Failed validating 'minProperties' in schema['properties']['allocations']['items']['properties']['resources']:
Mar 13 01:08:15.840362 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:     {'additionalProperties': False,
Mar 13 01:08:15.840584 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'minProperties': 1,
Mar 13 01:08:15.840799 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
Mar 13 01:08:15.841007 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:                                             'type': 'integer'}},
Mar 13 01:08:15.841218 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'type': 'object'}
Mar 13 01:08:15.841424 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: On instance['allocations'][0]['resources']:
Mar 13 01:08:15.841624 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:     {} {{(pid=18500) call_func /opt/stack/nova/nova/api/openstack/placement/wsgi_wrapper.py:31}}

** Affects: nova
     Importance: Undecided
         Status: Confirmed

** Also affects: nova
   Importance: Undecided
       Status: New

** No longer affects: tempest

** Summary changed:

- test_server_connectivity_live_migration intermittently fails with NoValidHost
+ test_server_connectivity_live_migration intermittently fails with NoValidHost due to PUT /allocations failure

** Changed in: nova
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1819910

Title:
  test_server_connectivity_live_migration intermittently fails with
  NoValidHost due to PUT /allocations failure

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/94/637594/1/gate/tempest-
  slow/90def65/controller/logs/screen-n-super-
  cond.txt.gz#_Mar_13_01_08_16_120854

  Mar 13 01:08:16.120854 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: WARNING nova.scheduler.utils [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
  Mar 13 01:08:16.121077 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: Traceback (most recent call last):
  Mar 13 01:08:16.121335 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
  Mar 13 01:08:16.121545 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     return func(*args, **kwargs)
  Mar 13 01:08:16.121819 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 154, in select_destinations
  Mar 13 01:08:16.122027 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     allocation_request_version, return_alternates)
  Mar 13 01:08:16.122232 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
  Mar 13 01:08:16.122518 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     allocation_request_version, return_alternates)
  Mar 13 01:08:16.122719 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 244, in _schedule
  Mar 13 01:08:16.122924 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     claimed_instance_uuids)
  Mar 13 01:08:16.123128 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 281, in _ensure_sufficient_hosts
  Mar 13 01:08:16.123322 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     raise exception.NoValidHost(reason=reason)
  Mar 13 01:08:16.123623 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: NoValidHost: No valid host was found. There are not enough hosts available.
  Mar 13 01:08:16.123829 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
  Mar 13 01:08:16.124035 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: WARNING nova.scheduler.utils [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] [instance: c2fd36f9-0b41-439c-9700-17aa0abe13c2] Setting instance to ACTIVE state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

  And it looks like that is actually due to this:

  http://logs.openstack.org/94/637594/1/gate/tempest-
  slow/90def65/controller/logs/screen-n-super-
  cond.txt.gz#_Mar_13_01_08_15_842358

  Mar 13 01:08:15.842358 ubuntu-xenial-inap-mtl01-0003740112 nova-
  conductor[25115]: WARNING nova.scheduler.client.report [None
  req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-
  TestNetworkAdvancedServerOps-840936673 tempest-
  TestNetworkAdvancedServerOps-840936673] Failed to save allocation for
  c2fd36f9-0b41-439c-9700-17aa0abe13c2. Got HTTP 400: {"errors":
  [{"status": 400, "request_id": "req-2ba69e9f-63ac-
  4f71-9dd0-2d5b97fcfbe8", "detail": "The server could not comply with
  the request since it is either malformed or otherwise incorrect.\n\n
  JSON does not validate: {} does not have enough properties  Failed
  validating 'minProperties' in
  schema['properties']['allocations']['items']['properties']['resources']:
  {'additionalProperties': False,      'minProperties': 1,
  'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
  'type': 'integer'}},      'type': 'object'}  On
  instance['allocations'][0]['resources']:     {}  ", "title": "Bad
  Request"}]}: DestinationHypervisorTooOld: The instance requires a
  newer hypervisor version than has been provided.

  The DestinationHypervisorTooOld error there is misleading, it looks
  like the real failure was an allocation claim in placement:

  http://logs.openstack.org/94/637594/1/gate/tempest-
  slow/90def65/controller/logs/screen-placement-
  api.txt.gz#_Mar_13_01_08_15_839694

  Mar 13 01:08:15.839694 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: DEBUG nova.api.openstack.placement.wsgi_wrapper [req-a9345308-4e20-4976-ac6d-a1c529d14b16 req-2ba69e9f-63ac-4f71-9dd0-2d5b97fcfbe8 service placement] Placement API returning an error response: JSON does not validate: {} does not have enough properties
  Mar 13 01:08:15.839924 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: Failed validating 'minProperties' in schema['properties']['allocations']['items']['properties']['resources']:
  Mar 13 01:08:15.840362 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:     {'additionalProperties': False,
  Mar 13 01:08:15.840584 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'minProperties': 1,
  Mar 13 01:08:15.840799 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
  Mar 13 01:08:15.841007 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:                                             'type': 'integer'}},
  Mar 13 01:08:15.841218 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'type': 'object'}
  Mar 13 01:08:15.841424 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: On instance['allocations'][0]['resources']:
  Mar 13 01:08:15.841624 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:     {} {{(pid=18500) call_func /opt/stack/nova/nova/api/openstack/placement/wsgi_wrapper.py:31}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1819910/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next