yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Looks like the allocations are swapped to the migration record here:

Mar 13 01:08:15.371920 ubuntu-xenial-inap-mtl01-0003740112 nova-
conductor[25115]: DEBUG nova.conductor.tasks.migrate [None
req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-
TestNetworkAdvancedServerOps-840936673 tempest-
TestNetworkAdvancedServerOps-840936673] Created allocations for
migration 1745e761-2a3b-4faa-921e-cc11a143ba85 on
f73de938-3b21-45c4-8227-80c4b0bab1f0 {{(pid=26101)
replace_allocation_with_migration
/opt/stack/nova/nova/conductor/tasks/migrate.py:75}}

Then I see this:

Mar 13 01:08:15.610924 ubuntu-xenial-inap-mtl01-0003740112 nova-
conductor[25115]: DEBUG nova.conductor.tasks.live_migrate [None
req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-
TestNetworkAdvancedServerOps-840936673 tempest-
TestNetworkAdvancedServerOps-840936673] Skipping host: ubuntu-xenial-
inap-mtl01-0003740112 because: The instance requires a newer hypervisor
version than has been provided. {{(pid=26101) _find_destination
/opt/stack/nova/nova/conductor/tasks/live_migrate.py:339}}

That seems very weird because this is not a grenade job so the 2
computes should have the same hypervisor version.

Looking at the qemu versions on the hosts:

http://logs.openstack.org/94/637594/1/gate/tempest-
slow/90def65/compute1/logs/dpkg-l.txt.gz

ii  qemu-system                         1:2.10+dfsg-0ubuntu3.8~cloud1

http://logs.openstack.org/94/637594/1/gate/tempest-
slow/90def65/controller/logs/dpkg-l.txt.gz

ii  qemu-system                         1:2.5+dfsg-5ubuntu10.34

That is definitely wrong - it looks like the subnode is running with
ubuntu bionic.

** Also affects: tempest
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1819910

Title:
  test_server_connectivity_live_migration intermittently fails with
  NoValidHost due to DestinationHypervisorTooOld

Status in OpenStack Compute (nova):
  Confirmed
Status in tempest:
  New

Bug description:
  Seen here:

  http://logs.openstack.org/94/637594/1/gate/tempest-
  slow/90def65/controller/logs/screen-n-super-
  cond.txt.gz#_Mar_13_01_08_16_120854

  Mar 13 01:08:16.120854 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: WARNING nova.scheduler.utils [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
  Mar 13 01:08:16.121077 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: Traceback (most recent call last):
  Mar 13 01:08:16.121335 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
  Mar 13 01:08:16.121545 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     return func(*args, **kwargs)
  Mar 13 01:08:16.121819 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 154, in select_destinations
  Mar 13 01:08:16.122027 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     allocation_request_version, return_alternates)
  Mar 13 01:08:16.122232 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
  Mar 13 01:08:16.122518 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     allocation_request_version, return_alternates)
  Mar 13 01:08:16.122719 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 244, in _schedule
  Mar 13 01:08:16.122924 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     claimed_instance_uuids)
  Mar 13 01:08:16.123128 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 281, in _ensure_sufficient_hosts
  Mar 13 01:08:16.123322 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]:     raise exception.NoValidHost(reason=reason)
  Mar 13 01:08:16.123623 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: NoValidHost: No valid host was found. There are not enough hosts available.
  Mar 13 01:08:16.123829 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
  Mar 13 01:08:16.124035 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: WARNING nova.scheduler.utils [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] [instance: c2fd36f9-0b41-439c-9700-17aa0abe13c2] Setting instance to ACTIVE state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

  And it looks like that is actually due to this:

  http://logs.openstack.org/94/637594/1/gate/tempest-
  slow/90def65/controller/logs/screen-n-super-
  cond.txt.gz#_Mar_13_01_08_15_842358

  Mar 13 01:08:15.842358 ubuntu-xenial-inap-mtl01-0003740112 nova-
  conductor[25115]: WARNING nova.scheduler.client.report [None
  req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-
  TestNetworkAdvancedServerOps-840936673 tempest-
  TestNetworkAdvancedServerOps-840936673] Failed to save allocation for
  c2fd36f9-0b41-439c-9700-17aa0abe13c2. Got HTTP 400: {"errors":
  [{"status": 400, "request_id": "req-2ba69e9f-63ac-
  4f71-9dd0-2d5b97fcfbe8", "detail": "The server could not comply with
  the request since it is either malformed or otherwise incorrect.\n\n
  JSON does not validate: {} does not have enough properties  Failed
  validating 'minProperties' in
  schema['properties']['allocations']['items']['properties']['resources']:
  {'additionalProperties': False,      'minProperties': 1,
  'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
  'type': 'integer'}},      'type': 'object'}  On
  instance['allocations'][0]['resources']:     {}  ", "title": "Bad
  Request"}]}: DestinationHypervisorTooOld: The instance requires a
  newer hypervisor version than has been provided.

  The DestinationHypervisorTooOld error there is misleading, it looks
  like the real failure was an allocation claim in placement:

  http://logs.openstack.org/94/637594/1/gate/tempest-
  slow/90def65/controller/logs/screen-placement-
  api.txt.gz#_Mar_13_01_08_15_839694

  Mar 13 01:08:15.839694 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: DEBUG nova.api.openstack.placement.wsgi_wrapper [req-a9345308-4e20-4976-ac6d-a1c529d14b16 req-2ba69e9f-63ac-4f71-9dd0-2d5b97fcfbe8 service placement] Placement API returning an error response: JSON does not validate: {} does not have enough properties
  Mar 13 01:08:15.839924 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: Failed validating 'minProperties' in schema['properties']['allocations']['items']['properties']['resources']:
  Mar 13 01:08:15.840362 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:     {'additionalProperties': False,
  Mar 13 01:08:15.840584 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'minProperties': 1,
  Mar 13 01:08:15.840799 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
  Mar 13 01:08:15.841007 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:                                             'type': 'integer'}},
  Mar 13 01:08:15.841218 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:      'type': 'object'}
  Mar 13 01:08:15.841424 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]: On instance['allocations'][0]['resources']:
  Mar 13 01:08:15.841624 ubuntu-xenial-inap-mtl01-0003740112 devstack@placement-api.service[18498]:     {} {{(pid=18500) call_func /opt/stack/nova/nova/api/openstack/placement/wsgi_wrapper.py:31}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1819910/+subscriptions