yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1966387] Re: Cannot recover the instance after finish_resize failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1966387@xxxxxxxxxxxxxxxxxx>
Date: Fri, 22 Apr 2022 15:27:31 -0000
Reply-to: Bug 1966387 <1966387@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
If you have compute servers with different HW capabilities and you want
that the nova scheduler consider the HW differences then we suggest to
group the servers with the same HW capabilities into host aggregates,
mark the capability on the aggregate with metadata, and request those
capabilities via flavor extra_spec. See the documentation for more
details[1].

I set this to Invalid as this is not a bug but a config error. If you
disagree then feel free to set this back to New.

[1] https://docs.openstack.org/nova/latest/admin/aggregates.html

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1966387

Title:
  Cannot recover the instance after finish_resize failure

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Cannot recover the instance after finish_resize failure

  
  Description
  ===========
  An instance which has attached fibre channel volumes must resize
  or migrate to a hba-supported host, or the instance will result
  in error state on the destination host after being resized or
  migrated. 
  And in that scenaria, the fibre channel volumes have already been 
  detached succeessfully and cannot be attached to the destination
  host and the instance cannot start even by hard rebooting it since
  it is unable to update attachment successfully due to the lack of 
  hba support. 

  
  Steps to reproduce
  ==================
  1. Prepare two compute hosts and one host has hba support, the other
  hasn't.
  2. Create a server on the hba-supported host and attach a fibre channel 
  volume to it.
  3. Stop the server and migrate it to the other host.

  
  Expected result
  ===============
  The server cannot be migrated to the other host since that host has not
  hba support.

  
  Actual result
  =============
  The server has been migrated to the other host. However, the server is in
  error state and cannot be recoverd by any method.

  Environment
  ===========
  Victoria version of OpenStack

  Logs & Configs
  ==============
  2022-02-21 11:28:36.113 3918565 INFO nova.compute.claims [None req-97574e72-e0b0-4906-8c34-7140245bf951 484e61647b5e4dff8719956bc86b74bc ffc6dd5c50574220a6f4e8b093470e6f - default default] [instance: c4f69427-7776-4a6d-b488-2055531791f3] Claim successful on node compute-002
  2022-02-21 11:28:36.238 3918565 INFO nova.compute.resource_tracker [None req-97574e72-e0b0-4906-8c34-7140245bf951 484e61647b5e4dff8719956bc86b74bc ffc6dd5c50574220a6f4e8b093470e6f - default default] [instance: c4f69427-7776-4a6d-b488-2055531791f3] Updating resource usage from migration 156d8dd8-7fd3-4fd8-b4db-197bb45db116
  2022-02-21 11:28:36.674 3918565 INFO nova.compute.manager [None req-97574e72-e0b0-4906-8c34-7140245bf951 484e61647b5e4dff8719956bc86b74bc ffc6dd5c50574220a6f4e8b093470e6f - default default] [instance: c4f69427-7776-4a6d-b488-2055531791f3] Migrating
  2022-02-21 11:28:40.213 3918565 INFO nova.network.neutron [None req-97574e72-e0b0-4906-8c34-7140245bf951 484e61647b5e4dff8719956bc86b74bc ffc6dd5c50574220a6f4e8b093470e6f - default default] [instance: c4f69427-7776-4a6d-b488-2055531791f3] Updating port 4c78d66a-908c-487c-9d53-8626e6ccc0ff with attributes {'binding:host_id': 'compute-002', 'device_owner': 'compute:az-stackdev'}
  2022-02-21 11:28:42.798 3918565 ERROR nova.volume.cinder [None req-97574e72-e0b0-4906-8c34-7140245bf951 484e61647b5e4dff8719956bc86b74bc ffc6dd5c50574220a6f4e8b093470e6f - default default] Update attachment failed for attachment fde2cca1-66cd-47fa-9f52-3ee43d15b096. Error: Unable to update attachment.(Invalid input received: Connector doesn't have required information: wwpns). (HTTP 500) (Request-ID: req-d9bc8a2c-4b28-46f4-9d77-9d4c69d57b60) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid input received: Connector doesn't have required information: wwpns). (HTTP 500) (Request-ID: req-d9bc8a2c-4b28-46f4-9d77-9d4c69d57b60)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [None req-97574e72-e0b0-4906-8c34-7140245bf951 484e61647b5e4dff8719956bc86b74bc ffc6dd5c50574220a6f4e8b093470e6f - default default] [instance: c4f69427-7776-4a6d-b488-2055531791f3] Setting instance vm_state to ERROR: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid input received: Connector doesn't have required information: wwpns). (HTTP 500) (Request-ID: req-d9bc8a2c-4b28-46f4-9d77-9d4c69d57b60)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3] Traceback (most recent call last):
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 10704, in _error_out_instance_on_exception
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     yield
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 5949, in _finish_resize_helper
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     network_info = self._finish_resize(context, instance, migration,
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 5853, in _finish_resize
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     self._update_volume_attachments(context, instance, bdms)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 5787, in _update_volume_attachments
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     self.volume_api.attachment_update(
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 397, in wrapper
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     res = method(self, ctx, *args, **kwargs)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 448, in wrapper
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     res = method(self, ctx, attachment_id, *args, **kwargs)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 880, in attachment_update
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     LOG.error('Update attachment failed for attachment '
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     self.force_reraise()
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     six.reraise(self.type_, self.value, self.tb)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/six.py", line 703, in reraise
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     raise value
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 872, in attachment_update
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     attachment_ref = cinderclient(
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     return method.func(obj, *args, **kwargs)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 74, in update
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     resp = self._update('/attachments/%s' % id, body)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/base.py", line 314, in _update
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     resp, body = self.api.client.put(url, body=body, **kwargs)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 221, in put
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     return self._cs_request(url, 'PUT', **kwargs)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 206, in _cs_request
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     return self.request(url, method, **kwargs)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]   File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 192, in request
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]     raise exceptions.from_response(resp, body)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid input received: Connector doesn't have required information: wwpns). (HTTP 500) (Request-ID: req-d9bc8a2c-4b28-46f4-9d77-9d4c69d57b60)
  2022-02-21 11:28:42.799 3918565 ERROR nova.compute.manager [instance: c4f69427-7776-4a6d-b488-2055531791f3]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1966387/+subscriptions
References

[Bug 1966387] [NEW] Cannot recover the instance after finish_resize failure
From: zhaoleilc, 2022-03-25