yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1784353] Re: Rescheduled boot from volume instances fail due to the premature removal of their attachments

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 02 Aug 2018 13:20:32 -0000
Reply-to: Bug 1784353 <1784353@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Bug 1488111 is what I was thinking about, but Lee clarified the issue
for me. The scenario is like:

1. spawn on host1 fails, reschedule to host2
2. prep_block_devices fails on host2 because of the volume attachment issue mentioned

** Also affects: nova/queens
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1784353

Title:
  Rescheduled boot from volume instances fail due to the premature
  removal of their attachments

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) queens series:
  New

Bug description:
  Description
  ===========
  This is caused by the cleanup code within the compute layer (_shutdown_instance) removing all volume attachments associated with an instance with no attempt being made to recreate these ahead of the instance being rescheduled.

  Steps to reproduce
  ==================
  - Attempt to boot an instance with volumes attached.
  - Ensure spawn() fails, for example by stopping the l2 network agent services on the compute host.

  Expected result
  ===============
  The instance is reschedule to another compute host and boots correctly.

  Actual result
  =============
  The instance fails to boot on all hosts that is rescheduled to due to a missing volume attachment.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     bf497cc47497d3a5603bf60de652054ac5ae1993

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM, however this shouldn't matter.

  3. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  4. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1] Traceback (most recent call last):  
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1579, in _prep_block_device
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     wait_func=self._await_block_device_map_created)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 837, in attach_block_devices
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     _log_and_attach(device)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 834, in _log_and_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     bdm.attach(*attach_args, **attach_kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     ret_val = method(obj, context, *args, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 617, in attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     virt_driver, do_driver_attach)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     return f(*args, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 614, in _do_locked_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     self._do_attach(*args, **_kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 599, in _do_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     do_driver_attach)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 513, in _volume_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     self['mount_device'])['connection_info']
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     res = method(self, ctx, *args, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 418, in wrapper
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     attachment_id=attachment_id))
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 450, in _reraise
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     res = method(self, ctx, attachment_id, *args, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 824, in attachment_update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     'code': getattr(ex, 'code', None)})
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     self.force_reraise()
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     six.reraise(self.type_, self.value, self.tb)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 814, in attachment_update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     attachment_id, _connector)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 67, in update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     resp = self._update('/attachments/%s' % id, body)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 344, in _update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     resp, body = self.api.client.put(url, body=body, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 206, in put
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     return self._cs_request(url, 'PUT', **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     return self.request(url, method, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]     raise exceptions.from_response(resp, body)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1] VolumeAttachmentNotFound: Volume attachment 11 [details]d518a9-16d4-4ccb-9487-ec2b35834945 could not be found.
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: d48c9894-2ba2-4752-bae5-36c437933ff1]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1784353/+subscriptions
References

[Bug 1784353] [NEW] Rescheduled boot from volume instances fail due to the premature removal of their attachments
From: Lee Yarwood, 2018-07-30