yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2019460] Re: nova-compute 23.2.2-0ubuntu1~cloud2 unable to detach volumes

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Corey Bryant <2019460@xxxxxxxxxxxxxxxxxx>
Date: Sat, 13 May 2023 13:47:22 -0000
Reply-to: Bug 2019460 <2019460@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
It looks like some volume drivers that exist prior to xena are missing
the force parameter. I'll get a fix going for this asap.

** Also affects: cloud-archive/wallaby
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/ussuri
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/victoria
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/wallaby
       Status: New => Triaged

** Changed in: cloud-archive/victoria
       Status: New => Triaged

** Changed in: cloud-archive/ussuri
       Status: New => Triaged

** Changed in: cloud-archive/wallaby
   Importance: Undecided => Critical

** Changed in: cloud-archive/victoria
   Importance: Undecided => Critical

** Changed in: cloud-archive/ussuri
   Importance: Undecided => Critical

** Changed in: cloud-archive
       Status: New => Invalid

** Also affects: nova (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: nova (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Changed in: nova (Ubuntu Focal)
       Status: New => Triaged

** Changed in: nova (Ubuntu Focal)
   Importance: Undecided => Critical

** Changed in: nova (Ubuntu)
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2019460

Title:
  nova-compute 23.2.2-0ubuntu1~cloud2 unable to detach volumes

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in Ubuntu Cloud Archive wallaby series:
  Triaged
Status in OpenStack Compute (nova):
  New
Status in nova package in Ubuntu:
  Invalid
Status in nova source package in Focal:
  Triaged

Bug description:
  The following packages were updated on Wallaby compute nodes to fix
  https://security.openstack.org/ossa/OSSA-2023-003.html:

  python3-nova:amd64 (3:23.2.2-0ubuntu1~cloud1, 3:23.2.2-0ubuntu1~cloud2),
  python3-os-brick:amd64 (4.3.3-0ubuntu1~cloud0, 4.3.3-0ubuntu1~cloud1),
  nova-compute-libvirt:amd64 (3:23.2.2-0ubuntu1~cloud1, 3:23.2.2-0ubuntu1~cloud2),
  nova-common:amd64 (3:23.2.2-0ubuntu1~cloud1, 3:23.2.2-0ubuntu1~cloud2),
  os-brick-common:amd64 (4.3.3-0ubuntu1~cloud0, 4.3.3-0ubuntu1~cloud1),
  nova-compute-kvm:amd64 (3:23.2.2-0ubuntu1~cloud1, 3:23.2.2-0ubuntu1~cloud2),
  nova-compute:amd64 (3:23.2.2-0ubuntu1~cloud1, 3:23.2.2-0ubuntu1~cloud2)

  nova-compute is now unable to detach volumes from instances:

  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server [req-470d3e0e-e59c-40c5-9597-6649c08add16 046191f8ebfd4695b3387a5ead3a9a55 85945271df8b4a6f9d37c37e4e52958d - default default] Exception during message handling: TypeError: disconnect_volume() got an unexpected keyword argument 'force'
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self.force_reraise()
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server raise self.value
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1434, in decorated_function
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 211, in decorated_function
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self.force_reraise()
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server raise self.value
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 200, in decorated_function
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7195, in detach_volume
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server do_detach_volume(context, volume_id, instance, attachment_id)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7192, in do_detach_volume
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self._detach_volume(context, bdm, instance,
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7143, in _detach_volume
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server driver_bdm.detach(context, instance, self.volume_api, self.driver,
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 476, in detach
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self._do_detach(context, instance, volume_api, virt_driver,
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 408, in _do_detach
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self.driver_detach(context, instance, volume_api, virt_driver)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 347, in driver_detach
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server volume_api.roll_detaching(context, volume_id)
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self.force_reraise()
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server raise self.value
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 328, in driver_detach
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server virt_driver.detach_volume(context, connection_info, instance, mp,
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 2592, in detach_volume
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server self._disconnect_volume(context, connection_info, instance,
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1862, in _disconnect_volume
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server TypeError: disconnect_volume() got an unexpected keyword argument 'force'
  2023-05-13 05:53:00.128 3219193 ERROR oslo_messaging.rpc.server

  Looks like it doesn't know about the "force" keyword that's being
  passed. This breaks basic volume functionality.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2019460/+subscriptions
References

[Bug 2019460] [NEW] nova-compute 23.2.2-0ubuntu1~cloud2 unable to detach volumes
From: Zakhar Kirpichenko, 2023-05-13