yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1861071] Re: Failing to extend an attached encrypted volume

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <1861071@xxxxxxxxxxxxxxxxxx>
Date: Tue, 15 Sep 2020 10:12:12 -0000
Reply-to: Bug 1861071 <1861071@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: In Progress => Fix Released

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1861071

Title:
  Failing to extend an attached encrypted volume

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description:
  While extending an attached encrypted volume, Cinder reports volume had been extended. 
  However inside the instance lsblk doesn't show new extended size. 
  This works fine for none encrypted attached volumes. 

  The error on Nova says suggest this isn't cause by Cinder or it's
  backend thus also Cinder backend agnostic.

  If this won't be fixed or supported, we should block or at the very
  least document that it's a known limitation.

  
  Steps to reproduce:
  1. Create an encrypted volume say 1G
  2. Attach said volume to instance
  3. check lsblk inside instance should report ~1G
  4. Cinder extend attached encrypted volume example, notice needs Cinder micro version
  cinder --os-volume-api-version 3.59 extend  [VolID] 3
  5. Recheck cinder show [VolID] notice volume size was extended
  6. Inside instance lsblk doesn't show the new extended size. 
  7. Notice Nova error (seen later below) 

  
  Version:
  openstack-nova-compute-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
  openstack-nova-migration-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
  python3-novaclient-15.1.0-0.20190919143437.cd396b8.el8ost.noarch
  python3-nova-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
  puppet-nova-15.4.1-0.20191126042922.b1bb388.el8ost.noarch
  openstack-nova-common-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch

  
  Expected result:
  lsblk should report new size, as it successfully does in case of extending an attached none encrypted volume.

  Actual result:
  Cinder show volume size increase
  But lsblk doesn't grow. 

  If not supported this should be blocked/alerted on logs.
  Or at the very least documented.  


  Traceback from nova about failures in resize:

  var/log/containers/nova/nova-compute.log:71:2020-01-28 07:14:17.394 7 ERROR nova.virt.libvirt.driver [instance: f93e235e-30e9-4bdc-ad48-8e972552bbb5] 
  /var/log/containers/nova/nova-compute.log:73:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server [req-ef88174f-6762-4842-a64c-76074a848ac7 b753e5c55ba94950b5463ae41bb623ab 08d5a55b6cad4413910abc863b4a2b15 - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
  /var/log/containers/nova/nova-compute.log:74:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  /var/log/containers/nova/nova-compute.log:75:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  /var/log/containers/nova/nova-compute.log:76:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  /var/log/containers/nova/nova-compute.log:77:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
  /var/log/containers/nova/nova-compute.log:78:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  /var/log/containers/nova/nova-compute.log:79:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  /var/log/containers/nova/nova-compute.log:80:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  /var/log/containers/nova/nova-compute.log:81:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  /var/log/containers/nova/nova-compute.log:82:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  /var/log/containers/nova/nova-compute.log:83:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  /var/log/containers/nova/nova-compute.log:84:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  /var/log/containers/nova/nova-compute.log:85:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  /var/log/containers/nova/nova-compute.log:86:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  /var/log/containers/nova/nova-compute.log:87:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  /var/log/containers/nova/nova-compute.log:88:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
  /var/log/containers/nova/nova-compute.log:89:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  /var/log/containers/nova/nova-compute.log:90:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  /var/log/containers/nova/nova-compute.log:91:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9223, in external_instance_event
  /var/log/containers/nova/nova-compute.log:92:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
  /var/log/containers/nova/nova-compute.log:93:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
  /var/log/containers/nova/nova-compute.log:94:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  /var/log/containers/nova/nova-compute.log:95:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
  /var/log/containers/nova/nova-compute.log:96:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  /var/log/containers/nova/nova-compute.log:97:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  /var/log/containers/nova/nova-compute.log:98:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  /var/log/containers/nova/nova-compute.log:99:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  /var/log/containers/nova/nova-compute.log:100:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  /var/log/containers/nova/nova-compute.log:101:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  /var/log/containers/nova/nova-compute.log:102:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
  /var/log/containers/nova/nova-compute.log:103:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
  /var/log/containers/nova/nova-compute.log:104:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  /var/log/containers/nova/nova-compute.log:105:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9081, in extend_volume
  /var/log/containers/nova/nova-compute.log:106:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     bdm.volume_size * units.Gi)
  /var/log/containers/nova/nova-compute.log:107:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2048, in extend_volume
  /var/log/containers/nova/nova-compute.log:108:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     instance=instance)
  /var/log/containers/nova/nova-compute.log:109:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  /var/log/containers/nova/nova-compute.log:110:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  /var/log/containers/nova/nova-compute.log:111:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  /var/log/containers/nova/nova-compute.log:112:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  /var/log/containers/nova/nova-compute.log:113:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  /var/log/containers/nova/nova-compute.log:114:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
  /var/log/containers/nova/nova-compute.log:115:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2037, in extend_volume
  /var/log/containers/nova/nova-compute.log:116:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     dev.resize(new_size // units.Ki)
  /var/log/containers/nova/nova-compute.log:117:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 809, in resize
  /var/log/containers/nova/nova-compute.log:118:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self._guest._domain.blockResize(self._disk, size_kb)
  /var/log/containers/nova/nova-compute.log:119:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
  /var/log/containers/nova/nova-compute.log:120:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  /var/log/containers/nova/nova-compute.log:121:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
  /var/log/containers/nova/nova-compute.log:122:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  /var/log/containers/nova/nova-compute.log:123:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
  /var/log/containers/nova/nova-compute.log:124:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  /var/log/containers/nova/nova-compute.log:125:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  /var/log/containers/nova/nova-compute.log:126:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
  /var/log/containers/nova/nova-compute.log:127:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
  /var/log/containers/nova/nova-compute.log:128:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  /var/log/containers/nova/nova-compute.log:129:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1006, in blockResize
  /var/log/containers/nova/nova-compute.log:130:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockResize() failed', dom=self)
  /var/log/containers/nova/nova-compute.log:131:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
  /var/log/containers/nova/nova-compute.log:132:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server 
  /var/log/containers/libvirt/libvirtd.log:2:2020-01-28 07:14:17.392+0000: 4425: error : qemuMonitorJSONCheckError:418 : internal error: unable to execute QEMU command 'block_resize': Cannot grow device files

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1861071/+subscriptions
References

[Bug 1861071] [NEW] Failing to extend an attached encrypted volume
From: Tzach Shefi, 2020-01-28