yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1849425] Re: Extend attached volume case failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1849425@xxxxxxxxxxxxxxxxxx>
Date: Fri, 06 Aug 2021 20:39:38 -0000
Reply-to: Bug 1849425 <1849425@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/nova/+/801714
Committed: https://opendev.org/openstack/nova/commit/49ba5a763f9d2c7c496f391ca8149a18541bfec7
Submitter: "Zuul (22348)"
Branch:    master

commit 49ba5a763f9d2c7c496f391ca8149a18541bfec7
Author: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date:   Thu Jul 22 09:05:51 2021 +0100

    libvirt: Handle silent failures to extend volume within os-brick
    
    As seen in bug #1849425 os-brick can fail to extend an underlying volume
    device on the compute silently, returning a new_size of None to n-cpu.
    
    While this should ultimatley be addressed in os-brick n-cpu can also
    handle this before we eventually run into a type error when attemting
    floor division later in the volume extend flow.
    
    Change-Id: Ic8091537274a5ad27fb5af8939f81ed154b7ad7c
    Closes-Bug: #1849425


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1849425

Title:
  Extend attached volume case failed

Status in Cinder:
  New
Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The online expansion volume use case failed,
  tempest.api.volume.test_volumes_extend:VolumesExtendAttachedTest.test_extend_attached_volume.
  This use case was originally successfully executed and failed after
  reinstalling devstack (stein-->train). The cinder driver has not
  changed, and the multipath.conf configuration has not changed. There
  are two kinds of errors in devstack@n-cpu.service, one is the error
  under the ubuntu18 system, and the other is the error under the
  centos7 system. as follows

  ubuntu18:
  Oct 23 09:34:52 stack nova-compute[11706]: WARNING nova.compute.manager [req-0ce8f571-5c04-4a72-8049-3839c934c7c9 req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] [instance: 6a4b5b05-20b7-46c1-b903-b6fe40a38aad] Extend volume failed, volume_id=bc3c5af6-b6c6-4e01-8d0e-774ad637debc, reason: unsupported operand type(s) for //: 'NoneType' and 'int'
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server [req-0ce8f571-5c04-4a72-8049-3839c934c7c9 req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] Exception during message handling: TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     raise value
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8587, in external_instance_event
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     raise value
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     bdm.volume_size * units.Gi)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1768, in extend_volume
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server     dev.resize(new_size // units.Ki)
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
  Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server 


  centos7:
  Oct 10 10:15:12 devstack nova-compute[59879]: DEBUG os_brick.initiator.connectors.iscsi [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] <== extend_volume: exception (371ms) ProcessExecutionError('fail\n', '', 1, 'multipathd resize map 3600b34245364482d2d43d02ffda900df', None) {{(pid=59879) trace_logging_wrapper /usr/local/lib/python3.7/site-packages/os_brick/utils.py:156}}
  Oct 10 10:15:12 devstack nova-compute[59879]: WARNING nova.compute.manager [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] [instance: 72d1a0fc-91bf-46eb-9ef8-96a81886a747] Extend volume failed, volume_id=79481f44-72dd-4c07-98cb-70f52048f10e, reason: Unexpected error while running command.
  Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
  Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
  Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
  Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
  Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
  Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
  Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: ''
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     raise value
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8587, in external_instance_event
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     raise value
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     bdm.volume_size * units.Gi)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1738, in extend_volume
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     requested_size)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1371, in _extend_volume
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     requested_size)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 86, in extend_volume
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     new_size = self.connector.extend_volume(connection_info['data'])
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 477, in extend_volume
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return self._linuxscsi.extend_volume(volume_paths)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line 572, in extend_volume
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     result = self.multipath_resize_map(scsi_wwn)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line 531, in multipath_resize_map
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     root_helper=self._root_helper)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/os_brick/executor.py", line 52, in _execute
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     result = self.__execute(*args, **kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return execute_root(*cmd, **kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     return self.channel.remote_call(name, args, kwargs)
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server     raise exc_type(*result[2])
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Exit code: 1
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Stdout: 'fail\n'
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Stderr: ''
  Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server 

  
  These two errors will be resolved after changing user_friendly_names in multipath.conf to no. What is the problem with this?

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1849425/+subscriptions