yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #80466
[Bug 1849425] Re: Extend attached volume case failed
First traceback appears to initiate from the Nova libvirt driver.
** Also affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1849425
Title:
Extend attached volume case failed
Status in Cinder:
New
Status in OpenStack Compute (nova):
New
Bug description:
The online expansion volume use case failed,
tempest.api.volume.test_volumes_extend:VolumesExtendAttachedTest.test_extend_attached_volume.
This use case was originally successfully executed and failed after
reinstalling devstack (stein-->train). The cinder driver has not
changed, and the multipath.conf configuration has not changed. There
are two kinds of errors in devstack@n-cpu.service, one is the error
under the ubuntu18 system, and the other is the error under the
centos7 system. as follows
ubuntu18:
Oct 23 09:34:52 stack nova-compute[11706]: WARNING nova.compute.manager [req-0ce8f571-5c04-4a72-8049-3839c934c7c9 req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] [instance: 6a4b5b05-20b7-46c1-b903-b6fe40a38aad] Extend volume failed, volume_id=bc3c5af6-b6c6-4e01-8d0e-774ad637debc, reason: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server [req-0ce8f571-5c04-4a72-8049-3839c934c7c9 req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] Exception during message handling: TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server raise value
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8587, in external_instance_event
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server raise value
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server bdm.volume_size * units.Gi)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1768, in extend_volume
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server dev.resize(new_size // units.Ki)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
centos7:
Oct 10 10:15:12 devstack nova-compute[59879]: DEBUG os_brick.initiator.connectors.iscsi [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] <== extend_volume: exception (371ms) ProcessExecutionError('fail\n', '', 1, 'multipathd resize map 3600b34245364482d2d43d02ffda900df', None) {{(pid=59879) trace_logging_wrapper /usr/local/lib/python3.7/site-packages/os_brick/utils.py:156}}
Oct 10 10:15:12 devstack nova-compute[59879]: WARNING nova.compute.manager [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] [instance: 72d1a0fc-91bf-46eb-9ef8-96a81886a747] Extend volume failed, volume_id=79481f44-72dd-4c07-98cb-70f52048f10e, reason: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: ''
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server raise value
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8587, in external_instance_event
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server raise value
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server bdm.volume_size * units.Gi)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1738, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server requested_size)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1371, in _extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server requested_size)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 86, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server new_size = self.connector.extend_volume(connection_info['data'])
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 477, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return self._linuxscsi.extend_volume(volume_paths)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line 572, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = self.multipath_resize_map(scsi_wwn)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line 531, in multipath_resize_map
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server root_helper=self._root_helper)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/executor.py", line 52, in _execute
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Stderr: ''
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
These two errors will be resolved after changing user_friendly_names in multipath.conf to no. What is the problem with this?
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1849425/+subscriptions