yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #78564
[Bug 1829881] [NEW] Swap of volumes with multiattach fails
Public bug reported:
Swap test case of volume with multiattach fails
tempest.api.compute.admin.test_volume_swap.TestMultiAttachVolumeSwap.test_volume_swap_with_multiattach
[337.211730s] ... FAILED
Error log in nova
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Failed to detach volume f33ba2e1-b649-478c-af4c-dba062423ad2 from /dev/vdb: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Exit code: 1
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stdout: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stderr: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Traceback (most recent call last):
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/block_device.py", line 326, in driver_detach
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] encryption=encryption)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1717, in detach_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] encryption=encryption)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1345, in _disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] vol_driver.disconnect_volume(connection_info, instance)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/libvirt/volume/fibrechannel.py", line 72, in disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] connection_info['data'])
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] result = f(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] return f(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/connectors/fibre_channel.py", line 336, in disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] wwn = self._linuxscsi.get_scsi_wwn(path)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/linuxscsi.py", line 163, in get_scsi_wwn
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] root_helper=self._root_helper)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/executor.py", line 52, in _execute
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] result = self.__execute(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] return execute_root(*cmd, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/priv_context.py", line 242, in _wrap
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] return self.channel.remote_call(name, args, kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] raise exc_type(*result[2])
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Exit code: 1
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stdout: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stderr: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]
May 07 01:56:28.170915 dellscfc nova-compute[1036]: DEBUG oslo_concurrency.lockutils [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Lock "2d08d9a4-6eab-4869-ba48-802c546c7eb7" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 1.992s {{(pid=1036) inner /usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py:339}}
May 07 01:56:28.227431 dellscfc nova-compute[1036]: ERROR oslo_messaging.rpc.server [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Exit code: 1
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stdout: ''
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stderr: ''
Seems when ‘scsi_id’ runs, the disk has already gone. No errors on the
cinder side
Driver: Dell EMC SC - FC Driver
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1829881
Title:
Swap of volumes with multiattach fails
Status in OpenStack Compute (nova):
New
Bug description:
Swap test case of volume with multiattach fails
tempest.api.compute.admin.test_volume_swap.TestMultiAttachVolumeSwap.test_volume_swap_with_multiattach
[337.211730s] ... FAILED
Error log in nova
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Failed to detach volume f33ba2e1-b649-478c-af4c-dba062423ad2 from /dev/vdb: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Exit code: 1
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stdout: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stderr: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Traceback (most recent call last):
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/block_device.py", line 326, in driver_detach
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] encryption=encryption)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1717, in detach_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] encryption=encryption)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1345, in _disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] vol_driver.disconnect_volume(connection_info, instance)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/opt/stack/new/nova/nova/virt/libvirt/volume/fibrechannel.py", line 72, in disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] connection_info['data'])
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] result = f(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] return f(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/connectors/fibre_channel.py", line 336, in disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] wwn = self._linuxscsi.get_scsi_wwn(path)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/linuxscsi.py", line 163, in get_scsi_wwn
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] root_helper=self._root_helper)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/executor.py", line 52, in _execute
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] result = self.__execute(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] return execute_root(*cmd, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/priv_context.py", line 242, in _wrap
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] return self.channel.remote_call(name, args, kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] raise exc_type(*result[2])
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Exit code: 1
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stdout: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stderr: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]
May 07 01:56:28.170915 dellscfc nova-compute[1036]: DEBUG oslo_concurrency.lockutils [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Lock "2d08d9a4-6eab-4869-ba48-802c546c7eb7" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 1.992s {{(pid=1036) inner /usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py:339}}
May 07 01:56:28.227431 dellscfc nova-compute[1036]: ERROR oslo_messaging.rpc.server [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Exit code: 1
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stdout: ''
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stderr: ''
Seems when ‘scsi_id’ runs, the disk has already gone. No errors on the
cinder side
Driver: Dell EMC SC - FC Driver
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1829881/+subscriptions
Follow ups