yahoo-eng-team team mailing list archive

[Rajini Karthik <rajini_karthik@xxxxxxxx>]

Public bug reported:

Swap test case of volume with multiattach fails

tempest.api.compute.admin.test_volume_swap.TestMultiAttachVolumeSwap.test_volume_swap_with_multiattach
[337.211730s] ... FAILED

Error log in nova
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Failed to detach volume f33ba2e1-b649-478c-af4c-dba062423ad2 from /dev/vdb: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Exit code: 1
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stdout: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stderr: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Traceback (most recent call last):
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 326, in driver_detach
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     encryption=encryption)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1717, in detach_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     encryption=encryption)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1345, in _disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     vol_driver.disconnect_volume(connection_info, instance)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/libvirt/volume/fibrechannel.py", line 72, in disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     connection_info['data'])
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     result = f(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     return f(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/connectors/fibre_channel.py", line 336, in disconnect_volume
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     wwn = self._linuxscsi.get_scsi_wwn(path)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/linuxscsi.py", line 163, in get_scsi_wwn
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     root_helper=self._root_helper)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/executor.py", line 52, in _execute
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     result = self.__execute(*args, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     return execute_root(*cmd, **kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/priv_context.py", line 242, in _wrap
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     return self.channel.remote_call(name, args, kwargs)
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     raise exc_type(*result[2])
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Exit code: 1
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stdout: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stderr: ''
May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] 
May 07 01:56:28.170915 dellscfc nova-compute[1036]: DEBUG oslo_concurrency.lockutils [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Lock "2d08d9a4-6eab-4869-ba48-802c546c7eb7" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 1.992s {{(pid=1036) inner /usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py:339}}
May 07 01:56:28.227431 dellscfc nova-compute[1036]: ERROR oslo_messaging.rpc.server [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Exit code: 1
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stdout: ''
May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stderr: ''


Seems when ‘scsi_id’ runs, the disk has already gone. No errors on the
cinder side

Driver: Dell EMC SC - FC Driver

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1829881

Title:
  Swap of volumes with multiattach fails

Status in OpenStack Compute (nova):
  New

Bug description:
  Swap test case of volume with multiattach fails

  tempest.api.compute.admin.test_volume_swap.TestMultiAttachVolumeSwap.test_volume_swap_with_multiattach
  [337.211730s] ... FAILED

  Error log in nova
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Failed to detach volume f33ba2e1-b649-478c-af4c-dba062423ad2 from /dev/vdb: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: Exit code: 1
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stdout: ''
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: Stderr: ''
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Traceback (most recent call last):
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 326, in driver_detach
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     encryption=encryption)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1717, in detach_volume
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     encryption=encryption)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1345, in _disconnect_volume
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     vol_driver.disconnect_volume(connection_info, instance)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/opt/stack/new/nova/nova/virt/libvirt/volume/fibrechannel.py", line 72, in disconnect_volume
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     connection_info['data'])
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     result = f(*args, **kwargs)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     return f(*args, **kwargs)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/connectors/fibre_channel.py", line 336, in disconnect_volume
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     wwn = self._linuxscsi.get_scsi_wwn(path)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/initiator/linuxscsi.py", line 163, in get_scsi_wwn
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     root_helper=self._root_helper)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/executor.py", line 52, in _execute
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     result = self.__execute(*args, **kwargs)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/os_brick/privileged/rootwrap.py", line 169, in execute
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     return execute_root(*cmd, **kwargs)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/priv_context.py", line 242, in _wrap
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     return self.channel.remote_call(name, args, kwargs)
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]   File "/usr/local/lib/python3.6/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7]     raise exc_type(*result[2])
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Exit code: 1
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stdout: ''
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] Stderr: ''
  May 07 01:56:28.117664 dellscfc nova-compute[1036]: ERROR nova.virt.block_device [instance: 2d08d9a4-6eab-4869-ba48-802c546c7eb7] 
  May 07 01:56:28.170915 dellscfc nova-compute[1036]: DEBUG oslo_concurrency.lockutils [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Lock "2d08d9a4-6eab-4869-ba48-802c546c7eb7" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 1.992s {{(pid=1036) inner /usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py:339}}
  May 07 01:56:28.227431 dellscfc nova-compute[1036]: ERROR oslo_messaging.rpc.server [None req-858576cb-bdf3-49c1-b0af-34cb538712b0 tempest-AttachVolumeMultiAttachTest-948792761 tempest-AttachVolumeMultiAttachTest-948792761] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  May 07 01:56:28.227431 dellscfc nova-compute[1036]: Command: /lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/pci-0000:04:00.0-fc-0x5000d3100101d530-lun-3
  May 07 01:56:28.227431 dellscfc nova-compute[1036]: Exit code: 1
  May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stdout: ''
  May 07 01:56:28.227431 dellscfc nova-compute[1036]: Stderr: ''


  Seems when ‘scsi_id’ runs, the disk has already gone. No errors on the
  cinder side

  Driver: Dell EMC SC - FC Driver

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1829881/+subscriptions