yahoo-eng-team team mailing list archive

[Walt Boring <walter.boring@xxxxxx>]

This bug isn't valid anymore.

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1192763

Title:
  Removing FC device causes exception preventing detachment completion

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Running a stress test tool have uncover a race condition where the
  detachment of a device from an instance failed due to an exception
  error. The test was run using devstack with the latest Havana code.

  From the nova-compute log note that a request was issue to remove
  sdar. The device still exist after the remove request was completed.
  An exception occur when it trys to remove it for the second time:

  2013-06-19 14:57:24.236 DEBUG nova.storage.linuxscsi [-] Trying (1) to
  remove device /dev/sdar from (pid=43515) _wait_for_remove
  /opt/stack/nova/nova/storage/linuxscsi.py:72

  2013-06-19 14:57:24.236 DEBUG nova.openstack.common.processutils [-]
  Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
  tee -a /sys/bus/scsi/drivers/sd/1:0:0:21/delete from (pid=43515)
  execute /opt/stack/nova/nova/openstack/common/processutils.py:142

  2013-06-19 14:57:24.481 DEBUG nova.openstack.common.processutils [-]
  Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
  sginfo -r from (pid=43515) execute
  /opt/stack/nova/nova/openstack/common/processutils.py:142

  2013-06-19 14:57:24.905 DEBUG nova.storage.linuxscsi [-] ### devices:
  ['/dev/sdcg', '/dev/sdcf', '/dev/sdu', '/dev/sdt', '/dev/sdm',
  '/dev/sdl', '/dev/sdce', '/dev/sdcd', '/dev/sdcc', '/dev/sdcb',
  '/dev/sdg', '/dev/sdf', '/dev/sdca', '/dev/sdbz', '/dev/sdby',
  '/dev/sdbx', '/dev/sdbw', '/dev/sdbv', '/dev/sdbu', '/dev/sdbt',
  '/dev/sdbs', '/dev/sdbr', '/dev/sdbq', '/dev/sdbp', '/dev/sdbo',
  '/dev/sdbn', '/dev/sdbm', '/dev/sdbl', '/dev/sdbk', '/dev/sdbj',
  '/dev/sdbi', '/dev/sdbh', '/dev/sdbg', '/dev/sdbf', '/dev/sdbe',
  '/dev/sdbd', '/dev/sdbc', '/dev/sdbb', '/dev/sdba', '/dev/sdaz',
  '/dev/sday', '/dev/sdax', '/dev/sdaw', '/dev/sdav', '/dev/sdau',
  '/dev/sdat', '/dev/sdas', '/dev/sdar', '/dev/sdaq', '/dev/sdap',
  '/dev/sdao', '/dev/sdan', '/dev/sdak', '/dev/sdaj', '/dev/sdai',
  '/dev/sdah', '/dev/sdag', '/dev/sdaf', '/dev/sdae', '/dev/sdac',
  '/dev/sdaa', '/dev/sdz', '/dev/sdy', '/dev/sdx', '/dev/sdw',
  '/dev/sdv', '/dev/sds', '/dev/sdr', '/dev/sdq', '/dev/sdp',
  '/dev/sdo', '/dev/sdn', '/dev/sdk', '/dev/sdj', '/dev/sdi',
  '/dev/sdh', '/dev/sde', '/dev/sdd', '/dev/sdc', '/dev/sdb',
  '/dev/sda'] from (pid=43515) _wait_for_remove
  /opt/stack/nova/nova/storage/linuxscsi.py:82

  2013-06-19 14:57:25.004 DEBUG requests.packages.urllib3.connectionpool
  [-] "POST /v1/fc4ee83b5ccf4b2dacd0a02ebd66c2a8/volumes/61be749a-
  68e3-4589-acbf-25eee64ef17d/action HTTP/1.1" 200 211 from (pid=43515)
  _make_request /usr/local/lib/python2.7/dist-
  packages/requests/packages/urllib3/connectionpool.py:289

  2013-06-19 14:57:26.236 DEBUG nova.storage.linuxscsi [-] Trying (1) to
  remove device /dev/sdar from (pid=43515) _wait_for_remove
  /opt/stack/nova/nova/storage/linuxscsi.py:72

  2013-06-19 14:57:26.236 DEBUG nova.openstack.common.processutils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf tee -a /sys/bus/scsi/drivers/sd/1:0:0:21/delete from (pid=43515) execute /opt/stack/nova/nova/openstack/common/processutils.py:142
  2013-06-19 14:57:26.343 DEBUG nova.openstack.common.processutils [-] Result was 1 from (pid=43515) execute /opt/stack/nova/nova/openstack/common/processutils.py:167
  2013-06-19 14:57:26.344 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall   File "/opt/stack/nova/nova/openstack/common/loopingcall.py", line 78, in _inner
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall     self.f(*self.args, **self.kw)
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall   File "/opt/stack/nova/nova/storage/linuxscsi.py", line 77, in _wait_for_remove
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall     "1")
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall   File "/opt/stack/nova/nova/storage/linuxscsi.py", line 31, in echo_scsi_command
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall     utils.execute('tee', *args, **kwargs)
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall   File "/opt/stack/nova/nova/utils.py", line 154, in execute
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall     return processutils.execute(*cmd, **kwargs)
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall   File "/opt/stack/nova/nova/openstack/common/processutils.py", line 173, in execute
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall     cmd=' '.join(cmd))
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall ProcessExecutionError: Unexpected error while running command.
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall Command: sudo nova-rootwrap /etc/nova/rootwrap.conf tee -a /sys/bus/scsi/drivers/sd/1:0:0:21/delete
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall Exit code: 1
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall Stdout: '1'
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall Stderr: '/usr/bin/tee: /sys/bus/scsi/drivers/sd/1:0:0:21/delete: No such file or directory\n'
  2013-06-19 14:57:26.344 TRACE nova.openstack.common.loopingcall

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1192763/+subscriptions