← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #11338

[Bug 1290566] [NEW] nova detach results in TypeError: 'NoneType' object is unsubscriptable

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

When detaching multiple volumes from multiple instances the stack trace
below was encountered. The db shows a device name present for all
volumes attempted to be detached (also below).

A similar stack trace was found at:
http://pastebin.com/sDR2AVGH


2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     **args)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in wrapped
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     payload)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 76, in wrapped
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 247, in decorated_function
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     pass
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 233, in decorated_function
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 275, in decorated_function
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 262, in decorated_function
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3784, in detach_volume
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     self._detach_volume(context, instance, bdm)
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3722, in _detach_volume
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     mp = bdm['device_name']
2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp TypeError: 'NoneType' object is unsubscriptable

[root@ci-5254003B9180 nova]# nova list
+--------------------------------------+----------------------------------------------+--------+------------+-------------+----------------------+
| ID                                   | Name                                         | Status | Task State | Power State | Networks             |
+--------------------------------------+----------------------------------------------+--------+------------+-------------+----------------------+
| 1d80f083-d64b-492e-90fe-f92da31efb07 | 3nsg-T5-313eff9a-7958-4a28-9188-7c9ac6ab29f8 | ACTIVE | None       | Running     | public=16.125.108.48 |
| 89a8c95c-9831-4645-acaf-3879b4ac9046 | 3nsg-T5-4b06bfe3-2507-4a5c-8d47-9d66af1b1383 | ACTIVE | None       | Running     | public=16.125.108.47 |
| 5c6fdac4-01e9-4e1c-9d15-dde9bc4f1b77 | 3nsg-T5-893f6fd3-93a7-4abb-91cf-5fee1474746d | ACTIVE | None       | Running     | public=16.125.108.51 |
+--------------------------------------+----------------------------------------------+--------+------------+-------------+----------------------+

[root@ci-5254003B9180 nova]# cinder list
+--------------------------------------+-----------+---------------------------------------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |                 Display Name                | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------------------------------------+------+-------------+----------+-------------+
| 031e915c-adf7-47b0-be84-61c41759b747 | available | vg6-T5-27e51943-4c38-4036-a874-bfb583d4adae |  5   |    jimvt1   |  false   |             |
| 08fa1624-a0b1-45c4-84cd-6cb7d2eddbd6 | available | vg5-T5-93e3a2c6-6e47-4ac8-b429-0209b5a73c7d |  5   |    jimvt1   |  false   |             |
| 23ce9948-5885-40b3-90f4-5f690411274b | available | vg4-T5-aeb8b220-12d2-4fc2-8b17-448477ebfbd3 |  5   |    jimvt1   |  false   |             |
| 3aac388e-5722-4a65-a70d-aaeab26abaf3 | available | vg2-T5-3d0a33e8-374e-4bc9-9b5c-3c1354b58769 |  5   |    jimvt1   |  false   |             |
| 488a7101-694e-485a-ab9b-a2fa31ca033c | available | vg5-T5-2a2b8141-8593-4639-bbab-7648fbc04281 |  5   |    jimvt1   |  false   |             |
| 5d4d2ce4-d7ba-4eff-af55-3e1d08c49e33 | available | vg5-T5-349c93bb-629e-4e8d-afa4-f7425a5bb477 |  5   |    jimvt1   |  false   |             |
| 7088730f-bdf6-46c7-84d7-822aaad17d18 | available | vg3-T5-08356a97-d648-44d7-a8c6-78a9c77ecd01 |  5   |    jimvt1   |  false   |             |
| 71828b9b-c966-4be2-8f54-0371a4dd3f02 | available | vg2-T5-f9d35eea-a36c-46c6-98db-4717494cf7c9 |  5   |    jimvt1   |  false   |             |
| 7a97cd36-1908-4a90-b991-dab391a97b95 | available | vg6-T5-cf116e7a-af7a-4714-bb5b-2b209eba2c23 |  5   |    jimvt1   |  false   |             |
| 7da825a7-f6db-4010-bdd6-353ee885092d | available | vg3-T5-4bf9ba8c-8668-4341-b259-504f1d8ad23c |  5   |    jimvt1   |  false   |             |
| b3dcf5c2-c3be-4e19-b4ae-4d216c99debb | available | vg1-T5-7ee09b70-eba2-4392-b299-c0f9f1ffefbb |  5   |    jimvt1   |  false   |             |
| b99d6e41-74d9-45cf-9b3c-c0471c9eda5b | available | vg4-T5-d39fd38f-8087-4f33-8f7a-fbd4c54b2191 |  5   |    jimvt1   |  false   |             |
| d0f1ff77-1e78-426f-9689-2a59e144bf8c | available | vg6-T5-3f49e820-4745-4253-9f69-3d995dae08b4 |  5   |    jimvt1   |  false   |             |
| f503598c-8d9e-49be-aeb3-b172940c7344 | available | vg2-T5-030206e0-a91d-463b-8a87-459aafe0d63d |  5   |    jimvt1   |  false   |             |
+--------------------------------------+-----------+---------------------------------------------+------+-------------+----------+-------------+

nova=> select device_name, volume_id from  block_device_mapping where volume_id='031e915c-adf7-47b0-be84-61c41759b747' or volume_id='08fa1624-a0b1-45c4-84cd-6cb7d2eddbd6' or volume_id='23ce9948-5885-40b3-90f4-5f690411274b' or volume_id='3aac388e-5722-4a65-a70d-aaeab26abaf3' or volume_id='488a7101-694e-485a-ab9b-a2fa31ca033c' or volume_id='5d4d2ce4-d7ba-4eff-af55-3e1d08c49e33' or volume_id='7088730f-bdf6-46c7-84d7-822aaad17d18' or volume_id='71828b9b-c966-4be2-8f54-0371a4dd3f02' or volume_id='7a97cd36-1908-4a90-b991-dab391a97b95' or volume_id='7da825a7-f6db-4010-bdd6-353ee885092d' or volume_id='b3dcf5c2-c3be-4e19-b4ae-4d216c99debb' or volume_id='b99d6e41-74d9-45cf-9b3c-c0471c9eda5b' or volume_id='d0f1ff77-1e78-426f-9689-2a59e144bf8c' or volume_id='f503598c-8d9e-49be-aeb3-b172940c7344';
device_name |              volume_id
-------------+--------------------------------------
/dev/vdf    | 5d4d2ce4-d7ba-4eff-af55-3e1d08c49e33
/dev/vdd    | 7da825a7-f6db-4010-bdd6-353ee885092d
/dev/vde    | b99d6e41-74d9-45cf-9b3c-c0471c9eda5b
/dev/vdf    | 488a7101-694e-485a-ab9b-a2fa31ca033c
/dev/vdg    | 7a97cd36-1908-4a90-b991-dab391a97b95
/dev/vdc    | 3aac388e-5722-4a65-a70d-aaeab26abaf3
/dev/vde    | 23ce9948-5885-40b3-90f4-5f690411274b
/dev/vdg    | 031e915c-adf7-47b0-be84-61c41759b747
/dev/vdf    | 08fa1624-a0b1-45c4-84cd-6cb7d2eddbd6
/dev/vdb    | b3dcf5c2-c3be-4e19-b4ae-4d216c99debb
/dev/vdc    | f503598c-8d9e-49be-aeb3-b172940c7344
/dev/vdc    | 71828b9b-c966-4be2-8f54-0371a4dd3f02
/dev/vdg    | d0f1ff77-1e78-426f-9689-2a59e144bf8c
/dev/vdd    | 7088730f-bdf6-46c7-84d7-822aaad17d18
(14 rows)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1290566

Title:
  nova detach results in  TypeError: 'NoneType' object is
  unsubscriptable

Status in OpenStack Compute (Nova):
  New

Bug description:
  When detaching multiple volumes from multiple instances the stack
  trace below was encountered. The db shows a device name present for
  all volumes attempted to be detached (also below).

  A similar stack trace was found at:
  http://pastebin.com/sDR2AVGH

  
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in wrapped
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 76, in wrapped
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 247, in decorated_function
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     pass
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 233, in decorated_function
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 275, in decorated_function
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 262, in decorated_function
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3784, in detach_volume
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     self._detach_volume(context, instance, bdm)
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3722, in _detach_volume
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp     mp = bdm['device_name']
  2014-03-10 07:59:38.220 22304 TRACE nova.openstack.common.rpc.amqp TypeError: 'NoneType' object is unsubscriptable

  [root@ci-5254003B9180 nova]# nova list
  +--------------------------------------+----------------------------------------------+--------+------------+-------------+----------------------+
  | ID                                   | Name                                         | Status | Task State | Power State | Networks             |
  +--------------------------------------+----------------------------------------------+--------+------------+-------------+----------------------+
  | 1d80f083-d64b-492e-90fe-f92da31efb07 | 3nsg-T5-313eff9a-7958-4a28-9188-7c9ac6ab29f8 | ACTIVE | None       | Running     | public=16.125.108.48 |
  | 89a8c95c-9831-4645-acaf-3879b4ac9046 | 3nsg-T5-4b06bfe3-2507-4a5c-8d47-9d66af1b1383 | ACTIVE | None       | Running     | public=16.125.108.47 |
  | 5c6fdac4-01e9-4e1c-9d15-dde9bc4f1b77 | 3nsg-T5-893f6fd3-93a7-4abb-91cf-5fee1474746d | ACTIVE | None       | Running     | public=16.125.108.51 |
  +--------------------------------------+----------------------------------------------+--------+------------+-------------+----------------------+

  [root@ci-5254003B9180 nova]# cinder list
  +--------------------------------------+-----------+---------------------------------------------+------+-------------+----------+-------------+
  |                  ID                  |   Status  |                 Display Name                | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+---------------------------------------------+------+-------------+----------+-------------+
  | 031e915c-adf7-47b0-be84-61c41759b747 | available | vg6-T5-27e51943-4c38-4036-a874-bfb583d4adae |  5   |    jimvt1   |  false   |             |
  | 08fa1624-a0b1-45c4-84cd-6cb7d2eddbd6 | available | vg5-T5-93e3a2c6-6e47-4ac8-b429-0209b5a73c7d |  5   |    jimvt1   |  false   |             |
  | 23ce9948-5885-40b3-90f4-5f690411274b | available | vg4-T5-aeb8b220-12d2-4fc2-8b17-448477ebfbd3 |  5   |    jimvt1   |  false   |             |
  | 3aac388e-5722-4a65-a70d-aaeab26abaf3 | available | vg2-T5-3d0a33e8-374e-4bc9-9b5c-3c1354b58769 |  5   |    jimvt1   |  false   |             |
  | 488a7101-694e-485a-ab9b-a2fa31ca033c | available | vg5-T5-2a2b8141-8593-4639-bbab-7648fbc04281 |  5   |    jimvt1   |  false   |             |
  | 5d4d2ce4-d7ba-4eff-af55-3e1d08c49e33 | available | vg5-T5-349c93bb-629e-4e8d-afa4-f7425a5bb477 |  5   |    jimvt1   |  false   |             |
  | 7088730f-bdf6-46c7-84d7-822aaad17d18 | available | vg3-T5-08356a97-d648-44d7-a8c6-78a9c77ecd01 |  5   |    jimvt1   |  false   |             |
  | 71828b9b-c966-4be2-8f54-0371a4dd3f02 | available | vg2-T5-f9d35eea-a36c-46c6-98db-4717494cf7c9 |  5   |    jimvt1   |  false   |             |
  | 7a97cd36-1908-4a90-b991-dab391a97b95 | available | vg6-T5-cf116e7a-af7a-4714-bb5b-2b209eba2c23 |  5   |    jimvt1   |  false   |             |
  | 7da825a7-f6db-4010-bdd6-353ee885092d | available | vg3-T5-4bf9ba8c-8668-4341-b259-504f1d8ad23c |  5   |    jimvt1   |  false   |             |
  | b3dcf5c2-c3be-4e19-b4ae-4d216c99debb | available | vg1-T5-7ee09b70-eba2-4392-b299-c0f9f1ffefbb |  5   |    jimvt1   |  false   |             |
  | b99d6e41-74d9-45cf-9b3c-c0471c9eda5b | available | vg4-T5-d39fd38f-8087-4f33-8f7a-fbd4c54b2191 |  5   |    jimvt1   |  false   |             |
  | d0f1ff77-1e78-426f-9689-2a59e144bf8c | available | vg6-T5-3f49e820-4745-4253-9f69-3d995dae08b4 |  5   |    jimvt1   |  false   |             |
  | f503598c-8d9e-49be-aeb3-b172940c7344 | available | vg2-T5-030206e0-a91d-463b-8a87-459aafe0d63d |  5   |    jimvt1   |  false   |             |
  +--------------------------------------+-----------+---------------------------------------------+------+-------------+----------+-------------+

  nova=> select device_name, volume_id from  block_device_mapping where volume_id='031e915c-adf7-47b0-be84-61c41759b747' or volume_id='08fa1624-a0b1-45c4-84cd-6cb7d2eddbd6' or volume_id='23ce9948-5885-40b3-90f4-5f690411274b' or volume_id='3aac388e-5722-4a65-a70d-aaeab26abaf3' or volume_id='488a7101-694e-485a-ab9b-a2fa31ca033c' or volume_id='5d4d2ce4-d7ba-4eff-af55-3e1d08c49e33' or volume_id='7088730f-bdf6-46c7-84d7-822aaad17d18' or volume_id='71828b9b-c966-4be2-8f54-0371a4dd3f02' or volume_id='7a97cd36-1908-4a90-b991-dab391a97b95' or volume_id='7da825a7-f6db-4010-bdd6-353ee885092d' or volume_id='b3dcf5c2-c3be-4e19-b4ae-4d216c99debb' or volume_id='b99d6e41-74d9-45cf-9b3c-c0471c9eda5b' or volume_id='d0f1ff77-1e78-426f-9689-2a59e144bf8c' or volume_id='f503598c-8d9e-49be-aeb3-b172940c7344';
  device_name |              volume_id
  -------------+--------------------------------------
  /dev/vdf    | 5d4d2ce4-d7ba-4eff-af55-3e1d08c49e33
  /dev/vdd    | 7da825a7-f6db-4010-bdd6-353ee885092d
  /dev/vde    | b99d6e41-74d9-45cf-9b3c-c0471c9eda5b
  /dev/vdf    | 488a7101-694e-485a-ab9b-a2fa31ca033c
  /dev/vdg    | 7a97cd36-1908-4a90-b991-dab391a97b95
  /dev/vdc    | 3aac388e-5722-4a65-a70d-aaeab26abaf3
  /dev/vde    | 23ce9948-5885-40b3-90f4-5f690411274b
  /dev/vdg    | 031e915c-adf7-47b0-be84-61c41759b747
  /dev/vdf    | 08fa1624-a0b1-45c4-84cd-6cb7d2eddbd6
  /dev/vdb    | b3dcf5c2-c3be-4e19-b4ae-4d216c99debb
  /dev/vdc    | f503598c-8d9e-49be-aeb3-b172940c7344
  /dev/vdc    | 71828b9b-c966-4be2-8f54-0371a4dd3f02
  /dev/vdg    | d0f1ff77-1e78-426f-9689-2a59e144bf8c
  /dev/vdd    | 7088730f-bdf6-46c7-84d7-822aaad17d18
  (14 rows)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1290566/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next