yahoo-eng-team team mailing list archive

[Alan Pevec <1332382@xxxxxxxxxxxxxxxxxx>]

** Also affects: nova/icehouse
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1332382

Title:
  block device mapping timeout in compute

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  New

Bug description:
  
  When booting instances passing in block-device and increasing the volume size , instances can go in to error state if the volume takes longer to create than the hard code value set in:

  nova/compute/manager.py

    def _await_block_device_map_created(self, context, vol_id, max_tries=180,
                                          wait_between=1):

  
  Here is the command used to repro:

  nova boot --flavor ca8d889e-6a4e-48f8-81ce-0fa2d153db16 --image 438b3f1f-1b23-4b8d-84e1-786ffc73a298  
  --block-device source=image,id=438b3f1f-1b23-4b8d-84e1-786ffc73a298,dest=volume,size=128  
  --nic net-id=5f847661-edef-4dff-9f4b-904d1b3ac422 --security-groups d9ce9fe3-983f-42a8-899e-609c01977e32  
  Test_Image_Instance

  max_retries should be made configurable.

  Looking through the different releases, Grizzly was 30, Havana was 60
  , IceHouse is 180.

  Here is a traceback:

  2014-06-19 06:54:24.303 17578 ERROR nova.compute.manager [req-050fc984-cfa2-4c34-9cde-c8aeea65e6ed  
  d0b8f2c3cf70445baae994004e602e11 1e83429a8157489fb7ce087bd037f5d9] [instance:  
  74f612ea-9722-4796-956f-32defd417000] Instance failed block device setup
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  Traceback (most recent call last):
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1394,  
  in _prep_block_device
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  self._await_block_device_map_created))
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283,  
  in attach_block_devices
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  block_device_mapping)
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 238,  
  in attach
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  wait_func(context, vol['id'])
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 909,  
  in _await_block_device_map_created
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  attempts=attempts)
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]  
  VolumeNotCreated: Volume 8489549e-d23e-45c2-ae6e-7fdb1a9c30d0 did not finish  
  being created even after we waited 65 seconds or 60 attempts.
  2014-06-19 06:54:24.303 17578 TRACE nova.compute.manager [instance: 74f612ea-9722-4796-956f-32defd417000]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1332382/+subscriptions