yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1439282

Title:
  Cinder API errors and timeouts can cause Nova to not save data about
  volumes

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  A user reports:

  Nova's Block device mappings can become invalid/inconsistent if errors
  are encountered while calling for Cinder to attach a volume.

  2014-12-18 11:14:41.594 19473 ERROR nova.compute.manager [req-6f65b7d5-0930-4adf-9b5f-dd20eb1a707e 96612f5455c44e95960e733c48eaccc9 1076a7e653b3465295131c495e7d4ae4] [instance: 463dbedc-00f4-4c66-a00-139a4d79a46e] Instance failed block device setup
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e] Traceback (most recent call last):
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1706, in _prep_block
  _device
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     self.driver, self._await_block_device_map_created))
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 367, in attach_blo
  ck_devices
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     map(_log_and_attach, block_device_mapping)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 365, in _log_and_a
  ttach
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     bdm.attach(*attach_args, **attach_kwargs)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 322, in attach
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     volume_api, virt_driver)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 44, in wrapped
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     ret_val = method(obj, context, *args, **kwargs)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 255, in attach
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     self['mount_device'], mode=mode)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 173, in wrapper
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     res = method(self, ctx, volume_id, *args, **kwargs)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 262, in attach
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     mountpoint, mode=mode)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 266, in attach
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     'mode': mode})
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 250, in _action
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     return self.api.client.post(url, body=body)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 223, in post
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     return self._cs_request(url, 'POST', **kwargs)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 212, in _cs_request
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e]     raise exceptions.ConnectionError(msg)
  2014-12-18 11:14:41.594 19473 TRACE nova.compute.manager [instance: 463dbedc-00f4-4c66-a020-139a4d79a46e] ConnectionError: Unable to establish connection: HTTPConnectionPool(): Max retries exceeded with url: /v1/1076a7e653b3465295131c495e7d4ae4/volumes/d827b809-e069-4be5-a27c-264e7bcafa65/action

  For example, the above trace led to Nova's block device mapping table
  listing the volume as being unattached , while Cinder lists the volume
  as attached.

  As a result the volume is not removed when removing the instance as
  while the mapping is present the volume_id is NULL :

  2014-12-18 11:46:17.830 19473 WARNING nova.compute.manager [req-
  ec9207f8-5553-46da-ad3c-ec18805fa343 96612f5455c44e95960e733c48eaccc9
  1076a7e653b3465295131c495e7d4ae4] [instance: 463dbedc-
  00f4-4c66-a020-139a4d79a46e] Ignoring VolumeNotFound: Volume None
  could not be found.

  Finally even direct manual removal via the Cinder CLI is blocked as
  the volume is still listed as attached. The only workaround is to hack
  the DB to reset the attach_state of the volume.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1439282/+subscriptions