yahoo-eng-team team mailing list archive

[Duncan Thomas <duncan.thomas@xxxxxxxxx>]

Should nova do some tidying up if it gets a permission denied?

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1413610

Title:
  Nova volume-update leaves volumes stuck in attaching/detaching

Status in Cinder:
  New
Status in OpenStack Compute (Nova):
  New

Bug description:
  There is a problem with the nova command 'volume-update' that leaves
  cinder volumes in the states 'attaching' and 'deleting'.

  If the nova command 'volume-update' is used by a non admin user the
  command fails and the volumes referenced in the command are left in
  the states 'attaching' and 'deleting'.

  
  For example, if a non admin user runs the command
   $ nova volume-update d39dc7f2-929d-49bb-b22f-56adb3f378c7 f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b 59b0cf66-67c8-4041-a505-78000b9c71f6

   Will result in the two volumes stuck like this:

   $ cinder list
   +--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
   |                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
   +--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
   | 59b0cf66-67c8-4041-a505-78000b9c71f6 | attaching |     vol2     |  1   |     None    |  false   |                                      |
   | f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b | detaching |     vol1     |  1   |     None    |  false   | d39dc7f2-929d-49bb-b22f-56adb3f378c7 |
   +--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

  
  And the following in the cinder-api log:

  
  2015-01-21 11:00:03.969 13588 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id None, project_id None, roles None service: user_id None, project_id None, roles None __call__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/keystonemiddleware/auth_token.py:746
  2015-01-21 11:00:03.970 13588 DEBUG routes.middleware [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] Matched POST /d40e3207e34a4b558bf2d58bd3fe268a/volumes/f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b/action __call__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/routes/middleware.py:100
  2015-01-21 11:00:03.971 13588 DEBUG routes.middleware [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] Route path: '/{project_id}/volumes/:(id)/action', defaults: {'action': u'action', 'controller': <cinder.api.openstack.wsgi.Resource object at 0x7febe7a89850>} __call__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/routes/middleware.py:102
  2015-01-21 11:00:03.971 13588 DEBUG routes.middleware [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] Match dict: {'action': u'action', 'controller': <cinder.api.openstack.wsgi.Resource object at 0x7febe7a89850>, 'project_id': u'd40e3207e34a4b558bf2d58bd3fe268a', 'id': u'f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b'} __call__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/routes/middleware.py:103
  2015-01-21 11:00:03.972 13588 INFO cinder.api.openstack.wsgi [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] POST http://192.0.2.24:8776/v1/d40e3207e34a4b558bf2d58bd3fe268a/volumes/f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b/action
  2015-01-21 11:00:03.972 13588 DEBUG cinder.api.openstack.wsgi [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] Action body: {"os-migrate_volume_completion": {"new_volume": "59b0cf66-67c8-4041-a505-78000b9c71f6", "error": false}} get_method /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py:1010
  2015-01-21 11:00:03.973 13588 INFO cinder.api.openstack.wsgi [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] http://192.0.2.24:8776/v1/d40e3207e34a4b558bf2d58bd3fe268a/volumes/f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b/action returned with HTTP 403
  2015-01-21 11:00:03.975 13588 INFO eventlet.wsgi.server [req-d08bc456-14e9-4f97-9f4e-a9f87e7d6add a00b77059c8c4bbbb00a4ff9a8496b2d d40e3207e34a4b558bf2d58bd3fe268a - - -] 127.0.0.1 - - [21/Jan/2015 11:00:03] "POST /v1/d40e3207e34a4b558bf2d58bd3fe268a/volumes/f0c3ea8f-c00f-4db8-aa20-e2dc6a1ddc9b/action HTTP/1.1" 403 429 0.123613


  
  The problem is that the nova policy.json file allows a non admin user to run the command 'volume-update', but the cinder policy.json file requires the admin role to run the action os-migrate_volume_completion, which is called by nova as part of the 'update-volume' process.

  The operation will complete successfully if it is performed by an
  admin user, or if it is run by a non admin user after the cinder
  policy.json file is updated.

  
  The simplest fix would be to remove the "rule:admin_api" requirement from the "os-migrate_volume_completion" action in cinder policy.json

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1413610/+subscriptions