yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1522705] [NEW] Cinder volumes are stuck when non admin user executes nova swap volume API

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Takashi NATSUME <natsume.takashi@xxxxxxxxxxxxx>
Date: Fri, 04 Dec 2015 07:20:15 -0000
Reply-to: Bug 1522705 <1522705@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Cinder volumes are stuck in attaching/detaching state when non admin user executes nova swap volume API.
Because cinder 'migrate_volume_completion' API can be executed by admin only in default settings of cinder policy.json.
So the default settings of cinder policy.json should be fixed.

[How to reproduce]
stack@devstack-master:/opt/devstack$ env | grep OS
(snipped...)
OS_USERNAME=demo
OS_TENANT_NAME=demo
(snipped...)
stack@devstack-master:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks                                                |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
| 5a4102cd-9e36-480c-a148-d2a127ff704e | server1 | ACTIVE | -          | Running     | private=10.0.10.3, fd61:9f1e:73ec:0:f816:3eff:fe34:8b61 |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
stack@devstack-master:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
|                  ID                  |   Status  |  Name | Size | Volume Type | Bootable | Multiattach |             Attached to              |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
| 1c768de0-949f-485f-b914-c430752e2984 | available | TEST2 |  1   | lvmdriver-1 |  false   |    False    |                                      |
| 999a256e-aadf-41b6-88f3-1412e3462cbe |   in-use  | TEST1 |  1   | lvmdriver-1 |  false   |    False    | 5a4102cd-9e36-480c-a148-d2a127ff704e |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
stack@devstack-master:/opt/devstack$ nova volume-update server1 999a256e-aadf-41b6-88f3-1412e3462cbe 1c768de0-949f-485f-b914-c430752e2984
stack@devstack-master:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
|                  ID                  |   Status  |  Name | Size | Volume Type | Bootable | Multiattach |             Attached to              |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
| 1c768de0-949f-485f-b914-c430752e2984 | attaching | TEST2 |  1   | lvmdriver-1 |  false   |    False    |                                      |
| 999a256e-aadf-41b6-88f3-1412e3462cbe | detaching | TEST1 |  1   | lvmdriver-1 |  false   |    False    | 5a4102cd-9e36-480c-a148-d2a127ff704e |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+

[cinder-api.log]
2015-12-04 15:41:51.986 INFO cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] POST http://10.0.2.15:8776/v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action
2015-12-04 15:41:51.986 DEBUG cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] Action body: {"os-migrate_volume_completion": {"new_volume": "1c768de0-949f-485f-b914-c430752e2984", "error": false}} from (pid=18203) get_method /opt/stack/cinder/cinder/api/openstack/wsgi.py:1093
2015-12-04 15:41:51.987 INFO cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] http://10.0.2.15:8776/v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action returned with HTTP 403
2015-12-04 15:41:51.989 INFO eventlet.wsgi.server [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] 10.0.2.15 - - [04/Dec/2015 15:41:51] "POST /v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action HTTP/1.1" 403 429 0.005410

[policy.json(cinder)]
The follwong line defines that 'migrate_volume_completion' can be executed by admin only.

"volume_extension:volume_admin_actions:migrate_volume_completion":
"rule:admin_api",

[Environment]
nova: commit 7df427fdb4d7a314b79b1e3977aeb86f019b5186(master)
cinder: commit 707902ca482d1653072c2cf419c566cd5fee1f96(master)
OS: Ubuntu 14.04

stack@devstack-master:/opt/devstack$ keystone user-role-list --user demo --tenant demo
+----------------------------------+-------------+----------------------------------+----------------------------------+
|                id                |     name    |             user_id              |            tenant_id             |
+----------------------------------+-------------+----------------------------------+----------------------------------+
| dec5e6b197d6424bbfb3854f6718cef0 |    Member   | 181cd8bc1c004030aec893fc7de79618 | 722380ff5288483191cd2712cbc99c5d |
| de75e50dec2e48b4a4c66feef8f53432 | anotherrole | 181cd8bc1c004030aec893fc7de79618 | 722380ff5288483191cd2712cbc99c5d |
+----------------------------------+-------------+----------------------------------+----------------------------------+

** Affects: cinder
     Importance: Undecided
     Assignee: Takashi NATSUME (natsume-takashi)
         Status: New

** Affects: nova
     Importance: Undecided
     Assignee: Takashi NATSUME (natsume-takashi)
         Status: New

** Changed in: cinder
     Assignee: (unassigned) => Takashi NATSUME (natsume-takashi)

** Also affects: nova
   Importance: Undecided
       Status: New

** Changed in: nova
     Assignee: (unassigned) => Takashi NATSUME (natsume-takashi)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1522705

Title:
  Cinder volumes are stuck when non admin user executes nova swap volume
  API

Status in Cinder:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  Cinder volumes are stuck in attaching/detaching state when non admin user executes nova swap volume API.
  Because cinder 'migrate_volume_completion' API can be executed by admin only in default settings of cinder policy.json.
  So the default settings of cinder policy.json should be fixed.

  [How to reproduce]
  stack@devstack-master:/opt/devstack$ env | grep OS
  (snipped...)
  OS_USERNAME=demo
  OS_TENANT_NAME=demo
  (snipped...)
  stack@devstack-master:/opt/devstack$ nova list
  +--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
  | ID                                   | Name    | Status | Task State | Power State | Networks                                                |
  +--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
  | 5a4102cd-9e36-480c-a148-d2a127ff704e | server1 | ACTIVE | -          | Running     | private=10.0.10.3, fd61:9f1e:73ec:0:f816:3eff:fe34:8b61 |
  +--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
  stack@devstack-master:/opt/devstack$ cinder list
  +--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
  |                  ID                  |   Status  |  Name | Size | Volume Type | Bootable | Multiattach |             Attached to              |
  +--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
  | 1c768de0-949f-485f-b914-c430752e2984 | available | TEST2 |  1   | lvmdriver-1 |  false   |    False    |                                      |
  | 999a256e-aadf-41b6-88f3-1412e3462cbe |   in-use  | TEST1 |  1   | lvmdriver-1 |  false   |    False    | 5a4102cd-9e36-480c-a148-d2a127ff704e |
  +--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
  stack@devstack-master:/opt/devstack$ nova volume-update server1 999a256e-aadf-41b6-88f3-1412e3462cbe 1c768de0-949f-485f-b914-c430752e2984
  stack@devstack-master:/opt/devstack$ cinder list
  +--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
  |                  ID                  |   Status  |  Name | Size | Volume Type | Bootable | Multiattach |             Attached to              |
  +--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
  | 1c768de0-949f-485f-b914-c430752e2984 | attaching | TEST2 |  1   | lvmdriver-1 |  false   |    False    |                                      |
  | 999a256e-aadf-41b6-88f3-1412e3462cbe | detaching | TEST1 |  1   | lvmdriver-1 |  false   |    False    | 5a4102cd-9e36-480c-a148-d2a127ff704e |
  +--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+

  [cinder-api.log]
  2015-12-04 15:41:51.986 INFO cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] POST http://10.0.2.15:8776/v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action
  2015-12-04 15:41:51.986 DEBUG cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] Action body: {"os-migrate_volume_completion": {"new_volume": "1c768de0-949f-485f-b914-c430752e2984", "error": false}} from (pid=18203) get_method /opt/stack/cinder/cinder/api/openstack/wsgi.py:1093
  2015-12-04 15:41:51.987 INFO cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] http://10.0.2.15:8776/v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action returned with HTTP 403
  2015-12-04 15:41:51.989 INFO eventlet.wsgi.server [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] 10.0.2.15 - - [04/Dec/2015 15:41:51] "POST /v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action HTTP/1.1" 403 429 0.005410

  [policy.json(cinder)]
  The follwong line defines that 'migrate_volume_completion' can be executed by admin only.

  "volume_extension:volume_admin_actions:migrate_volume_completion":
  "rule:admin_api",

  [Environment]
  nova: commit 7df427fdb4d7a314b79b1e3977aeb86f019b5186(master)
  cinder: commit 707902ca482d1653072c2cf419c566cd5fee1f96(master)
  OS: Ubuntu 14.04

  stack@devstack-master:/opt/devstack$ keystone user-role-list --user demo --tenant demo
  +----------------------------------+-------------+----------------------------------+----------------------------------+
  |                id                |     name    |             user_id              |            tenant_id             |
  +----------------------------------+-------------+----------------------------------+----------------------------------+
  | dec5e6b197d6424bbfb3854f6718cef0 |    Member   | 181cd8bc1c004030aec893fc7de79618 | 722380ff5288483191cd2712cbc99c5d |
  | de75e50dec2e48b4a4c66feef8f53432 | anotherrole | 181cd8bc1c004030aec893fc7de79618 | 722380ff5288483191cd2712cbc99c5d |
  +----------------------------------+-------------+----------------------------------+----------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1522705/+subscriptions
Follow ups

[Bug 1522705] Re: Cinder volumes are stuck when non admin user executes nova swap volume API
From: OpenStack Infra, 2016-04-11
[Bug 1522705] Re: Cinder volumes are stuck when non admin user executes nova swap volume API
From: Matt Riedemann, 2016-03-31