yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #83874
[Bug 1425352] Re: A volume remains attached and cannot be detached after attaching it fails
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1425352
Title:
A volume remains attached and cannot be detached after attaching it
fails
Status in OpenStack Compute (nova):
Fix Released
Bug description:
A volume has been attached in nova after attaching it fails.
But its status is 'available' in cinder.
And then, the volume cannot be detached
because an error(Invalid volume: Volume must be attached in order to detach) occurs.
The cause of the failure of attaching the volume is a rpc call timeout between nova-api and nova-compute
in 'reserve_block_device_name' call.
(a 'reserve_block_device_name' rpc call is in '_attach_volume' method in nova/compute/api.py)
A rpc call timeout occurs because of waiting to get the semaphore in nova-compute.
It takes long time when many requests for same VM instance comes.
The cause of the failure of detaching the volume is that the record
remains in a 'block_device_mapping' table.
stack@devstack-juno:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| 16eb66e3-3cda-44c5-9134-cb759ffb211b | server1 | ACTIVE | - | Running | public=10.0.2.195 |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
stack@devstack-juno:/opt/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 474964ed-e334-44c0-a904-4612c52e6618 | available | TEST | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@devstack-juno:/opt/devstack$ nova volume-attach server1 474964ed-e334-44c0-a904-4612c52e6618
ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-c371e287-5e28-410f-812c-43c193bdcffc)
* RPC time out occurred between nova-api and nova-compute in
'reserve_block_device_name' call.
stack@devstack-juno:/opt/devstack$ nova show server1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack-juno |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack-juno |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-02-24T07:54:06.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-02-24T07:53:54Z |
| flavor | m1.tiny (1) |
| hostId | 358f8c0b777e6a5218b4e297676604bbc730afe5bcc2760e64a32408 |
| id | 16eb66e3-3cda-44c5-9134-cb759ffb211b |
| image | cirros-0.3.2-x86_64-uec (7f38517f-adb5-4fe6-86be-f60d9065f00d) |
| key_name | - |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [{"id": "474964ed-e334-44c0-a904-4612c52e6618"}] |
| progress | 0 |
| public network | 10.0.2.195 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 47b5fac1e13c4e97b36cd76bcb4ca661 |
| updated | 2015-02-24T07:54:06Z |
| user_id | 94cf43ae770c4a32bf8ce817cd2608cb |
+--------------------------------------+----------------------------------------------------------------+
stack@devstack-juno:/opt/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 474964ed-e334-44c0-a904-4612c52e6618 | available | TEST | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@devstack-juno:/opt/devstack$ nova volume-detach server1 474964ed-e334-44c0-a904-4612c52e6618
ERROR (BadRequest): Invalid volume: Volume must be attached in order to detach. (HTTP 400) (Request-ID: req-cdc4b447-60b8-424c-b151-73d2f2874068)
stack@devstack-juno:/opt/devstack$ nova show server1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack-juno |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack-juno |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-02-24T07:54:06.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-02-24T07:53:54Z |
| flavor | m1.tiny (1) |
| hostId | 358f8c0b777e6a5218b4e297676604bbc730afe5bcc2760e64a32408 |
| id | 16eb66e3-3cda-44c5-9134-cb759ffb211b |
| image | cirros-0.3.2-x86_64-uec (7f38517f-adb5-4fe6-86be-f60d9065f00d) |
| key_name | - |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [{"id": "474964ed-e334-44c0-a904-4612c52e6618"}] |
| progress | 0 |
| public network | 10.0.2.195 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 47b5fac1e13c4e97b36cd76bcb4ca661 |
| updated | 2015-02-24T07:54:06Z |
| user_id | 94cf43ae770c4a32bf8ce817cd2608cb |
+--------------------------------------+----------------------------------------------------------------+
[Environment]
OS: Ubuntu 14.04 LTS
nova: stable/juno(2014.2.1)
cinder: stable/juno(2014.2.1)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1425352/+subscriptions
References
