yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #71188
[Bug 1633249] Re: Boot volume creation failure leaves secondary volume attached to broken server
Reviewed: https://review.openstack.org/528385
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=61f6751a1807d3c3ee76d0351d17a82c6e1a915a
Submitter: Zuul
Branch: master
commit 61f6751a1807d3c3ee76d0351d17a82c6e1a915a
Author: Ameed Ashour <ameed.ashour.ext@xxxxxxxxx>
Date: Wed Jan 24 09:32:24 2018 -0500
Detach volumes when VM creation fails
If the boot-volume creation fails, the data volume is left in state
"in-use", attached to the server which is now in "error" state.
The user can't detach the volume because of the server's error state.
They can delete the server, which then leaves the volume apparently
attached to a server that no longer exists, which is being fixed
separately: https://review.openstack.org/#/c/340614/
The only way out of this is to ask an administrator to reset the state of
the data volume (this option is not available to regular users by
default policy).
This change fixes the problem in the compute service such that
when the creation fails, compute manager detaches the created volumes
before putting the VM into error state. Then you can delete the instance
without care about attached volumes.
Change-Id: I8b1c05317734e14ea73dc868941351bb31210bf0
Closes-bug: #1633249
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1633249
Title:
Boot volume creation failure leaves secondary volume attached to
broken server
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) ocata series:
In Progress
Status in OpenStack Compute (nova) pike series:
In Progress
Status in OpenStack Compute (nova) queens series:
In Progress
Bug description:
Attempt to boot a server with a block device mapping that includes a
boot volume created from an image, plus an existing data volume. If
the boot-volume creation fails, the data volume is left in state "in-
use", attached to the server which is now in "error" state". The user
can't detach the volume because of the server's error state. They can
delete the server, which then leaves the volume apparently attached to
a server that no longer exists. The only way out of this is to ask an
administrator to reset the state of the data volume (this option is
not available to regular users by default policy).
The easiest way to reproduce this is to attempt to create the boot
volume from qcow2 image where the volume size is less than the image
(virtual) size.
~$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | available | data | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
~$ nova boot --flavor m1.large --availability-zone=imot04-1 --block-device 'id=9e122d18-d7a4-406d-b8f2-446cfddaa7c7,source=image,dest=volume,device=vda,size=5,bootindex=0' --block-device 'id=2e733722-8b19-4bff-bd8d-bb770554582a,source=volume,dest=volume,device=vdb,size=1,bootindex=1' ol4
+--------------------------------------+--------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | imot04-1 |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | DNTr8MG3kVmC |
| config_drive | |
| created | 2016-10-13T21:54:08Z |
| flavor | m1.large (4) |
| hostId | |
| id | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | ol4 |
| os-extended-volumes:volumes_attached | [{"id": "2e733722-8b19-4bff-bd8d-bb770554582a"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 66234fea2ccc42398a1ae5300c594d49 |
| updated | 2016-10-13T21:54:08Z |
| user_id | b2ae6b7bdac142ddb708a3550f61d998 |
+--------------------------------------+--------------------------------------------------+
~$ cinder list
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | creating | | 5 | - | false | |
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | error | | 5 | - | false | |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
~$ nova volume-detach 9541b63c-e003-4bcc-bcb8-5c0461522387 2e733722-8b19-4bff-bd8d-bb770554582a
ERROR (Conflict): Cannot 'detach_volume' instance 9541b63c-e003-4bcc-bcb8-5c0461522387 while it is in vm_state error (HTTP 409) (Request-ID: req-c2855350-f06b-4c17-b429-87a068eddfb1)
~$ nova delete 9541b63c-e003-4bcc-bcb8-5c0461522387
Request to delete server 9541b63c-e003-4bcc-bcb8-5c0461522387 has been accepted.
~$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | error | | 5 | - | false | |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
~$ nova show 9541b63c-e003-4bcc-bcb8-5c0461522387
ERROR (CommandError): No server with a name or ID of '9541b63c-e003-4bcc-bcb8-5c0461522387' exists.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1633249/+subscriptions
References
