yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #57752
[Bug 1633249] [NEW] Boot volume creation leaves secondary volume attached to broken server
Public bug reported:
Attempt to boot a server with a block device mapping that includes a
boot volume created from an image, plus an existing data volume. If the
boot-volume creation fails, the data volume is left in state "in-use",
attached to the server which is now in "error" state". The user can't
detach the volume because of the server's error state. They can delete
the server, which then leaves the volume apparently attached to a server
that no longer exists. The only way out of this is to ask an
administrator to reset the state of the data volume (this option is not
available to regular users by default policy).
The easiest way to reproduce this is to attempt to create the boot
volume from qcow2 image where the volume size is less than the image
(virtual) size.
~$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | available | data | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
~$ nova boot --flavor m1.large --availability-zone=imot04-1 --block-device 'id=9e122d18-d7a4-406d-b8f2-446cfddaa7c7,source=image,dest=volume,device=vda,size=5,bootindex=0' --block-device 'id=2e733722-8b19-4bff-bd8d-bb770554582a,source=volume,dest=volume,device=vdb,size=1,bootindex=1' ol4
+--------------------------------------+--------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | imot04-1 |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | DNTr8MG3kVmC |
| config_drive | |
| created | 2016-10-13T21:54:08Z |
| flavor | m1.large (4) |
| hostId | |
| id | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | ol4 |
| os-extended-volumes:volumes_attached | [{"id": "2e733722-8b19-4bff-bd8d-bb770554582a"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 66234fea2ccc42398a1ae5300c594d49 |
| updated | 2016-10-13T21:54:08Z |
| user_id | b2ae6b7bdac142ddb708a3550f61d998 |
+--------------------------------------+--------------------------------------------------+
~$ cinder list
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | creating | | 5 | - | false | |
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | error | | 5 | - | false | |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
~$ nova volume-detach 9541b63c-e003-4bcc-bcb8-5c0461522387 2e733722-8b19-4bff-bd8d-bb770554582a
ERROR (Conflict): Cannot 'detach_volume' instance 9541b63c-e003-4bcc-bcb8-5c0461522387 while it is in vm_state error (HTTP 409) (Request-ID: req-c2855350-f06b-4c17-b429-87a068eddfb1)
~$ nova delete 9541b63c-e003-4bcc-bcb8-5c0461522387
Request to delete server 9541b63c-e003-4bcc-bcb8-5c0461522387 has been accepted.
~$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | error | | 5 | - | false | |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
~$ nova show 9541b63c-e003-4bcc-bcb8-5c0461522387
ERROR (CommandError): No server with a name or ID of '9541b63c-e003-4bcc-bcb8-5c0461522387' exists.
** Affects: cinder
Importance: Undecided
Status: New
** Affects: nova
Importance: Undecided
Status: New
** Also affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1633249
Title:
Boot volume creation leaves secondary volume attached to broken server
Status in Cinder:
New
Status in OpenStack Compute (nova):
New
Bug description:
Attempt to boot a server with a block device mapping that includes a
boot volume created from an image, plus an existing data volume. If
the boot-volume creation fails, the data volume is left in state "in-
use", attached to the server which is now in "error" state". The user
can't detach the volume because of the server's error state. They can
delete the server, which then leaves the volume apparently attached to
a server that no longer exists. The only way out of this is to ask an
administrator to reset the state of the data volume (this option is
not available to regular users by default policy).
The easiest way to reproduce this is to attempt to create the boot
volume from qcow2 image where the volume size is less than the image
(virtual) size.
~$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | available | data | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
~$ nova boot --flavor m1.large --availability-zone=imot04-1 --block-device 'id=9e122d18-d7a4-406d-b8f2-446cfddaa7c7,source=image,dest=volume,device=vda,size=5,bootindex=0' --block-device 'id=2e733722-8b19-4bff-bd8d-bb770554582a,source=volume,dest=volume,device=vdb,size=1,bootindex=1' ol4
+--------------------------------------+--------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | imot04-1 |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | DNTr8MG3kVmC |
| config_drive | |
| created | 2016-10-13T21:54:08Z |
| flavor | m1.large (4) |
| hostId | |
| id | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | ol4 |
| os-extended-volumes:volumes_attached | [{"id": "2e733722-8b19-4bff-bd8d-bb770554582a"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 66234fea2ccc42398a1ae5300c594d49 |
| updated | 2016-10-13T21:54:08Z |
| user_id | b2ae6b7bdac142ddb708a3550f61d998 |
+--------------------------------------+--------------------------------------------------+
~$ cinder list
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | creating | | 5 | - | false | |
+--------------------------------------+----------+------+------+-------------+----------+--------------------------------------+
~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | error | | 5 | - | false | |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
~$ nova volume-detach 9541b63c-e003-4bcc-bcb8-5c0461522387 2e733722-8b19-4bff-bd8d-bb770554582a
ERROR (Conflict): Cannot 'detach_volume' instance 9541b63c-e003-4bcc-bcb8-5c0461522387 while it is in vm_state error (HTTP 409) (Request-ID: req-c2855350-f06b-4c17-b429-87a068eddfb1)
~$ nova delete 9541b63c-e003-4bcc-bcb8-5c0461522387
Request to delete server 9541b63c-e003-4bcc-bcb8-5c0461522387 has been accepted.
~$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 2e733722-8b19-4bff-bd8d-bb770554582a | in-use | data | 1 | - | false | 9541b63c-e003-4bcc-bcb8-5c0461522387 |
| a5a9f27b-8c8b-4cd5-bda8-998cc4cc6f32 | error | | 5 | - | false | |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
~$ nova show 9541b63c-e003-4bcc-bcb8-5c0461522387
ERROR (CommandError): No server with a name or ID of '9541b63c-e003-4bcc-bcb8-5c0461522387' exists.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1633249/+subscriptions
Follow ups
