yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1353874] [NEW] a volume will leave over caused by attaching volume failing when we create an instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: zhangtralon <zhangchunlong1@xxxxxxxxxx>
Date: Thu, 07 Aug 2014 07:39:56 -0000
Reply-to: Bug 1353874 <1353874@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

a volume will leave over caused by  attaching volume failing when we
create an instance through a boot volume created by  an image.


an example to reproduce the problem on latest juno:
(1)root@devstack:/var/log/nova# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
(2)root@devstack:/var/log/nova# cinder list
+----+--------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+----+--------+------+------+-------------+----------+-------------+
+----+--------+------+------+-------------+----------+-------------+

(3)root@devstack:/var/log/nova# nova boot --flavor m1.tiny --block-device id=d2a028fa-c430-414a-851a-01af19f6585e,source=image,dest=volume,device=vda,size=1,shutdown=removed,bootindex=0 --nic net-id=a4f849f1-fa7a-436c-832f-7d6581e77c9b tralon_volume_instance
root@devstack:/var/log/nova# nova list
+--------------------------------------+------------------------+--------+----------------------+-------------+-------------------+
| ID                                   | Name                   | Status | Task State           | Power State | Networks          |
+--------------------------------------+------------------------+--------+----------------------+-------------+-------------------+
| 00dd4141-67b7-4035-b941-281b4876f16a | tralon_volume_instance | BUILD  | block_device_mapping | NOSTATE     | private=10.0.0.10 |
+--------------------------------------+------------------------+--------+----------------------+-------------+-------------------+
(4)root@devstack:/var/log/nova# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 1cf9a1f9-dcad-427c-b20c-5961995ca08f | available |      |  1   |     None    |   true   |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+ 
(5)root@devstack:/var/log/nova# nova delete 00dd4141-67b7-4035-b941-281b4876f16a
root@devstack:/var/log/nova# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
(6)root@devstack:/var/log/nova# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 1cf9a1f9-dcad-427c-b20c-5961995ca08f | available |      |  1   |     None    |   true   |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

related log:
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00mTraceback (most recent call last):
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1717, in _prep_block_device
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    self.driver, self._await_block_device_map_created))
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 370, in attach_block_devices
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    map(_log_and_attach, block_device_mapping)
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 368, in _log_and_attach
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    bdm.attach(*attach_args, **attach_kwargs)
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 325, in attach
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    volume_api, virt_driver)
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 47, in wrapped
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    ret_val = method(obj, context, *args, **kwargs)
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 47, in wrapped
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    ret_val = method(obj, context, *args, **kwargs)
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    return self.dispatch_line(frame)
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/usr/lib/python2.7/bdb.py", line 68, in dispatch_line
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    if self.quitting: raise BdbQuit
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00mBdbQuit
2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m
2014-08-07 15:06:50.086 ERROR nova.compute.manager [^[[01;36mreq-17bcd9b3-86d4-475a-bcee-2a40f06c8ba7 ^[[00;36madmin admin] ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] Failure prepping block device^[[00m

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: volume

** Tags added: volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1353874

Title:
  a volume will leave over caused by  attaching volume failing when we
  create an instance

Status in OpenStack Compute (Nova):
  New

Bug description:
  a volume will leave over caused by  attaching volume failing when we
  create an instance through a boot volume created by  an image.

  
  an example to reproduce the problem on latest juno:
  (1)root@devstack:/var/log/nova# nova list
  +----+------+--------+------------+-------------+----------+
  | ID | Name | Status | Task State | Power State | Networks |
  +----+------+--------+------------+-------------+----------+
  +----+------+--------+------------+-------------+----------+
  (2)root@devstack:/var/log/nova# cinder list
  +----+--------+------+------+-------------+----------+-------------+
  | ID | Status | Name | Size | Volume Type | Bootable | Attached to |
  +----+--------+------+------+-------------+----------+-------------+
  +----+--------+------+------+-------------+----------+-------------+

  (3)root@devstack:/var/log/nova# nova boot --flavor m1.tiny --block-device id=d2a028fa-c430-414a-851a-01af19f6585e,source=image,dest=volume,device=vda,size=1,shutdown=removed,bootindex=0 --nic net-id=a4f849f1-fa7a-436c-832f-7d6581e77c9b tralon_volume_instance
  root@devstack:/var/log/nova# nova list
  +--------------------------------------+------------------------+--------+----------------------+-------------+-------------------+
  | ID                                   | Name                   | Status | Task State           | Power State | Networks          |
  +--------------------------------------+------------------------+--------+----------------------+-------------+-------------------+
  | 00dd4141-67b7-4035-b941-281b4876f16a | tralon_volume_instance | BUILD  | block_device_mapping | NOSTATE     | private=10.0.0.10 |
  +--------------------------------------+------------------------+--------+----------------------+-------------+-------------------+
  (4)root@devstack:/var/log/nova# cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |                  ID                  |   Status  | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | 1cf9a1f9-dcad-427c-b20c-5961995ca08f | available |      |  1   |     None    |   true   |             |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+ 
  (5)root@devstack:/var/log/nova# nova delete 00dd4141-67b7-4035-b941-281b4876f16a
  root@devstack:/var/log/nova# nova list
  +----+------+--------+------------+-------------+----------+
  | ID | Name | Status | Task State | Power State | Networks |
  +----+------+--------+------------+-------------+----------+
  +----+------+--------+------------+-------------+----------+
  (6)root@devstack:/var/log/nova# cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |                  ID                  |   Status  | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | 1cf9a1f9-dcad-427c-b20c-5961995ca08f | available |      |  1   |     None    |   true   |             |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+

  related log:
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00mTraceback (most recent call last):
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1717, in _prep_block_device
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    self.driver, self._await_block_device_map_created))
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 370, in attach_block_devices
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    map(_log_and_attach, block_device_mapping)
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 368, in _log_and_attach
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    bdm.attach(*attach_args, **attach_kwargs)
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 325, in attach
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    volume_api, virt_driver)
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 47, in wrapped
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    ret_val = method(obj, context, *args, **kwargs)
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/opt/stack/nova/nova/virt/block_device.py", line 47, in wrapped
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    ret_val = method(obj, context, *args, **kwargs)
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    return self.dispatch_line(frame)
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m  File "/usr/lib/python2.7/bdb.py", line 68, in dispatch_line
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m    if self.quitting: raise BdbQuit
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00mBdbQuit
  2014-08-07 15:06:50.085 TRACE nova.compute.manager ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] ^[[00m
  2014-08-07 15:06:50.086 ERROR nova.compute.manager [^[[01;36mreq-17bcd9b3-86d4-475a-bcee-2a40f06c8ba7 ^[[00;36madmin admin] ^[[01;35m[instance: 00dd4141-67b7-4035-b941-281b4876f16a] Failure prepping block device^[[00m

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1353874/+subscriptions
Follow ups

[Bug 1353874] Re: a volume will leave over caused by attaching volume failing when we create an instance
From: zhangtralon, 2014-08-07
[Bug 1353874] [NEW] a volume will leave over caused by attaching volume failing when we create an instance
From: zhangtralon, 2014-08-07
References

[Bug 1353874] [NEW] a volume will leave over caused by attaching volume failing when we create an instance
From: zhangtralon, 2014-08-07