yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1550639] [NEW] After migrate volume being attached instance, the instance cann't run normally

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: YaoZheng_ZTE <zheng.yao1@xxxxxxxxxx>
Date: Sat, 27 Feb 2016 04:47:21 -0000
Reply-to: Bug 1550639 <1550639@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Reproducing method as following:
1.create a volume from image 
[root@2C5_10_DELL05 ~(keystone_admin)]# cinder  create --image-id fd8330b3-a307-4140-8fe0-01341b583e26 --name test_image_volume  --volume-type  KSIP 1 
+---------------------------------------+--------------------------------------+
|                Property               |                Value                 |
+---------------------------------------+--------------------------------------+
|              attachments              |                  []                  |
|           availability_zone           |                 nova                 |
|                bootable               |                false                 |
|          consistencygroup_id          |                 None                 |
|               created_at              |      2016-02-27T04:20:37.000000      |
|              description              |                 None                 |
|               encrypted               |                False                 |
|                   id                  | a0dae16a-2669-49c7-a118-250c31adc655 |
|                metadata               |                  {}                  |
|              multiattach              |                False                 |
|                  name                 |          test_image_volume           |
|         os-vol-host-attr:host         |                 None                 |
|     os-vol-mig-status-attr:migstat    |                 None                 |
|     os-vol-mig-status-attr:name_id    |                 None                 |
|      os-vol-tenant-attr:tenant_id     |   181a578bc97642f2b9e153bec622f130   |
|   os-volume-replication:driver_data   |                 None                 |
| os-volume-replication:extended_status |                 None                 |
|           replication_status          |               disabled               |
|                  size                 |                  1                   |
|              snapshot_id              |                 None                 |
|              source_volid             |                 None                 |
|                 status                |               creating               |
|                user_id                |   8b34e1ab75024fcba0ea69a6fd0937c3   |
|              volume_type              |                 KSIP                 |
+---------------------------------------+--------------------------------------+
2、boot a instance from the step 1 volume.
[root@2C5_10_DELL05 ~(keystone_admin)]# nova boot --flavor 1  --block-device id=a0dae16a-2669-49c7-a118-250c31adc655,source=volume,dest=volume,bootindex=0 --nic net-id=5c8f7e7a-5a75-48eb-9c68-096278585c18 test_vm
+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | nova                                             |
| OS-EXT-SRV-ATTR:host                 | -                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000647                                |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | JEeW4BR4WL3a                                     |
| autostart                            | TRUE                                             |
| boot_index_type                      |                                                  |
| config_drive                         |                                                  |
| created                              | 2016-02-27T04:22:42Z                             |
| flavor                               | m1.tiny (1)                                      |
| hostId                               |                                                  |
| id                                   | a740b3da-42e7-4cba-9408-8df3b4846dcc             |
| image                                | Attempt to boot from volume - no image supplied  |
| key_name                             | -                                                |
| metadata                             | {}                                               |
| move                                 | TRUE                                             |
| name                                 | test_vm                                          |
| novnc                                | TRUE                                             |
| os-extended-volumes:volumes_attached | [{"id": "a0dae16a-2669-49c7-a118-250c31adc655"}] |
| priority                             | 50                                               |
| progress                             | 0                                                |
| qos                                  |                                                  |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | 181a578bc97642f2b9e153bec622f130                 |
| updated                              | 2016-02-27T04:22:43Z                             |
| user_id                              | 8b34e1ab75024fcba0ea69a6fd0937c3                 |
+--------------------------------------+--------------------------------------------------+
3. migrate the in-use status volume
[root@2C5_10_DELL05 ~(keystone_admin)]# cinder migrate a0dae16a-2669-49c7-a118-250c31adc655 2C5_10_DELL05@KS3200ISCSIDriver-2#KS3200_IPSAN
4. migrate volume fail, nova-compute.log as following:
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8699, in swap_volume
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     new_volume_id)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     payload)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 379, in decorated_function
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 407, in decorated_function
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 395, in decorated_function
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5965, in swap_volume
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     new_volume_id)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5932, in _swap_volume
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, new_volume_id)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5913, in _swap_volume
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     resize_to)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in swap_volume
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     self._disconnect_volume(old_connection_info, disk_dev)
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1092, in _disconnect_volume
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     raise
2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
5. Then, the instance is still running ,active. but  login to the virtual machine system, find the guest OS  changed read-only file system.

** Affects: nova
     Importance: Undecided
     Assignee: YaoZheng_ZTE (zheng-yao1)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => YaoZheng_ZTE (zheng-yao1)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1550639

Title:
  After migrate volume  being attached instance, the instance cann't
  run normally

Status in OpenStack Compute (nova):
  New

Bug description:
  Reproducing method as following:
  1.create a volume from image 
  [root@2C5_10_DELL05 ~(keystone_admin)]# cinder  create --image-id fd8330b3-a307-4140-8fe0-01341b583e26 --name test_image_volume  --volume-type  KSIP 1 
  +---------------------------------------+--------------------------------------+
  |                Property               |                Value                 |
  +---------------------------------------+--------------------------------------+
  |              attachments              |                  []                  |
  |           availability_zone           |                 nova                 |
  |                bootable               |                false                 |
  |          consistencygroup_id          |                 None                 |
  |               created_at              |      2016-02-27T04:20:37.000000      |
  |              description              |                 None                 |
  |               encrypted               |                False                 |
  |                   id                  | a0dae16a-2669-49c7-a118-250c31adc655 |
  |                metadata               |                  {}                  |
  |              multiattach              |                False                 |
  |                  name                 |          test_image_volume           |
  |         os-vol-host-attr:host         |                 None                 |
  |     os-vol-mig-status-attr:migstat    |                 None                 |
  |     os-vol-mig-status-attr:name_id    |                 None                 |
  |      os-vol-tenant-attr:tenant_id     |   181a578bc97642f2b9e153bec622f130   |
  |   os-volume-replication:driver_data   |                 None                 |
  | os-volume-replication:extended_status |                 None                 |
  |           replication_status          |               disabled               |
  |                  size                 |                  1                   |
  |              snapshot_id              |                 None                 |
  |              source_volid             |                 None                 |
  |                 status                |               creating               |
  |                user_id                |   8b34e1ab75024fcba0ea69a6fd0937c3   |
  |              volume_type              |                 KSIP                 |
  +---------------------------------------+--------------------------------------+
  2、boot a instance from the step 1 volume.
  [root@2C5_10_DELL05 ~(keystone_admin)]# nova boot --flavor 1  --block-device id=a0dae16a-2669-49c7-a118-250c31adc655,source=volume,dest=volume,bootindex=0 --nic net-id=5c8f7e7a-5a75-48eb-9c68-096278585c18 test_vm
  +--------------------------------------+--------------------------------------------------+
  | Property                             | Value                                            |
  +--------------------------------------+--------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                           |
  | OS-EXT-AZ:availability_zone          | nova                                             |
  | OS-EXT-SRV-ATTR:host                 | -                                                |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00000647                                |
  | OS-EXT-STS:power_state               | 0                                                |
  | OS-EXT-STS:task_state                | scheduling                                       |
  | OS-EXT-STS:vm_state                  | building                                         |
  | OS-SRV-USG:launched_at               | -                                                |
  | OS-SRV-USG:terminated_at             | -                                                |
  | accessIPv4                           |                                                  |
  | accessIPv6                           |                                                  |
  | adminPass                            | JEeW4BR4WL3a                                     |
  | autostart                            | TRUE                                             |
  | boot_index_type                      |                                                  |
  | config_drive                         |                                                  |
  | created                              | 2016-02-27T04:22:42Z                             |
  | flavor                               | m1.tiny (1)                                      |
  | hostId                               |                                                  |
  | id                                   | a740b3da-42e7-4cba-9408-8df3b4846dcc             |
  | image                                | Attempt to boot from volume - no image supplied  |
  | key_name                             | -                                                |
  | metadata                             | {}                                               |
  | move                                 | TRUE                                             |
  | name                                 | test_vm                                          |
  | novnc                                | TRUE                                             |
  | os-extended-volumes:volumes_attached | [{"id": "a0dae16a-2669-49c7-a118-250c31adc655"}] |
  | priority                             | 50                                               |
  | progress                             | 0                                                |
  | qos                                  |                                                  |
  | security_groups                      | default                                          |
  | status                               | BUILD                                            |
  | tenant_id                            | 181a578bc97642f2b9e153bec622f130                 |
  | updated                              | 2016-02-27T04:22:43Z                             |
  | user_id                              | 8b34e1ab75024fcba0ea69a6fd0937c3                 |
  +--------------------------------------+--------------------------------------------------+
  3. migrate the in-use status volume
  [root@2C5_10_DELL05 ~(keystone_admin)]# cinder migrate a0dae16a-2669-49c7-a118-250c31adc655 2C5_10_DELL05@KS3200ISCSIDriver-2#KS3200_IPSAN
  4. migrate volume fail, nova-compute.log as following:
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8699, in swap_volume
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     new_volume_id)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 379, in decorated_function
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 407, in decorated_function
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 395, in decorated_function
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5965, in swap_volume
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     new_volume_id)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5932, in _swap_volume
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, new_volume_id)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5913, in _swap_volume
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     resize_to)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1241, in swap_volume
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     self._disconnect_volume(old_connection_info, disk_dev)
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1092, in _disconnect_volume
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher     raise
  2016-02-27 11:32:47.986 29370 TRACE oslo_messaging.rpc.dispatcher TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
  5. Then, the instance is still running ,active. but  login to the virtual machine system, find the guest OS  changed read-only file system.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1550639/+subscriptions
Follow ups

[Bug 1550639] Re: After migrate volume being attached instance, the instance cann't run normally
From: Matt Riedemann, 2016-03-06