yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #40802
[Bug 1457291] Re: Volume connection to destination host is not terminated after failed to block live migrate a VM with attached volume
This has been fixed in Liberty - https://review.openstack.org/214434.
Because kilo is security-supported only marking this one as invalid.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1457291
Title:
Volume connection to destination host is not terminated after failed
to block live migrate a VM with attached volume
Status in OpenStack Compute (nova):
Invalid
Bug description:
I was tried to block live migrate a VM with a attached volume. It was failed as expected due to change in bug https://bugs.launchpad.net/nova/+bug/1398999
However, after the migration failed, the volume connection to the destination host is not terminated. This result in that the volume is not able to be deleted after attached from the VM(VNX is used a cinder backend).
Log at the Destination host:
Exception that made the live migration failed
2015-05-20 23:01:43.644 ERROR oslo_messaging._drivers.common [req-ac891c95-e958-4166-9eb6-a459f05356f0 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 6681, in pre_live_migration\n disk, migrate_data=migrate_data)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 443, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 355, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 343, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 5163, in pre_live_migration\n migrate_data)\n', ' File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5825, in pre_live_migration\n raise exception.MigrationError(reason=msg)\n', 'MigrationError: Migration error: Cannot block migrate instance 728f053b-0333-4594-b25b-1c104be66313 with mapped volumes\n']
There is log to initialize the connection between the volume and the target host
stack@ubuntu-server12:/opt/stack/logs/screen$ grep req-ac891c95-e958-4166-9eb6-a459f05356f0 screen-n-cpu.log | grep initialize
2015-05-20 23:01:39.379 DEBUG keystoneclient.session [req-ac891c95-e958-4166-9eb6-a459f05356f0 admin admin] REQ: curl -g -i -X POST http://192.168.1.12:8776/v2/e6c8e065eee54e369a0fe7bca2759213/volumes/b895ded9-d337-45a0-8eb8-658faabf3e7e/action -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}7f619e45d624a874185329f549070513a45eb324" -d '{"os-initialize_connection": {"connector": {"ip": "192.168.1.12", "host": "ubuntu-server12", "wwnns": ["20000090fa534685", "20000090fa534684"], "initiator": "iqn.1993-08.org.debian:01:f261dc5728b2", "wwpns": ["10000090fa534685", "10000090fa534684"]}}}' _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:195
But there is no request to terminate the connection between the volume and the target host
stack@ubuntu-server12:/opt/stack/logs/screen$ grep req-ac891c95-e958-4166-9eb6-a459f05356f0 screen-n-cpu.log | grep terminate_connection
In the cinder api log, the last request about the volume is initialize connection to the target host. There is no terminate connection request after that.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep b895ded9-d337-45a0-8eb8-658faabf3e7e screen-c-api.log
........
2015-05-20 23:01:39.444 INFO cinder.api.openstack.wsgi [req-43a51010-2cb9-4cae-8684-3fa5f82c71de admin] POST http://192.168.1.12:8776/v2/e6c8e065eee54e369a0fe7bca2759213/volumes/b895ded9-d337-45a0-8eb8-658faabf3e7e/action
2015-05-20 23:01:39.484 DEBUG cinder.volume.api [req-43a51010-2cb9-4cae-8684-3fa5f82c71de admin] initialize connection for volume-id: b895ded9-d337-45a0-8eb8-658faabf3e7e, and connector: {u'ip': u'192.168.1.12', u'host': u'ubuntu-server12', u'wwnns': [u'20000090fa534685', u'20000090fa534684'], u'initiator': u'iqn.1993-08.org.debian:01:f261dc5728b2', u'wwpns': [u'10000090fa534685', u'10000090fa534684']}. initialize_connection /opt/stack/cinder/cinder/volume/api.py:563
2015-05-20 23:01:41.205 INFO cinder.api.openstack.wsgi [req-43a51010-2cb9-4cae-8684-3fa5f82c71de admin] http://192.168.1.12:8776/v2/e6c8e065eee54e369a0fe7bca2759213/volumes/b895ded9-d337-45a0-8eb8-658faabf3e7e/action returned with HTTP 200
2015-05-20 23:01:41.206 INFO eventlet.wsgi.server [req-43a51010-2cb9-4cae-8684-3fa5f82c71de admin] 192.168.1.12 - - [20/May/2015 23:01:41] "POST /v2/e6c8e065eee54e369a0fe7bca2759213/volumes/b895ded9-d337-45a0-8eb8-658faabf3e7e/action HTTP/1.1" 200 503 1.824016
stack@ubuntu-server12:/opt/stack/logs/screen$
My reproduce Steps:
1. Shared storage is not used between 2 compute nodes.
2. KVM is used as hyperviser and libvirt dirver is used. And VNX is
used as Cinder back end
3. Create a VM with image specified.
stack@ubuntu-server12:/opt/stack/logs/screen$ nova boot --image 7cbaae55-0c68-4776-a6df-0d5fcfa32f0f --flavor 1 VM3
stack@ubuntu-server12:/opt/stack/logs/screen$ nova show 728f053b-0333-4594-b25b-1c104be66313 | grep host
| OS-EXT-SRV-ATTR:host | ubuntu-server13 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ubuntu-server13 |
| hostId | cc27ab91a64a466dff1a6e8d1a2ccd2ae2e146ee2c553e88289e4dec |
4. Create a volume and attach it to the VM created in step 3.
stack@ubuntu-server12:/opt/stack/logs/screen$ nova volume-attach 728f053b-0333-4594-b25b-1c104be66313 b895ded9-d337-45a0-8eb8-658faabf3e7e
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | b895ded9-d337-45a0-8eb8-658faabf3e7e |
| serverId | 728f053b-0333-4594-b25b-1c104be66313 |
| volumeId | b895ded9-d337-45a0-8eb8-658faabf3e7e |
+----------+--------------------------------------+
5. Live migrate the VM to another host
stack@ubuntu-server12:/opt/stack/logs/screen$ nova live-migration --block-migrate 728f053b-0333-4594-b25b-1c104be66313 ubuntu-server12
stack@ubuntu-server12:/opt/stack/logs/screen$ nova show 728f053b-0333-4594-b25b-1c104be66313
+--------------------------------------+-----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ubuntu-server13 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ubuntu-server13 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-05-21T02:58:47.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-05-21T02:58:29Z |
| flavor | m1.tiny (1) |
| hostId | cc27ab91a64a466dff1a6e8d1a2ccd2ae2e146ee2c553e88289e4dec |
| id | 728f053b-0333-4594-b25b-1c104be66313 |
| image | cirros-0.3.1-x86_64-disk (7cbaae55-0c68-4776-a6df-0d5fcfa32f0f) |
| key_name | - |
| metadata | {} |
| name | VM3 |
| os-extended-volumes:volumes_attached | [{"id": "b895ded9-d337-45a0-8eb8-658faabf3e7e"}] |
| progress | 0 |
| public network | 172.24.4.5 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | e6c8e065eee54e369a0fe7bca2759213 |
| updated | 2015-05-21T03:01:44Z |
| user_id | f1f912916ac04f5eb18855f1c7795d0c |
+--------------------------------------+-----------------------------------------------------------------+
6. The migration is failed, the VM is till on the original host. However on the destination host the volume connection is not terminated.
Which caused the volume cannot be deleted after detached from the VM3.
stack@ubuntu-server12:/opt/stack/logs/screen$ nova volume-detach VM3 b895ded9-d337-45a0-8eb8-658faabf3e7e
stack@ubuntu-server12:/opt/stack/logs/screen$ cinder list
+--------------------------------------+-----------+----------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+----------+------+-------------+----------+--------------------------------------+
| b895ded9-d337-45a0-8eb8-658faabf3e7e | available | vol3 | 1 | None | false | |
+--------------------------------------+-----------+----------+------+-------------+----------+--------------------------------------+
stack@ubuntu-server12:/opt/stack/logs/screen$ cinder delete b895ded9-d337-45a0-8eb8-658faabf3e7e
stack@ubuntu-server12:/opt/stack/logs/screen$ cinder list
+--------------------------------------+----------------+----------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+----------+------+-------------+----------+--------------------------------------+
| b895ded9-d337-45a0-8eb8-658faabf3e7e | error_deleting | vol3 | 1 | None | false | |
+--------------------------------------+----------------+----------+------+-------------+----------+--------------------------------------+
==== Version =====
stack@ubuntu-server12:/opt/stack/nova$ git status
On branch stable/kilo
Your branch is up-to-date with 'origin/stable/kilo'.
nothing to commit, working directory clean
stack@ubuntu-server12:/opt/stack/nova$ git log -1
commit 36fb00291d819b65f46d530eefbf07b883ca8a29
Merge: 0c60aca 8e9ecf7
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date: Fri May 15 16:16:00 2015 +0000
Merge "Libvirt: Use tpool to invoke guestfs api" into stable/kilo
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1457291/+subscriptions
References
