← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1638381] [NEW] Live Migration failure: 'unicode' object has no attribute '_o'

 

Public bug reported:

nova live-migration a7ff526d-0cab-4da7-92c2-49d8e89d1d27
dev02-compute-01 --block-migrate


gives this in nova log:
2016-11-01 21:48:49.242 14425 ERROR nova.virt.libvirt.driver [req-750740d4-c988-4d91-b690-78fca9ac20b4 08e955c2ffdb49b1abeb5b5c4f8fad03 805b50c70a744113b7d87dd248111275 - - -] [instance: a7ff526d-0cab-4da7-92c2-49d8e89d1d27] Live Migration failure: 'unicode' object has no attribute '_o'
2016-11-01 21:48:49.288 14425 ERROR nova.virt.libvirt.driver [req-750740d4-c988-4d91-b690-78fca9ac20b4 08e955c2ffdb49b1abeb5b5c4f8fad03 805b50c70a744113b7d87dd248111275 - - -] [instance: a7ff526d-0cab-4da7-92c2-49d8e89d1d27] Migration operation has aborted


and instance is not live-migrated.

regular migrating works:


root@dev02-controller-01:~# nova migrate a7ff526d-0cab-4da7-92c2-49d8e89d1d27 
root@dev02-controller-01:~# nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------+----------------------------------+---------+------------------+-------------+------------------------+
| ID                                   | Name                 | Tenant ID                        | Status  | Task State       | Power State | Networks               |
+--------------------------------------+----------------------+----------------------------------+---------+------------------+-------------+------------------------+
| a7ff526d-0cab-4da7-92c2-49d8e89d1d27 | Centos7-provider-01  | 975ce89aac384a44be29269198817d00 | RESIZE  | resize_migrating | Running     | provider=10.100.16.53  |
| edbf68ea-fa00-4493-813e-214fa87599b1 | centos7-selfserved   | 975ce89aac384a44be29269198817d00 | ACTIVE  | -                | Running     | selfservice=172.16.1.5 |
| d303140e-447f-4ec6-9226-fa041a777f3c | democirr             | 975ce89aac384a44be29269198817d00 | SHUTOFF | -                | Shutdown    | provider=10.100.16.57  |
| 93aeb4c8-46b9-44f2-b32f-152305e3e747 | frontend             | 8e37b03ab46f4cc8bfc21f1659f2613b | SHUTOFF | -                | Shutdown    | provider=10.100.16.55  |
| 24104cff-d89d-4ecf-a53a-786e68a5d1e4 | fsdfs                | 975ce89aac384a44be29269198817d00 | ACTIVE  | -                | Running     | provider=10.100.16.63  |
| 9538b44e-2c92-40cc-a37e-032af7dd3cf7 | provider1            | 975ce89aac384a44be29269198817d00 | SHUTOFF | -                | Shutdown    | provider=10.100.16.52  |
| 2744b560-7105-4cb8-ada0-0dfefa92fdf4 | selfservice-instance | 975ce89aac384a44be29269198817d00 | ACTIVE  | -                | Running     | selfservice=172.16.1.3 |
+--------------------------------------+----------------------+----------------------------------+---------+------------------+-------------+------------------------+
root@dev02-controller-01:~# nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------+----------------------------------+---------------+------------+-------------+------------------------+
| ID                                   | Name                 | Tenant ID                        | Status        | Task State | Power State | Networks               |
+--------------------------------------+----------------------+----------------------------------+---------------+------------+-------------+------------------------+
| a7ff526d-0cab-4da7-92c2-49d8e89d1d27 | Centos7-provider-01  | 975ce89aac384a44be29269198817d00 | VERIFY_RESIZE | -          | Running     | provider=10.100.16.53  |
| edbf68ea-fa00-4493-813e-214fa87599b1 | centos7-selfserved   | 975ce89aac384a44be29269198817d00 | ACTIVE        | -          | Running     | selfservice=172.16.1.5 |
| d303140e-447f-4ec6-9226-fa041a777f3c | democirr             | 975ce89aac384a44be29269198817d00 | SHUTOFF       | -          | Shutdown    | provider=10.100.16.57  |
| 93aeb4c8-46b9-44f2-b32f-152305e3e747 | frontend             | 8e37b03ab46f4cc8bfc21f1659f2613b | SHUTOFF       | -          | Shutdown    | provider=10.100.16.55  |
| 24104cff-d89d-4ecf-a53a-786e68a5d1e4 | fsdfs                | 975ce89aac384a44be29269198817d00 | ACTIVE        | -          | Running     | provider=10.100.16.63  |
| 9538b44e-2c92-40cc-a37e-032af7dd3cf7 | provider1            | 975ce89aac384a44be29269198817d00 | SHUTOFF       | -          | Shutdown    | provider=10.100.16.52  |
| 2744b560-7105-4cb8-ada0-0dfefa92fdf4 | selfservice-instance | 975ce89aac384a44be29269198817d00 | ACTIVE        | -          | Running     | selfservice=172.16.1.3 |
+--------------------------------------+----------------------+----------------------------------+---------------+------------+-------------+------------------------+
root@dev02-controller-01:~# nova resize-confirm a7ff526d-0cab-4da7-92c2-49d8e89d1d27


root@dev02-controller-01:~# nova --debug live-migration a7ff526d-0cab-4da7-92c2-49d8e89d1d27 dev02-compute-01 --block-migrate
DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:35357/v3 -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.6"
INFO (connectionpool:208) Starting new HTTP connection (1): dev02-controller-01
DEBUG (connectionpool:388) "GET /v3 HTTP/1.1" 200 259
DEBUG (session:277) RESP: [200] Content-Length: 259 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Tue, 01 Nov 2016 20:53:49 GMT x-openstack-request-id: req-a726a5de-c67c-4200-8a50-95fe8a8336f0 Content-Type: application/json X-Distribution: Ubuntu 
RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6", "links": [{"href": "http://dev02-controller-01:35357/v3/";, "rel": "self"}]}}

DEBUG (base:165) Making authentication request to http://dev02-controller-01:35357/v3/auth/tokens
DEBUG (connectionpool:388) "POST /v3/auth/tokens HTTP/1.1" 201 2945
DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5"
INFO (connectionpool:208) Starting new HTTP connection (1): dev02-controller-01
DEBUG (connectionpool:388) "GET /v2.1/805b50c70a744113b7d87dd248111275 HTTP/1.1" 404 52
DEBUG (session:277) RESP: [404] Date: Tue, 01 Nov 2016 20:53:50 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 52 X-Compute-Request-Id: req-905c2e09-57d1-4895-8dde-9de26bba034b 
RESP BODY: 404 Not Found

The resource could not be found.


DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:8774/v2.1/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5"
DEBUG (connectionpool:388) "GET /v2.1/ HTTP/1.1" 200 393
DEBUG (session:277) RESP: [200] Content-Length: 393 X-Compute-Request-Id: req-9438b4be-4e92-4f81-85ad-0e7128723592 Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.1 Date: Tue, 01 Nov 2016 20:53:50 GMT Content-Type: application/json 
RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://dev02-controller-01:8774/v2.1/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.25", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}

DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5"
DEBUG (connectionpool:388) "GET /v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27 HTTP/1.1" 200 2117
DEBUG (session:277) RESP: [200] Content-Length: 2117 X-Compute-Request-Id: req-b1b33174-eeb3-4e0b-9eb7-f278f0c02c91 Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.25 Date: Tue, 01 Nov 2016 20:53:50 GMT Content-Type: application/json 
RESP BODY: {"server": {"status": "ACTIVE", "OS-EXT-SRV-ATTR:ramdisk_id": "", "updated": "2016-11-01T20:45:22Z", "hostId": "23c69f0f16f38cd104ed15d12fd769cddb331f652ce8d4fd5c4a373e", "OS-EXT-SRV-ATTR:host": "dev02-compute-02", "addresses": {"provider": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:84:56:17", "version": 4, "addr": "10.100.16.53", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27";, "rel": "self"}, {"href": "http://dev02-controller-01:8774/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27";, "rel": "bookmark"}], "key_name": "mykey", "image": {"id": "586e1ef7-7d83-4c64-ad33-59a4e24920cb", "links": [{"href": "http://dev02-controller-01:8774/805b50c70a744113b7d87dd248111275/images/586e1ef7-7d83-4c64-ad33-59a4e24920cb";, "rel": "bookmark"}]}, "OS-EXT-SRV-ATTR:user_data": null, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000003", "OS-EXT-SRV-ATTR:root_device_name": "/dev/vda", "OS-SRV-USG:launched_at": "2016-11-01T20:47:38.000000", "locked": false, "flavor": {"id": "3", "links": [{"href": "http://dev02-controller-01:8774/805b50c70a744113b7d87dd248111275/flavors/3";, "rel": "bookmark"}]}, "id": "a7ff526d-0cab-4da7-92c2-49d8e89d1d27", "security_groups": [{"name": "default"}], "description": "Centos7-provider-01", "OS-EXT-SRV-ATTR:kernel_id": "", "host_status": "UP", "OS-EXT-AZ:availability_zone": "nova", "user_id": "0fb207dcae87430e8e0ae8010b669717", "name": "Centos7-provider-01", "OS-EXT-SRV-ATTR:launch_index": 0, "created": "2016-09-06T18:12:02Z", "tenant_id": "975ce89aac384a44be29269198817d00", "OS-DCF:diskConfig": "MANUAL", "OS-EXT-SRV-ATTR:hypervisor_hostname": "dev02-compute-02.dev.iaas.ntnu.no", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "OS-EXT-SRV-ATTR:reservation_id": "r-800tt0si", "OS-EXT-SRV-ATTR:hostname": "centos7-provider-01", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "OS-SRV-USG:terminated_at": null, "metadata": {}}}

DEBUG (session:248) REQ: curl -g -i -X POST http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27/action -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5" -d '{"os-migrateLive": {"block_migration": true, "host": "dev02-compute-01"}}'
DEBUG (connectionpool:388) "POST /v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27/action HTTP/1.1" 202 0
DEBUG (session:277) RESP: [202] Content-Length: 0 X-Compute-Request-Id: req-be31ee6d-b5d8-4384-b857-3c2ba686422e Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.25 Date: Tue, 01 Nov 2016 20:53:50 GMT Content-Type: application/json

on controller:
ii  nova-api                             2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - API frontend
ii  nova-common                          2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - common files
ii  nova-conductor                       2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - conductor service
ii  nova-consoleauth                     2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                      2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                       2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - virtual machine scheduler
ii  python-nova                          2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7


on compute 1

ii  nova-common                         2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - common files
ii  nova-compute                        2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node base
ii  nova-compute-kvm                    2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node libvirt support
ii  python-nova                         2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
ii  python-novaclient                   2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7


on compute 2

ii  nova-common                          2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - common files
ii  nova-compute                         2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node base
ii  nova-compute-kvm                     2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                 2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node libvirt support
ii  python-nova                          2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1638381

Title:
  Live Migration failure: 'unicode' object has no attribute '_o'

Status in OpenStack Compute (nova):
  New

Bug description:
  nova live-migration a7ff526d-0cab-4da7-92c2-49d8e89d1d27
  dev02-compute-01 --block-migrate

  
  gives this in nova log:
  2016-11-01 21:48:49.242 14425 ERROR nova.virt.libvirt.driver [req-750740d4-c988-4d91-b690-78fca9ac20b4 08e955c2ffdb49b1abeb5b5c4f8fad03 805b50c70a744113b7d87dd248111275 - - -] [instance: a7ff526d-0cab-4da7-92c2-49d8e89d1d27] Live Migration failure: 'unicode' object has no attribute '_o'
  2016-11-01 21:48:49.288 14425 ERROR nova.virt.libvirt.driver [req-750740d4-c988-4d91-b690-78fca9ac20b4 08e955c2ffdb49b1abeb5b5c4f8fad03 805b50c70a744113b7d87dd248111275 - - -] [instance: a7ff526d-0cab-4da7-92c2-49d8e89d1d27] Migration operation has aborted

  
  and instance is not live-migrated.

  regular migrating works:

  
  root@dev02-controller-01:~# nova migrate a7ff526d-0cab-4da7-92c2-49d8e89d1d27 
  root@dev02-controller-01:~# nova list --all
  WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
  +--------------------------------------+----------------------+----------------------------------+---------+------------------+-------------+------------------------+
  | ID                                   | Name                 | Tenant ID                        | Status  | Task State       | Power State | Networks               |
  +--------------------------------------+----------------------+----------------------------------+---------+------------------+-------------+------------------------+
  | a7ff526d-0cab-4da7-92c2-49d8e89d1d27 | Centos7-provider-01  | 975ce89aac384a44be29269198817d00 | RESIZE  | resize_migrating | Running     | provider=10.100.16.53  |
  | edbf68ea-fa00-4493-813e-214fa87599b1 | centos7-selfserved   | 975ce89aac384a44be29269198817d00 | ACTIVE  | -                | Running     | selfservice=172.16.1.5 |
  | d303140e-447f-4ec6-9226-fa041a777f3c | democirr             | 975ce89aac384a44be29269198817d00 | SHUTOFF | -                | Shutdown    | provider=10.100.16.57  |
  | 93aeb4c8-46b9-44f2-b32f-152305e3e747 | frontend             | 8e37b03ab46f4cc8bfc21f1659f2613b | SHUTOFF | -                | Shutdown    | provider=10.100.16.55  |
  | 24104cff-d89d-4ecf-a53a-786e68a5d1e4 | fsdfs                | 975ce89aac384a44be29269198817d00 | ACTIVE  | -                | Running     | provider=10.100.16.63  |
  | 9538b44e-2c92-40cc-a37e-032af7dd3cf7 | provider1            | 975ce89aac384a44be29269198817d00 | SHUTOFF | -                | Shutdown    | provider=10.100.16.52  |
  | 2744b560-7105-4cb8-ada0-0dfefa92fdf4 | selfservice-instance | 975ce89aac384a44be29269198817d00 | ACTIVE  | -                | Running     | selfservice=172.16.1.3 |
  +--------------------------------------+----------------------+----------------------------------+---------+------------------+-------------+------------------------+
  root@dev02-controller-01:~# nova list --all
  WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
  +--------------------------------------+----------------------+----------------------------------+---------------+------------+-------------+------------------------+
  | ID                                   | Name                 | Tenant ID                        | Status        | Task State | Power State | Networks               |
  +--------------------------------------+----------------------+----------------------------------+---------------+------------+-------------+------------------------+
  | a7ff526d-0cab-4da7-92c2-49d8e89d1d27 | Centos7-provider-01  | 975ce89aac384a44be29269198817d00 | VERIFY_RESIZE | -          | Running     | provider=10.100.16.53  |
  | edbf68ea-fa00-4493-813e-214fa87599b1 | centos7-selfserved   | 975ce89aac384a44be29269198817d00 | ACTIVE        | -          | Running     | selfservice=172.16.1.5 |
  | d303140e-447f-4ec6-9226-fa041a777f3c | democirr             | 975ce89aac384a44be29269198817d00 | SHUTOFF       | -          | Shutdown    | provider=10.100.16.57  |
  | 93aeb4c8-46b9-44f2-b32f-152305e3e747 | frontend             | 8e37b03ab46f4cc8bfc21f1659f2613b | SHUTOFF       | -          | Shutdown    | provider=10.100.16.55  |
  | 24104cff-d89d-4ecf-a53a-786e68a5d1e4 | fsdfs                | 975ce89aac384a44be29269198817d00 | ACTIVE        | -          | Running     | provider=10.100.16.63  |
  | 9538b44e-2c92-40cc-a37e-032af7dd3cf7 | provider1            | 975ce89aac384a44be29269198817d00 | SHUTOFF       | -          | Shutdown    | provider=10.100.16.52  |
  | 2744b560-7105-4cb8-ada0-0dfefa92fdf4 | selfservice-instance | 975ce89aac384a44be29269198817d00 | ACTIVE        | -          | Running     | selfservice=172.16.1.3 |
  +--------------------------------------+----------------------+----------------------------------+---------------+------------+-------------+------------------------+
  root@dev02-controller-01:~# nova resize-confirm a7ff526d-0cab-4da7-92c2-49d8e89d1d27


  root@dev02-controller-01:~# nova --debug live-migration a7ff526d-0cab-4da7-92c2-49d8e89d1d27 dev02-compute-01 --block-migrate
  DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
  DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
  DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
  DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
  DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
  DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
  DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:35357/v3 -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.6"
  INFO (connectionpool:208) Starting new HTTP connection (1): dev02-controller-01
  DEBUG (connectionpool:388) "GET /v3 HTTP/1.1" 200 259
  DEBUG (session:277) RESP: [200] Content-Length: 259 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Tue, 01 Nov 2016 20:53:49 GMT x-openstack-request-id: req-a726a5de-c67c-4200-8a50-95fe8a8336f0 Content-Type: application/json X-Distribution: Ubuntu 
  RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6", "links": [{"href": "http://dev02-controller-01:35357/v3/";, "rel": "self"}]}}

  DEBUG (base:165) Making authentication request to http://dev02-controller-01:35357/v3/auth/tokens
  DEBUG (connectionpool:388) "POST /v3/auth/tokens HTTP/1.1" 201 2945
  DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5"
  INFO (connectionpool:208) Starting new HTTP connection (1): dev02-controller-01
  DEBUG (connectionpool:388) "GET /v2.1/805b50c70a744113b7d87dd248111275 HTTP/1.1" 404 52
  DEBUG (session:277) RESP: [404] Date: Tue, 01 Nov 2016 20:53:50 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 52 X-Compute-Request-Id: req-905c2e09-57d1-4895-8dde-9de26bba034b 
  RESP BODY: 404 Not Found

  The resource could not be found.


  DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:8774/v2.1/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5"
  DEBUG (connectionpool:388) "GET /v2.1/ HTTP/1.1" 200 393
  DEBUG (session:277) RESP: [200] Content-Length: 393 X-Compute-Request-Id: req-9438b4be-4e92-4f81-85ad-0e7128723592 Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.1 Date: Tue, 01 Nov 2016 20:53:50 GMT Content-Type: application/json 
  RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://dev02-controller-01:8774/v2.1/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.25", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}

  DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
  DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
  DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
  DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
  DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
  DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
  DEBUG (session:248) REQ: curl -g -i -X GET http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5"
  DEBUG (connectionpool:388) "GET /v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27 HTTP/1.1" 200 2117
  DEBUG (session:277) RESP: [200] Content-Length: 2117 X-Compute-Request-Id: req-b1b33174-eeb3-4e0b-9eb7-f278f0c02c91 Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.25 Date: Tue, 01 Nov 2016 20:53:50 GMT Content-Type: application/json 
  RESP BODY: {"server": {"status": "ACTIVE", "OS-EXT-SRV-ATTR:ramdisk_id": "", "updated": "2016-11-01T20:45:22Z", "hostId": "23c69f0f16f38cd104ed15d12fd769cddb331f652ce8d4fd5c4a373e", "OS-EXT-SRV-ATTR:host": "dev02-compute-02", "addresses": {"provider": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:84:56:17", "version": 4, "addr": "10.100.16.53", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27";, "rel": "self"}, {"href": "http://dev02-controller-01:8774/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27";, "rel": "bookmark"}], "key_name": "mykey", "image": {"id": "586e1ef7-7d83-4c64-ad33-59a4e24920cb", "links": [{"href": "http://dev02-controller-01:8774/805b50c70a744113b7d87dd248111275/images/586e1ef7-7d83-4c64-ad33-59a4e24920cb";, "rel": "bookmark"}]}, "OS-EXT-SRV-ATTR:user_data": null, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000003", "OS-EXT-SRV-ATTR:root_device_name": "/dev/vda", "OS-SRV-USG:launched_at": "2016-11-01T20:47:38.000000", "locked": false, "flavor": {"id": "3", "links": [{"href": "http://dev02-controller-01:8774/805b50c70a744113b7d87dd248111275/flavors/3";, "rel": "bookmark"}]}, "id": "a7ff526d-0cab-4da7-92c2-49d8e89d1d27", "security_groups": [{"name": "default"}], "description": "Centos7-provider-01", "OS-EXT-SRV-ATTR:kernel_id": "", "host_status": "UP", "OS-EXT-AZ:availability_zone": "nova", "user_id": "0fb207dcae87430e8e0ae8010b669717", "name": "Centos7-provider-01", "OS-EXT-SRV-ATTR:launch_index": 0, "created": "2016-09-06T18:12:02Z", "tenant_id": "975ce89aac384a44be29269198817d00", "OS-DCF:diskConfig": "MANUAL", "OS-EXT-SRV-ATTR:hypervisor_hostname": "dev02-compute-02.dev.iaas.ntnu.no", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "OS-EXT-SRV-ATTR:reservation_id": "r-800tt0si", "OS-EXT-SRV-ATTR:hostname": "centos7-provider-01", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "OS-SRV-USG:terminated_at": null, "metadata": {}}}

  DEBUG (session:248) REQ: curl -g -i -X POST http://dev02-controller-01:8774/v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27/action -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}2b925bd156a461395f89d54de774916f93c8f0d5" -d '{"os-migrateLive": {"block_migration": true, "host": "dev02-compute-01"}}'
  DEBUG (connectionpool:388) "POST /v2.1/805b50c70a744113b7d87dd248111275/servers/a7ff526d-0cab-4da7-92c2-49d8e89d1d27/action HTTP/1.1" 202 0
  DEBUG (session:277) RESP: [202] Content-Length: 0 X-Compute-Request-Id: req-be31ee6d-b5d8-4384-b857-3c2ba686422e Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.25 Date: Tue, 01 Nov 2016 20:53:50 GMT Content-Type: application/json

  on controller:
  ii  nova-api                             2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - API frontend
  ii  nova-common                          2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - common files
  ii  nova-conductor                       2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - conductor service
  ii  nova-consoleauth                     2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                      2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                       2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                          2:13.1.0-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7


  on compute 1

  ii  nova-common                         2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - common files
  ii  nova-compute                        2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                   2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7


  
  on compute 2

  ii  nova-common                          2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - common files
  ii  nova-compute                         2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                     2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                 2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute - compute node libvirt support
  ii  python-nova                          2:13.1.1-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1638381/+subscriptions


Follow ups