yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #67488
[Bug 1716234] Re: Live migrate fails on Ocata
Looks like you have a misconfiguration problem or an issue with your
message queue. Could you please verify your nova.conf parameters for
accessing your Rabbit instance ?
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1716234
Title:
Live migrate fails on Ocata
Status in OpenStack Compute (nova):
Invalid
Bug description:
Instructed to report this bug by Nova.
Instance was running on Host compute3O
root@controllerO:~# openstack server show eb335c1a-836e-4e65-b900-d7fbd1b78227
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute3O |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute3O |
| OS-EXT-SRV-ATTR:instance_name | instance-0000002a |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-09-10T11:55:41.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | selfservice=172.16.1.6, 192.168.179.104 |
| config_drive | |
| created | 2017-09-10T11:54:17Z |
| flavor | C1R1D10 (1) |
| hostId | c1f359b371ca8c010ed1f0d992903906455ec5668656fb2b588b7db4 |
| id | eb335c1a-836e-4e65-b900-d7fbd1b78227 |
| image | |
| key_name | marcus-key |
| name | d1 |
| progress | 0 |
| project_id | 83a0f55c38b14931bf1fed6758af8691 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2017-09-10T14:36:41Z |
| user_id | bf2bd08f16a34c629c7d211b71b4f0b8 |
| volumes_attached | id='3771d614-11f1-4757-8b00-1181995d3fde' |
+-------------------------------------+----------------------------------------------------------+
root@controllerO:~# openstack hypervisor show compute3O
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| aggregates | [] |
| cpu_info | {"vendor": "AMD", "model": "Opteron_G3", "arch": "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "msr", "nodeid_msr", "cmov", "fpu", "pat", |
| | "monitor", "lm", "tsc", "3dnowprefetch", "nx", "fxsr", "pae", "wdt", "cmp_legacy", "skinit", "sse4a", "mmx", "cx8", "mce", "de", "cr8legacy", "ht", "pse", |
| | "3dnowext", "abm", "fxsr_opt", "popcnt", "mca", "pdpe1gb", "apic", "sse", "3dnow", "invtsc", "pni", "rdtscp", "svm", "mmxext", "osvw", "sse2", "lahf_lm", |
| | "misalignsse", "cx16", "extapic", "pse36", "mtrr", "ibs"], "topology": {"cores": 6, "cells": 8, "threads": 1, "sockets": 1}} |
| current_workload | 0 |
| disk_available_least | 186 |
| free_disk_gb | 186 |
| free_ram_mb | 256412 |
| host_ip | 192.168.178.237 |
| host_time | 16:46:00 |
| hypervisor_hostname | compute3O |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 2 |
| load_average | 0.11, 0.03, 0.01 |
| local_gb | 196 |
| local_gb_used | 0 |
| memory_mb | 257948 |
| memory_mb_used | 1312 |
| running_vms | 1 |
| service_host | compute3O |
| service_id | 8 |
| state | up |
| status | enabled |
| uptime | 2:56 |
| users | 0 |
| vcpus | 48 |
| vcpus_used | 1 |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
and should be migrated to compute1O:
root@controllerO:~# openstack hypervisor show compute1O
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| aggregates | [] |
| cpu_info | {"vendor": "AMD", "model": "Opteron_G3", "arch": "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "msr", "cmov", "fpu", "pat", "monitor", "lm", |
| | "tsc", "3dnowprefetch", "nx", "fxsr", "pae", "wdt", "cmp_legacy", "skinit", "sse4a", "mmx", "cx8", "mce", "de", "cr8legacy", "ht", "pse", "3dnowext", "abm", |
| | "fxsr_opt", "popcnt", "mca", "pdpe1gb", "apic", "sse", "3dnow", "invtsc", "pni", "rdtscp", "svm", "mmxext", "osvw", "sse2", "lahf_lm", "misalignsse", "cx16", |
| | "extapic", "pse36", "mtrr", "ibs"], "topology": {"cores": 4, "cells": 4, "threads": 1, "sockets": 1}} |
| current_workload | 0 |
| disk_available_least | 114 |
| free_disk_gb | 115 |
| free_ram_mb | 128427 |
| host_ip | 192.168.178.235 |
| host_time | 16:47:45 |
| hypervisor_hostname | compute1O |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 3 |
| load_average | 1.61, 1.71, 1.78 |
| local_gb | 115 |
| local_gb_used | 0 |
| memory_mb | 128939 |
| memory_mb_used | 734 |
| running_vms | 0 |
| service_host | compute1O |
| service_id | 9 |
| state | up |
| status | enabled |
| uptime | 3:14 |
| users | 1 |
| vcpus | 16 |
| vcpus_used | 0 |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Command was:
root@controllerO:~# openstack server migrate --live compute1o eb335c1a-836e-4e65-b900-d7fbd1b78227
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-a84a7163-98f6-496e-8965-656db5773a10)
nova-api.log excerpt follows:
:
:
2017-09-10 16:36:36.902 3260 INFO nova.osapi_compute.wsgi.server [req-bd9d463c-fcc1-4d09-ad81-b3b19d9880ba bf2bd08f16a34c629c7d211b71b4f0b8 83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET /v2.1/servers/eb335c1a-836e-4e65-b900-d7fbd1b78227 HTTP/1.1" status: 200 len: 1763 time: 0.2645531
2017-09-10 16:36:36.930 3260 INFO nova.osapi_compute.wsgi.server [req-135317cd-f1e6-4e49-bb63-520db5b5567a bf2bd08f16a34c629c7d211b71b4f0b8 83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET /v2.1/flavors/1 HTTP/1.1" status: 200 len: 691 time: 0.0239990
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions [req-a84a7163-98f6-496e-8965-656db5773a10 ca81492f80b3453ab0b27db819c0396e a504f787d71245109b33822cfb9771cf - default default] Unexpected exception in API method
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/migrate_server.py", line 100, in _migrate_live
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions disk_over_commit, host, force, async)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 170, in inner
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 178, in _wrapped
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return fn(self, context, instance, *args, **kwargs)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in inner
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return f(self, context, instance, *args, **kw)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3910, in live_migrate
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions messaging_timeout)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions self.force_reraise()
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3902, in live_migrate
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions request_spec=request_spec, async=async)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 112, in live_migrate_instance
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions request_spec=request_spec)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 322, in migrate_server
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions return cctxt.call(context, 'migrate_server', **kw)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions retry=self.retry)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions retry=retry)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in _send
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in wait
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 77a90c5ad6ca4097a1407619e37f1890
2017-09-10 16:36:41.174 3258 ERROR nova.api.openstack.extensions
2017-09-10 16:36:41.178 3258 INFO nova.api.openstack.wsgi [req-a84a7163-98f6-496e-8965-656db5773a10 ca81492f80b3453ab0b27db819c0396e a504f787d71245109b33822cfb9771cf - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2017-09-10 16:36:41.180 3258 INFO nova.osapi_compute.wsgi.server [req-a84a7163-98f6-496e-8965-656db5773a10 ca81492f80b3453ab0b27db819c0396e a504f787d71245109b33822cfb9771cf - default default] 192.168.178.212 "POST /v2.1/servers/eb335c1a-836e-4e65-b900-d7fbd1b78227/action HTTP/1.1" status: 500 len: 583 time: 60.1477718
2017-09-10 16:36:41.345 3258 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 77a90c5ad6ca4097a1407619e37f1890
2017-09-10 16:36:50.448 3250 INFO nova.osapi_compute.wsgi.server [req-997edc7b-fc4d-4d96-8d12-2bb0810e4af5 bf2bd08f16a34c629c7d211b71b4f0b8 83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET /v2.1/servers/eb335c1a-836e-4e65-b900-d7fbd1b78227 HTTP/1.1" status: 200 len: 1753 time: 0.3524611
2017-09-10 16:36:50.475 3250 INFO nova.osapi_compute.wsgi.server [req-9126da6b-10bb-4ed0-b432-3b80c6f7b80f bf2bd08f16a34c629c7d211b71b4f0b8 83a0f55c38b14931bf1fed6758af8691 - default default] 192.168.178.212 "GET /v2.1/flavors/1 HTTP/1.1" status: 200 len: 691 time: 0.0227520
:
:
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1716234/+subscriptions
References
