yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #77374
[Bug 1818873] [NEW] When post_live_migration_at_destination fails the instance is not put into ERROR/None vm_state/task_state
Public bug reported:
Seen here:
http://logs.openstack.org/43/635343/4/check/tempest-slow-
py3/a2497ae/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_05_26_25_035103
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [-] [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Post live migration at destination ubuntu-bionic-rax-iad-0003421804 failed: AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: res = self.dispatcher.dispatch(message)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return self._do_dispatch(endpoint, method, ctxt, args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: result = func(ctxt, **new_args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: function_name, call_dict, binary, tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return f(self, context, *args, **kw)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: kwargs['instance'], e, sys.exc_info())
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: context, instance, migration.dest_compute, migration=migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6846, in _post_live_migration
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance, block_migration, dest)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/rpcapi.py", line 716, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance=instance, block_migration=block_migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/client.py", line 178, in call
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=self.retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/transport.py", line 128, in _send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] call_monitor_timeout, retry=retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] raise result
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] res = self.dispatcher.dispatch(message)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return self._do_dispatch(endpoint, method, ctxt, args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] result = func(ctxt, **new_args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] function_name, call_dict, binary, tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return f(self, context, *args, **kw)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] kwargs['instance'], e, sys.exc_info())
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] context, instance, migration.dest_compute, migration=migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Apparently we don't hit this except block which would set the instance
to ERROR state because I don't see that traceback and message in the
logs:
https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6537
So we end up with something like this (failed tempest test):
b'Details:
(TestNetworkAdvancedServerOps:test_server_connectivity_live_migration)
Server b78e10fb-44b8-4010-a72d-c86410950c15 failed to reach ACTIVE
status and task state "None" within the required time (196 s). Current
status: MIGRATING. Current task state: migrating.'
It looks like the error from post_live_migration_at_destination is
logged and ignored here:
https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6844
And post_live_migration_at_destination is what sets the fields on the
instance:
https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6971
So the task_state is never reset.
This is a mess which will take some thought to clean up because if we
get to _post_live_migration the guest is on the destination host, but
the database is all messed up if post_live_migration_at_destination
fails early.
** Affects: nova
Importance: Medium
Status: Triaged
** Tags: live-migration
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1818873
Title:
When post_live_migration_at_destination fails the instance is not put
into ERROR/None vm_state/task_state
Status in OpenStack Compute (nova):
Triaged
Bug description:
Seen here:
http://logs.openstack.org/43/635343/4/check/tempest-slow-
py3/a2497ae/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_05_26_25_035103
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [-] [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Post live migration at destination ubuntu-bionic-rax-iad-0003421804 failed: AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: res = self.dispatcher.dispatch(message)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return self._do_dispatch(endpoint, method, ctxt, args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: result = func(ctxt, **new_args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: function_name, call_dict, binary, tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return f(self, context, *args, **kw)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: kwargs['instance'], e, sys.exc_info())
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: context, instance, migration.dest_compute, migration=migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6846, in _post_live_migration
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance, block_migration, dest)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/rpcapi.py", line 716, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance=instance, block_migration=block_migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/client.py", line 178, in call
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=self.retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/transport.py", line 128, in _send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] call_monitor_timeout, retry=retry)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] raise result
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last):
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] res = self.dispatcher.dispatch(message)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return self._do_dispatch(endpoint, method, ctxt, args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] result = func(ctxt, **new_args)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] function_name, call_dict, binary, tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return f(self, context, *args, **kw)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] kwargs['instance'], e, sys.exc_info())
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise()
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] context, instance, migration.dest_compute, migration=migration)
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute'
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15]
Apparently we don't hit this except block which would set the instance
to ERROR state because I don't see that traceback and message in the
logs:
https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6537
So we end up with something like this (failed tempest test):
b'Details:
(TestNetworkAdvancedServerOps:test_server_connectivity_live_migration)
Server b78e10fb-44b8-4010-a72d-c86410950c15 failed to reach ACTIVE
status and task state "None" within the required time (196 s). Current
status: MIGRATING. Current task state: migrating.'
It looks like the error from post_live_migration_at_destination is
logged and ignored here:
https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6844
And post_live_migration_at_destination is what sets the fields on the
instance:
https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6971
So the task_state is never reset.
This is a mess which will take some thought to clean up because if we
get to _post_live_migration the guest is on the destination host, but
the database is all messed up if post_live_migration_at_destination
fails early.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1818873/+subscriptions
