yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #76144
[Bug 1807434] [NEW] Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns.
Public bug reported:
Description
===========
Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns later.
Steps to reproduce
==================
- Launch an instance:
- Stop compute service on the host:
$ sudo systemctl stop devstack@n-cpu
$ openstack compute service list
[..]
| 3 | nova-compute | host2.example.com | nova | enabled | down | 2018-12-07T18:29:35.000000 |
[..]
- Attempt to migrate the instance to another host:
$ openstack server migrate test
- Review the task_state of the instance:
$ openstack server show test
[..]
|
| OS-EXT-STS:task_state | resize_prep [..]
- Restart the compute service:
$ sudo systemctl start devstack@n-cpu
$ openstack compute service list
[..]
| 3 | nova-compute | host2.example.com | nova | enabled | up | 2018-12-07T18:42:00.000000 |
[..]
- - Review the task_state and status of the instance:
$ openstack server show test
[..]
| OS-EXT-STS:task_state | resize_migrating [..]
| status | ERROR [..]
Expected result
===============
No attempt to migrate an instance is made while the source compute host
is marked as down.
Actual result
=============
Instance left in an ERROR state after the compute service returns.
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
master
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
Trace when the compute service restarts after attempting the migration:
Dec 07 13:32:15 host2.example.com nova-compute[27522]: ERROR nova.compute.manager [None req-13de2cb9-0eac-4e59-b777-e83ce5bb1d3c admin admin] [instance: b86238ba-2f3b-4b59-bdcb-490fbb90fb52] Setting instance vm_state to ERROR: UnexpectedTaskStateError_Remote: Conflict upda>
Dec 07 13:32:15 host2.example.com nova-compute[27522]: Traceback (most recent call last):
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/conductor/manager.py", line 128, in _object_dispatch
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return getattr(target, method)(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return fn(self, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/objects/instance.py", line 843, in save
Dec 07 13:32:15 host2.example.com nova-compute[27522]: columns_to_join=_expected_cols(expected_attrs))
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/api.py", line 889, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]: expected=expected)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 170, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: ectxt.value = e.inner_exc
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 07 13:32:15 host2.example.com nova-compute[27522]: self.force_reraise()
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 07 13:32:15 host2.example.com nova-compute[27522]: six.reraise(self.type_, self.value, self.tb)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(context, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2736, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]: context, instance_uuid, values, expected, original=instance_ref))
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2872, in _instance_update
Dec 07 13:32:15 host2.example.com nova-compute[27522]: raise exc(**exc_props)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: UnexpectedTaskStateError: Conflict updating instance b86238ba-2f3b-4b59-bdcb-490fbb90fb52. Expected: {'task_state': [u'resize_prep']}. Actual: {'task_state': None}
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1807434
Title:
Instance stuck in resize_prep when migrating from down compute, moves
to ERROR when compute returns.
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns later.
Steps to reproduce
==================
- Launch an instance:
- Stop compute service on the host:
$ sudo systemctl stop devstack@n-cpu
$ openstack compute service list
[..]
| 3 | nova-compute | host2.example.com | nova | enabled | down | 2018-12-07T18:29:35.000000 |
[..]
- Attempt to migrate the instance to another host:
$ openstack server migrate test
- Review the task_state of the instance:
$ openstack server show test
[..]
|
| OS-EXT-STS:task_state | resize_prep [..]
- Restart the compute service:
$ sudo systemctl start devstack@n-cpu
$ openstack compute service list
[..]
| 3 | nova-compute | host2.example.com | nova | enabled | up | 2018-12-07T18:42:00.000000 |
[..]
- - Review the task_state and status of the instance:
$ openstack server show test
[..]
| OS-EXT-STS:task_state | resize_migrating [..]
| status | ERROR [..]
Expected result
===============
No attempt to migrate an instance is made while the source compute
host is marked as down.
Actual result
=============
Instance left in an ERROR state after the compute service returns.
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
master
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
Trace when the compute service restarts after attempting the
migration:
Dec 07 13:32:15 host2.example.com nova-compute[27522]: ERROR nova.compute.manager [None req-13de2cb9-0eac-4e59-b777-e83ce5bb1d3c admin admin] [instance: b86238ba-2f3b-4b59-bdcb-490fbb90fb52] Setting instance vm_state to ERROR: UnexpectedTaskStateError_Remote: Conflict upda>
Dec 07 13:32:15 host2.example.com nova-compute[27522]: Traceback (most recent call last):
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/conductor/manager.py", line 128, in _object_dispatch
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return getattr(target, method)(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return fn(self, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/objects/instance.py", line 843, in save
Dec 07 13:32:15 host2.example.com nova-compute[27522]: columns_to_join=_expected_cols(expected_attrs))
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/api.py", line 889, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]: expected=expected)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 170, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: ectxt.value = e.inner_exc
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 07 13:32:15 host2.example.com nova-compute[27522]: self.force_reraise()
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 07 13:32:15 host2.example.com nova-compute[27522]: six.reraise(self.type_, self.value, self.tb)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped
Dec 07 13:32:15 host2.example.com nova-compute[27522]: return f(context, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2736, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]: context, instance_uuid, values, expected, original=instance_ref))
Dec 07 13:32:15 host2.example.com nova-compute[27522]: File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2872, in _instance_update
Dec 07 13:32:15 host2.example.com nova-compute[27522]: raise exc(**exc_props)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: UnexpectedTaskStateError: Conflict updating instance b86238ba-2f3b-4b59-bdcb-490fbb90fb52. Expected: {'task_state': [u'resize_prep']}. Actual: {'task_state': None}
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1807434/+subscriptions
Follow ups
