← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1807434] [NEW] Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns.

 

Public bug reported:

Description
===========
Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns later.

Steps to reproduce
==================

- Launch an instance:

- Stop compute service on the host:

$ sudo systemctl stop devstack@n-cpu

$ openstack compute service list
[..]
|  3 | nova-compute     | host2.example.com | nova     | enabled | down  | 2018-12-07T18:29:35.000000 |
[..]

- Attempt to migrate the instance to another host:

$ openstack server migrate test

- Review the task_state of the instance:

$ openstack server show test
[..]
|
| OS-EXT-STS:task_state               | resize_prep                                                     [..]

- Restart the compute service:

$ sudo systemctl start devstack@n-cpu

$ openstack compute service list
[..]
|  3 | nova-compute     | host2.example.com | nova     | enabled | up    | 2018-12-07T18:42:00.000000 |
[..]

- - Review the task_state and status of the instance:

$ openstack server show test
[..]
| OS-EXT-STS:task_state               | resize_migrating                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             [..]
| status                              | ERROR                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        [..]


Expected result
===============

No attempt to migrate an instance is made while the source compute host
is marked as down.

Actual result
=============

Instance left in an ERROR state after the compute service returns.


Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   master

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

Trace when the compute service restarts after attempting the migration:

Dec 07 13:32:15 host2.example.com nova-compute[27522]: ERROR nova.compute.manager [None req-13de2cb9-0eac-4e59-b777-e83ce5bb1d3c admin admin] [instance: b86238ba-2f3b-4b59-bdcb-490fbb90fb52] Setting instance vm_state to ERROR: UnexpectedTaskStateError_Remote: Conflict upda>
Dec 07 13:32:15 host2.example.com nova-compute[27522]: Traceback (most recent call last):
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/conductor/manager.py", line 128, in _object_dispatch
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return getattr(target, method)(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return fn(self, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/objects/instance.py", line 843, in save
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     columns_to_join=_expected_cols(expected_attrs))
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/api.py", line 889, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     expected=expected)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 170, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     ectxt.value = e.inner_exc
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     self.force_reraise()
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     six.reraise(self.type_, self.value, self.tb)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return f(*args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return f(context, *args, **kwargs)
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2736, in instance_update_and_get_original
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     context, instance_uuid, values, expected, original=instance_ref))
Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2872, in _instance_update
Dec 07 13:32:15 host2.example.com nova-compute[27522]:     raise exc(**exc_props)
Dec 07 13:32:15 host2.example.com nova-compute[27522]: UnexpectedTaskStateError: Conflict updating instance b86238ba-2f3b-4b59-bdcb-490fbb90fb52. Expected: {'task_state': [u'resize_prep']}. Actual: {'task_state': None}

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1807434

Title:
  Instance stuck in resize_prep when migrating from down compute, moves
  to ERROR when compute returns.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Instance stuck in resize_prep when migrating from down compute, moves to ERROR when compute returns later.

  Steps to reproduce
  ==================

  - Launch an instance:

  - Stop compute service on the host:

  $ sudo systemctl stop devstack@n-cpu

  $ openstack compute service list
  [..]
  |  3 | nova-compute     | host2.example.com | nova     | enabled | down  | 2018-12-07T18:29:35.000000 |
  [..]

  - Attempt to migrate the instance to another host:

  $ openstack server migrate test

  - Review the task_state of the instance:

  $ openstack server show test
  [..]
  |
  | OS-EXT-STS:task_state               | resize_prep                                                     [..]

  - Restart the compute service:

  $ sudo systemctl start devstack@n-cpu

  $ openstack compute service list
  [..]
  |  3 | nova-compute     | host2.example.com | nova     | enabled | up    | 2018-12-07T18:42:00.000000 |
  [..]

  - - Review the task_state and status of the instance:

  $ openstack server show test
  [..]
  | OS-EXT-STS:task_state               | resize_migrating                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             [..]
  | status                              | ERROR                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        [..]

  
  Expected result
  ===============

  No attempt to migrate an instance is made while the source compute
  host is marked as down.

  Actual result
  =============

  Instance left in an ERROR state after the compute service returns.

  
  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     master

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

  Trace when the compute service restarts after attempting the
  migration:

  Dec 07 13:32:15 host2.example.com nova-compute[27522]: ERROR nova.compute.manager [None req-13de2cb9-0eac-4e59-b777-e83ce5bb1d3c admin admin] [instance: b86238ba-2f3b-4b59-bdcb-490fbb90fb52] Setting instance vm_state to ERROR: UnexpectedTaskStateError_Remote: Conflict upda>
  Dec 07 13:32:15 host2.example.com nova-compute[27522]: Traceback (most recent call last):
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/conductor/manager.py", line 128, in _object_dispatch
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return getattr(target, method)(*args, **kwargs)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return fn(self, *args, **kwargs)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/objects/instance.py", line 843, in save
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     columns_to_join=_expected_cols(expected_attrs))
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/api.py", line 889, in instance_update_and_get_original
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     expected=expected)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 170, in wrapper
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return f(*args, **kwargs)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     ectxt.value = e.inner_exc
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     self.force_reraise()
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     six.reraise(self.type_, self.value, self.tb)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return f(*args, **kwargs)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapped
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     return f(context, *args, **kwargs)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2736, in instance_update_and_get_original
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     context, instance_uuid, values, expected, original=instance_ref))
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2872, in _instance_update
  Dec 07 13:32:15 host2.example.com nova-compute[27522]:     raise exc(**exc_props)
  Dec 07 13:32:15 host2.example.com nova-compute[27522]: UnexpectedTaskStateError: Conflict updating instance b86238ba-2f3b-4b59-bdcb-490fbb90fb52. Expected: {'task_state': [u'resize_prep']}. Actual: {'task_state': None}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1807434/+subscriptions


Follow ups