yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1669924] [NEW] InstanceUpdateConflict: Conflict updating instance Expected: {'vm_state': ['building', None]}. Actual: {'vm_state': u'error'}

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: sajuptpm <1669924@xxxxxxxxxxxxxxxxxx>
Date: Fri, 03 Mar 2017 22:56:01 -0000
Reply-to: Bug 1669924 <1669924@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

VM creation got stuck at nova-cells. I can see following error in nova-cells.
Because of this issue, nova scheduler not getting any Compute nodes to schedule the VM.
Please le me know if you need more infos.
I am using devstack master (master nova)

-------------------

Log form nova-cells
--------------------------

2017-03-03 22:43:08.723 ERROR nova.cells.messaging [req-719c3e5b-8a74-446e-88ea-cd21a4c038ed nfv nfv_user] Error processing message locally
2017-03-03 22:43:08.723 TRACE nova.cells.messaging Traceback (most recent call last):
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 188, in _process_locally
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1275, in _process_message_locally
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1024, in instance_update_at_top
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     expected_task_state=expected_task_state)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrappe
r
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return fn(self, *args, **kwargs)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/objects/instance.py", line 777, in save
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     columns_to_join=_expected_cols(expected_attrs))
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/api.py", line 840, in instance_update_and_get_original
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     expected=expected)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return f(*args, **kwargs)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     ectxt.value = e.inner_exc
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     self.force_reraise()
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     six.reraise(self.type_, self.value, self.tb)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return f(*args, **kwargs)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return f(context, *args, **kwargs)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2710, in instance_update_and_get_original
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     context, instance_uuid, values, expected, original=instance_ref))
2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2846, in _instance_update
2017-03-03 22:43:08.723 TRACE nova.cells.messaging     raise exc(**exc_props)
2017-03-03 22:43:08.723 TRACE nova.cells.messaging InstanceUpdateConflict: Conflict updating instance 4d3dae25-906a-4e38-9075-1417c6fa7ad9. Expected:
 {'vm_state': ['building', None]}. Actual: {'vm_state': u'error'}
2017-03-03 22:43:08.723 TRACE nova.cells.messaging

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1669924

Title:
  InstanceUpdateConflict: Conflict updating instance Expected:
  {'vm_state': ['building', None]}. Actual: {'vm_state': u'error'}

Status in OpenStack Compute (nova):
  New

Bug description:
  VM creation got stuck at nova-cells. I can see following error in nova-cells.
  Because of this issue, nova scheduler not getting any Compute nodes to schedule the VM.
  Please le me know if you need more infos.
  I am using devstack master (master nova)

  -------------------

  Log form nova-cells
  --------------------------

  2017-03-03 22:43:08.723 ERROR nova.cells.messaging [req-719c3e5b-8a74-446e-88ea-cd21a4c038ed nfv nfv_user] Error processing message locally
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging Traceback (most recent call last):
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 188, in _process_locally
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1275, in _process_message_locally
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1024, in instance_update_at_top
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     expected_task_state=expected_task_state)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrappe
  r
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return fn(self, *args, **kwargs)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/objects/instance.py", line 777, in save
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     columns_to_join=_expected_cols(expected_attrs))
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/api.py", line 840, in instance_update_and_get_original
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     expected=expected)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return f(*args, **kwargs)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     ectxt.value = e.inner_exc
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     self.force_reraise()
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     six.reraise(self.type_, self.value, self.tb)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return f(*args, **kwargs)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     return f(context, *args, **kwargs)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2710, in instance_update_and_get_original
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     context, instance_uuid, values, expected, original=instance_ref))
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2846, in _instance_update
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging     raise exc(**exc_props)
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging InstanceUpdateConflict: Conflict updating instance 4d3dae25-906a-4e38-9075-1417c6fa7ad9. Expected:
   {'vm_state': ['building', None]}. Actual: {'vm_state': u'error'}
  2017-03-03 22:43:08.723 TRACE nova.cells.messaging

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1669924/+subscriptions
Follow ups

[Bug 1669924] Re: InstanceUpdateConflict: Conflict updating instance Expected: {'vm_state': ['building', None]}. Actual: {'vm_state': u'error'}
From: Launchpad Bug Tracker, 2017-08-28