yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1739325] [NEW] Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mohammed Naser <1739325@xxxxxxxxxxxxxxxxxx>
Date: Wed, 20 Dec 2017 01:16:20 -0000
Reply-to: Bug 1739325 <1739325@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

With versioned notifications, the instance payload tries to attach a
flavor payload which it looks up from the instance.  It uses the one
which is attached in instance_extras however there seems to be a
scenario where the disabled field is missing in the database, causing
all operations to fail in the notification stage.

The JSON string for the flavor in the database is attached below (note
this is a cloud with a long lifetime so it might be some weird
conversion at some point in the life time of the cloud).

The temporary workaround as suggested by Matt was to switch to
unversioned notification which did the trick.

== flavor ==
{"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb", "vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight", "id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 80, "name": "nb.2G", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 4, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "8c6a8477-20cb-4db9-ad1d-be3bc05cdae9", "vcpu_weight": null, "id": 8}, "nova_object.namespace": "nova"}}
== flavor ==

== stack ==
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server [req-edc9fb83-63ff-4c4b-b6c6-704d331905a8 604d5fd332904975a26b6e89c60a9d51 d6ebcbe536f848b3af4403f922377f80 - default default] Exception during message handling: ValueError: Field `disabled' cannot be None
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 189, in decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 159, in decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 874, in decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 217, in decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 205, in decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4125, in pause_instance
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     phase=fields.NotificationPhase.START)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 365, in notify_about_instance_action
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     fault=fault)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 138, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     super(InstanceActionPayload, self).__init__(instance=instance)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 115, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/flavor.py", line 85, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.populate_schema(flavor=flavor)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 130, in populate_schema
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     setattr(self, key, None)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     field_value = field.coerce(self, name, value)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193, in coerce
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._null(obj, attr)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171, in _null
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     raise ValueError(_("Field `%s' cannot be None") % attr)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server ValueError: Field `disabled' cannot be None
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server 
== stack ==

** Affects: nova
     Importance: High
     Assignee: Matt Riedemann (mriedem)
         Status: In Progress

** Affects: nova/pike
     Importance: Undecided
         Status: New


** Tags: notifications

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1739325

Title:
  Server operations fail to complete with versioned notifications if
  payload contains unset non-nullable fields

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) pike series:
  New

Bug description:
  With versioned notifications, the instance payload tries to attach a
  flavor payload which it looks up from the instance.  It uses the one
  which is attached in instance_extras however there seems to be a
  scenario where the disabled field is missing in the database, causing
  all operations to fail in the notification stage.

  The JSON string for the flavor in the database is attached below (note
  this is a cloud with a long lifetime so it might be some weird
  conversion at some point in the life time of the cloud).

  The temporary workaround as suggested by Matt was to switch to
  unversioned notification which did the trick.

  == flavor ==
  {"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb", "vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight", "id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 80, "name": "nb.2G", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 4, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "8c6a8477-20cb-4db9-ad1d-be3bc05cdae9", "vcpu_weight": null, "id": 8}, "nova_object.namespace": "nova"}}
  == flavor ==

  == stack ==
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server [req-edc9fb83-63ff-4c4b-b6c6-704d331905a8 604d5fd332904975a26b6e89c60a9d51 d6ebcbe536f848b3af4403f922377f80 - default default] Exception during message handling: ValueError: Field `disabled' cannot be None
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 189, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 159, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 874, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 217, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 205, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4125, in pause_instance
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     phase=fields.NotificationPhase.START)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 365, in notify_about_instance_action
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     fault=fault)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 138, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     super(InstanceActionPayload, self).__init__(instance=instance)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 115, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/flavor.py", line 85, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.populate_schema(flavor=flavor)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 130, in populate_schema
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     setattr(self, key, None)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     field_value = field.coerce(self, name, value)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193, in coerce
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._null(obj, attr)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171, in _null
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     raise ValueError(_("Field `%s' cannot be None") % attr)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server ValueError: Field `disabled' cannot be None
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server 
  == stack ==

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1739325/+subscriptions
Follow ups

[Bug 1739325] Re: Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields
From: OpenStack Infra, 2018-07-05
[Bug 1739325] Re: Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields
From: Matt Riedemann, 2018-07-03
[Bug 1739325] Re: Server operations fail to complete with versioned notifications
From: Matt Riedemann, 2017-12-20