yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1739325] Re: Server operations fail to complete with versioned notifications

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 20 Dec 2017 01:23:54 -0000
Reply-to: Bug 1739325 <1739325@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Here is a prettier form of the json:

{
   "new":null,
   "old":null,
   "cur":{
      "nova_object.version":"1.1",
      "nova_object.changes":[
         "root_gb",
         "name",
         "ephemeral_gb",
         "memory_mb",
         "vcpus",
         "extra_specs",
         "swap",
         "rxtx_factor",
         "flavorid",
         "vcpu_weight",
         "id"
      ],
      "nova_object.name":"Flavor",
      "nova_object.data":{
         "root_gb":80,
         "name":"nb.2G",
         "ephemeral_gb":0,
         "memory_mb":2048,
         "vcpus":4,
         "extra_specs":{

         },
         "swap":0,
         "rxtx_factor":1.0,
         "flavorid":"8c6a8477-20cb-4db9-ad1d-be3bc05cdae9",
         "vcpu_weight":null,
         "id":8
      },
      "nova_object.namespace":"nova"
   }
}

** Tags added: notifications

** Changed in: nova
   Importance: Undecided => High

** Summary changed:

- Server operations fail to complete with versioned notifications
+ Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields

** Also affects: nova/pike
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1739325

Title:
  Server operations fail to complete with versioned notifications if
  payload contains unset non-nullable fields

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) pike series:
  New

Bug description:
  With versioned notifications, the instance payload tries to attach a
  flavor payload which it looks up from the instance.  It uses the one
  which is attached in instance_extras however there seems to be a
  scenario where the disabled field is missing in the database, causing
  all operations to fail in the notification stage.

  The JSON string for the flavor in the database is attached below (note
  this is a cloud with a long lifetime so it might be some weird
  conversion at some point in the life time of the cloud).

  The temporary workaround as suggested by Matt was to switch to
  unversioned notification which did the trick.

  == flavor ==
  {"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb", "vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight", "id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 80, "name": "nb.2G", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 4, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "8c6a8477-20cb-4db9-ad1d-be3bc05cdae9", "vcpu_weight": null, "id": 8}, "nova_object.namespace": "nova"}}
  == flavor ==

  == stack ==
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server [req-edc9fb83-63ff-4c4b-b6c6-704d331905a8 604d5fd332904975a26b6e89c60a9d51 d6ebcbe536f848b3af4403f922377f80 - default default] Exception during message handling: ValueError: Field `disabled' cannot be None
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 189, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 159, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 874, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 217, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 205, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4125, in pause_instance
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     phase=fields.NotificationPhase.START)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 365, in notify_about_instance_action
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     fault=fault)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 138, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     super(InstanceActionPayload, self).__init__(instance=instance)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 115, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/flavor.py", line 85, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.populate_schema(flavor=flavor)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 130, in populate_schema
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     setattr(self, key, None)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     field_value = field.coerce(self, name, value)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193, in coerce
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._null(obj, attr)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171, in _null
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     raise ValueError(_("Field `%s' cannot be None") % attr)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server ValueError: Field `disabled' cannot be None
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server 
  == stack ==

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1739325/+subscriptions
References

[Bug 1739325] [NEW] Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields
From: Mohammed Naser, 2017-12-20