yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1739325] Re: Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1739325@xxxxxxxxxxxxxxxxxx>
Date: Thu, 05 Jul 2018 23:33:16 -0000
Reply-to: Bug 1739325 <1739325@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/579925
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8df21c1e5fc46f2774a7653e1c9a4c2ac0b5dbfd
Submitter: Zuul
Branch:    master

commit 8df21c1e5fc46f2774a7653e1c9a4c2ac0b5dbfd
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Tue Jul 3 13:03:54 2018 -0400

    Default embedded instance.flavor.disabled attribute
    
    Before we stored flavors in instance_extra, certain fields, defined
    in nova.compute.flavors.system_metadata_flavor_props, were stored
    in the instance.system_metadata for the embedded instance.flavor.
    The "disabled" field wasn't one of those keys, however, so really
    old instances that had their embedded flavor converted to the
    serialized instance_extra form won't have the disabled attribute
    set and we need to default those here so callers, like the
    versioned notfication FlavorPayload, don't explode trying to load
    instance.flavor.disabled.
    
    Change-Id: I72f8c11b8b2df1416dec8e2d9250eac0693a30e9
    Closes-Bug: #1739325


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1739325

Title:
  Server operations fail to complete with versioned notifications if
  payload contains unset non-nullable fields

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  In Progress
Status in OpenStack Compute (nova) pike series:
  In Progress
Status in OpenStack Compute (nova) queens series:
  In Progress

Bug description:
  With versioned notifications, the instance payload tries to attach a
  flavor payload which it looks up from the instance.  It uses the one
  which is attached in instance_extras however there seems to be a
  scenario where the disabled field is missing in the database, causing
  all operations to fail in the notification stage.

  The JSON string for the flavor in the database is attached below (note
  this is a cloud with a long lifetime so it might be some weird
  conversion at some point in the life time of the cloud).

  The temporary workaround as suggested by Matt was to switch to
  unversioned notification which did the trick.

  == flavor ==
  {"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb", "vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight", "id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 80, "name": "nb.2G", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 4, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "8c6a8477-20cb-4db9-ad1d-be3bc05cdae9", "vcpu_weight": null, "id": 8}, "nova_object.namespace": "nova"}}
  == flavor ==

  == stack ==
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server [req-edc9fb83-63ff-4c4b-b6c6-704d331905a8 604d5fd332904975a26b6e89c60a9d51 d6ebcbe536f848b3af4403f922377f80 - default default] Exception during message handling: ValueError: Field `disabled' cannot be None
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 189, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 159, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 874, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 217, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 205, in decorated_function
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4125, in pause_instance
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     phase=fields.NotificationPhase.START)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 365, in notify_about_instance_action
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     fault=fault)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 138, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     super(InstanceActionPayload, self).__init__(instance=instance)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line 115, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/flavor.py", line 85, in __init__
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     self.populate_schema(flavor=flavor)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 130, in populate_schema
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     setattr(self, key, None)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     field_value = field.coerce(self, name, value)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193, in coerce
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     return self._null(obj, attr)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171, in _null
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server     raise ValueError(_("Field `%s' cannot be None") % attr)
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server ValueError: Field `disabled' cannot be None
  2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server 
  == stack ==

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1739325/+subscriptions
References

[Bug 1739325] [NEW] Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields
From: Mohammed Naser, 2017-12-20