← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1939193] [NEW] interface attach with QoS port fails with "cannot unpack non-iterable NoneType object" if Placement is down

 

Public bug reported:

The interface attach code does not handle if placement is down while
nova try to queries allocation candidates from placement. The
safe_connect decorator on the report client method returns None when
placement is not available but the manager code tries to unpack the
return value to a 3 tuple[1].

        res = self.reportclient.get_allocation_candidates(context, rr)
        alloc_reqs, provider_sums, version = res

This leads to a TypeError and HTTP 500 response code.

2021-08-07 09:59:28,967 WARNING [nova.scheduler.client.report] Placement API service is not responding.
2021-08-07 09:59:29,130 ERROR [oslo_messaging.rpc.server] Exception during message handling
Traceback (most rece
    nt call last):
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 71, in wrapped
    _emit_versioned_exception_notification(
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 63, in wrapped
    return f(self, context, *args, **kw)
  File "/root/rtox/nova/functional-py38/nova/compute/utils.py", line 1433, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 211, in decorated_function
    compute_utils.add_instance_fault_from_exc(context,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 200, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7735, in attach_interface
    return do_attach_interface(context, instance, network_id, port_id,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
    return f(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7732, in do_attach_interface
    return self._attach_interface(context, instance, network_id,
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7787, in _attach_interface
    result = self._allocate_port_resource_for_instance(
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7656, in _allocate_port_resource_for_instance
    alloc_reqs, provider_sums, version = res
TypeError: cannot unpack non-iterable NoneType object
2021-08-07 09:59:29,131 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method
Traceback (most recent call last):
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/urllib3/connectionpool.py", line 436, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/rtox/nova/functional-py38/nova/api/openstack/wsgi.py", line 658, in wrapped
    return f(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/api/openstack/compute/attach_interfaces.py", line 168, in create
    vif = self.compute_api.attach_interface(context,
  File "/root/rtox/nova/functional-py38/nova/comp
    ute/api.py", line 226, in inner
    return function(self, context, instance, *args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/api.py", line 153, in inner
    return f(self, context, instance, *args, **kw)
  File "/root/rtox/nova/functional-py38/nova/compute/api.py", line 5046, in attach_interface
    return self.compute_rpcapi.attach_interface(context,
  File "/root/rtox/nova/functional-py38/nova/compute/rpcapi.py", line 570, in attach_interface
    return cctxt.call(ctxt, 'attach_interface', **kw)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 175, in call
    self.transport._send(self.target, msg_ctxt, msg,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send
    return self._driver.send(target, ctxt, message,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
    return self._send(target, ctxt, message, wait_for_reply, timeout,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 209, in _send
    raise failure
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 71, in wrapped
    _emit_versioned_exception_notification(
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 63, in wrapped
    return f(self, context, *args, **kw)
  File "/root/rtox/nova/functional-py38/nova/compute/utils.py", line 1433, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 211, in decorated_function
    compute_utils.add_instance_fault_from_exc(context,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 200, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7735, in attach_interface
    return do_attach_interface(context, instance, network_id, port_id,
  File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
    return f(*args, **kwargs)
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7732, in do_attach_interface
    return self._attach_interface(context, instance, network_id,
  File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7787, in _attach_interface
    result = self._allocate_port_resource_for_instance(
  File "/root/rtox/nova/functional-
    py38/nova/compute/manager.py", line 7656, in _allocate_port_resource_for_instance
    alloc_reqs, provider_sums, version = res
TypeError: cannot unpack non-iterable NoneType object
2021-08-07 09:59:29,133 INFO [nova.api.openstack.wsgi] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'TypeError'>

[1]
https://github.com/openstack/nova/blob/99536b95bcc4c1dd09099abd858233c9352afeed/nova/compute/manager.py#L7646

** Affects: nova
     Importance: Undecided
     Assignee: Balazs Gibizer (balazs-gibizer)
         Status: New

** Affects: nova/wallaby
     Importance: Undecided
         Status: New

** Also affects: nova/wallaby
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1939193

Title:
  interface attach with QoS port fails with "cannot unpack non-iterable
  NoneType object" if Placement is down

Status in OpenStack Compute (nova):
  New
Status in OpenStack Compute (nova) wallaby series:
  New

Bug description:
  The interface attach code does not handle if placement is down while
  nova try to queries allocation candidates from placement. The
  safe_connect decorator on the report client method returns None when
  placement is not available but the manager code tries to unpack the
  return value to a 3 tuple[1].

          res = self.reportclient.get_allocation_candidates(context, rr)
          alloc_reqs, provider_sums, version = res

  This leads to a TypeError and HTTP 500 response code.

  2021-08-07 09:59:28,967 WARNING [nova.scheduler.client.report] Placement API service is not responding.
  2021-08-07 09:59:29,130 ERROR [oslo_messaging.rpc.server] Exception during message handling
  Traceback (most rece
      nt call last):
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
      result = func(ctxt, **new_args)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 71, in wrapped
      _emit_versioned_exception_notification(
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 63, in wrapped
      return f(self, context, *args, **kw)
    File "/root/rtox/nova/functional-py38/nova/compute/utils.py", line 1433, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 211, in decorated_function
      compute_utils.add_instance_fault_from_exc(context,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 200, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7735, in attach_interface
      return do_attach_interface(context, instance, network_id, port_id,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
      return f(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7732, in do_attach_interface
      return self._attach_interface(context, instance, network_id,
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7787, in _attach_interface
      result = self._allocate_port_resource_for_instance(
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7656, in _allocate_port_resource_for_instance
      alloc_reqs, provider_sums, version = res
  TypeError: cannot unpack non-iterable NoneType object
  2021-08-07 09:59:29,131 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method
  Traceback (most recent call last):
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/urllib3/connectionpool.py", line 436, in _make_request
      httplib_response = conn.getresponse(buffering=True)
  TypeError: getresponse() got an unexpected keyword argument 'buffering'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/root/rtox/nova/functional-py38/nova/api/openstack/wsgi.py", line 658, in wrapped
      return f(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/api/validation/__init__.py", line 110, in wrapper
      return func(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/api/validation/__init__.py", line 110, in wrapper
      return func(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/api/openstack/compute/attach_interfaces.py", line 168, in create
      vif = self.compute_api.attach_interface(context,
    File "/root/rtox/nova/functional-py38/nova/comp
      ute/api.py", line 226, in inner
      return function(self, context, instance, *args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/api.py", line 153, in inner
      return f(self, context, instance, *args, **kw)
    File "/root/rtox/nova/functional-py38/nova/compute/api.py", line 5046, in attach_interface
      return self.compute_rpcapi.attach_interface(context,
    File "/root/rtox/nova/functional-py38/nova/compute/rpcapi.py", line 570, in attach_interface
      return cctxt.call(ctxt, 'attach_interface', **kw)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 175, in call
      self.transport._send(self.target, msg_ctxt, msg,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send
      return self._driver.send(target, ctxt, message,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
      return self._send(target, ctxt, message, wait_for_reply, timeout,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 209, in _send
      raise failure
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
      result = func(ctxt, **new_args)
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 71, in wrapped
      _emit_versioned_exception_notification(
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/root/rtox/nova/functional-py38/nova/exception_wrapper.py", line 63, in wrapped
      return f(self, context, *args, **kw)
    File "/root/rtox/nova/functional-py38/nova/compute/utils.py", line 1433, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 211, in decorated_function
      compute_utils.add_instance_fault_from_exc(context,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 200, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7735, in attach_interface
      return do_attach_interface(context, instance, network_id, port_id,
    File "/root/rtox/nova/functional-py38/.tox/functional-py38/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
      return f(*args, **kwargs)
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7732, in do_attach_interface
      return self._attach_interface(context, instance, network_id,
    File "/root/rtox/nova/functional-py38/nova/compute/manager.py", line 7787, in _attach_interface
      result = self._allocate_port_resource_for_instance(
    File "/root/rtox/nova/functional-
      py38/nova/compute/manager.py", line 7656, in _allocate_port_resource_for_instance
      alloc_reqs, provider_sums, version = res
  TypeError: cannot unpack non-iterable NoneType object
  2021-08-07 09:59:29,133 INFO [nova.api.openstack.wsgi] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'TypeError'>

  [1]
  https://github.com/openstack/nova/blob/99536b95bcc4c1dd09099abd858233c9352afeed/nova/compute/manager.py#L7646

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1939193/+subscriptions