yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1272600] Re: object of type 'NoneType' has no len() from neutronclient in get_instance_nwinfo if neutron credentials are missing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 17 Sep 2014 21:18:51 -0000
Reply-to: Bug 1272600 <1272600@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1272600

Title:
  object of type 'NoneType' has no len() from neutronclient in
  get_instance_nwinfo if neutron credentials are missing

Status in OpenStack Compute (Nova):
  Invalid
Status in tripleo - openstack on openstack:
  Fix Released

Bug description:
  To reproduce:

  1. boot an instance
  2. rebuild instance with a new image id
  3. immediately restart nova-compute

  -- instance is now stuck in REBUILD/rebuild_spawning state.

  4. nova delete instance

  -- instanec now says it is ACTIVE, but it cannot be deleted.

  This traceback appears in logs:

  2014-01-25 01:35:22,096.096 27134 ERROR nova.openstack.common.rpc.amqp [req-9f5bbb1e-4cab-4f60-9f28-57366b519533 2de96ce5da994575a08447c93bcafd53 4956c533154c476799c688eda7ed65ab] Exception during message handling
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 90, in wrapped
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 73, in wrapped
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 244, in decorated_function
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     pass
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 230, in decorated_function
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 295, in decorated_function
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 272, in decorated_function
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 259, in decorated_function
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2024, in terminate_instance
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in do_terminate_instance
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py", line 105, in inner
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1994, in _delete_instance
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     user_id=user_id)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1964, in _delete_instance
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, db_inst, bdms)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1874, in _shutdown_instance
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 891, in _get_instance_nw_info
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     instance)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/api.py", line 50, in wrapper
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     res = f(self, context, *args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 439, in get_instance_nw_info
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 446, in _get_instance_nw_info
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 998, in _build_network_info_model
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 108, in with_params
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in list_ports
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     **_params)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1197, in list
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1210, in _pagination
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1183, in get
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1168, in retry_request
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1097, in do_request
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     self.httpclient.authenticate_and_fetch_endpoint_url()
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/client.py", line 172, in authenticate_and_fetch_endpoint_url
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     self.authenticate()
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/client.py", line 237, in authenticate
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp     raise exceptions.Unauthorized(message=body)
  2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp Unauthorized: {"error": {"message": "object of type 'NoneType' has no len()", "code": 400, "title": "Bad Request"}}

  
  Note that this was seen with a recent pull of master as of today, with the nova baremetal driver.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1272600/+subscriptions
References

[Bug 1272600] [NEW] Deleting an instance that is in the REBUILD/rebuild_spawning state results in an undeletable instance
From: Clint Byrum, 2014-01-25