yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1705141] [NEW] select_destinations results in "TypeError: 'NoneType' object is not iterable" randomly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 18 Jul 2017 23:46:19 -0000
Reply-to: Bug 1705141 <1705141@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/99/471899/25/check/gate-grenade-dsvm-neutron-
ubuntu-
xenial/3f9a9e3/logs/new/screen-n-cond.txt.gz?level=TRACE#_2017-07-18_22_04_39_026

2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager Traceback (most recent call last):
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/conductor/manager.py", line 920, in schedule_and_build_instances
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     instance_uuids)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/conductor/manager.py", line 624, in _schedule_instances
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     request_spec, instance_uuids)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/utils.py", line 464, in wrapped
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return func(*args, **kwargs)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 52, in select_destinations
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     instance_uuids)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/client/query.py", line 33, in select_destinations
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     instance_uuids)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/rpcapi.py", line 136, in select_destinations
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     retry=self.retry)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     timeout=timeout, retry=retry)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     retry=retry)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     raise result
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager TypeError: 'NoneType' object is not iterable
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager Traceback (most recent call last):
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 153, in _process_incoming
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     res = self.dispatcher.dispatch(message)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return self._do_dispatch(endpoint, method, ctxt, args)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     result = func(ctxt, **new_args)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 225, in inner
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return func(*args, **kwargs)
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/manager.py", line 114, in select_destinations
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     alloc_reqs, p_sums = res
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager TypeError: 'NoneType' object is not iterable
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20select_destinations%5C%22%20AND%20message%3A%5C%22alloc_reqs%2C%20p_sums%20%3D%20res%5C%22%20AND%20message%3A%5C%22TypeError%3A%20'NoneType'%20object%20is%20not%20iterable%5C%22%20AND%20tags%3A%5C%22screen-n-cond.txt%5C%22&from=7d

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: placement scheduler

** Summary changed:

- select_destinations reults in "TypeError: 'NoneType' object is not iterable" randomly
+ select_destinations results in "TypeError: 'NoneType' object is not iterable" randomly

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1705141

Title:
  select_destinations results in "TypeError: 'NoneType' object is not
  iterable" randomly

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  http://logs.openstack.org/99/471899/25/check/gate-grenade-dsvm-
  neutron-ubuntu-
  xenial/3f9a9e3/logs/new/screen-n-cond.txt.gz?level=TRACE#_2017-07-18_22_04_39_026

  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager Traceback (most recent call last):
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/conductor/manager.py", line 920, in schedule_and_build_instances
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     instance_uuids)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/conductor/manager.py", line 624, in _schedule_instances
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     request_spec, instance_uuids)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/utils.py", line 464, in wrapped
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return func(*args, **kwargs)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 52, in select_destinations
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     instance_uuids)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/client/query.py", line 33, in select_destinations
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     instance_uuids)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/rpcapi.py", line 136, in select_destinations
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     retry=self.retry)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     timeout=timeout, retry=retry)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     retry=retry)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     raise result
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager TypeError: 'NoneType' object is not iterable
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager Traceback (most recent call last):
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 153, in _process_incoming
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     res = self.dispatcher.dispatch(message)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     result = func(ctxt, **new_args)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 225, in inner
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     return func(*args, **kwargs)
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager   File "/opt/stack/new/nova/nova/scheduler/manager.py", line 114, in select_destinations
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager     alloc_reqs, p_sums = res
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager TypeError: 'NoneType' object is not iterable
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 
  2017-07-18 22:04:39.026 32126 ERROR nova.conductor.manager 

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20select_destinations%5C%22%20AND%20message%3A%5C%22alloc_reqs%2C%20p_sums%20%3D%20res%5C%22%20AND%20message%3A%5C%22TypeError%3A%20'NoneType'%20object%20is%20not%20iterable%5C%22%20AND%20tags%3A%5C%22screen-n-cond.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1705141/+subscriptions
Follow ups

[Bug 1705141] Re: select_destinations results in "TypeError: 'NoneType' object is not iterable" if get_allocation_candidates fails to connect to Placement
From: OpenStack Infra, 2017-07-20