yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1770175] Re: When list the servers, report : 'TypeError: 'object' object is not iterable'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Surya Seetharaman <1770175@xxxxxxxxxxxxxxxxxx>
Date: Wed, 09 May 2018 15:19:19 -0000
Reply-to: Bug 1770175 <1770175@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This means that the command could not connect to the nova cell DB. As
you can see from the logs there was a time out waiting for connection to
the cells due to which did_not_respond_sentinels were attached to the
cell results
(https://github.com/openstack/nova/blob/master/nova/context.py#L464) .
That is why it fails. Please check your cell DB connection and see if
you are able to connect to it.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1770175

Title:
  When list the servers, report : 'TypeError: 'object' object is not
  iterable'

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========
  When do the query of the servers(by CLI or Dashboard), sometimes report errors like the follow: 

  [root@ctrl01 ~]# openstack server list
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409)

  Steps to reproduce
  ==================
  (See Description)

  Expected result
  ===============
  Get the correct result(the vm server list)

  
  Actual result
  =============

  
  Environment
  ===========
  [root@ctrl01 ~]# rpm -qa | grep nova
  python2-novaclient-9.1.1-1.el7.noarch
  openstack-nova-novncproxy-17.0.2-1.el7.noarch
  openstack-nova-conductor-17.0.2-1.el7.noarch
  openstack-nova-common-17.0.2-1.el7.noarch
  openstack-nova-console-17.0.2-1.el7.noarch
  openstack-nova-placement-api-17.0.2-1.el7.noarch
  openstack-nova-scheduler-17.0.2-1.el7.noarch
  python-nova-17.0.2-1.el7.noarch
  openstack-nova-api-17.0.2-1.el7.noarch

  
  Logs & Configs
  ==============
  2018-05-09 22:08:47.085 10667 DEBUG nova.osapi_compute.wsgi.server [req-483c7fc7-8094-4a73-83f4-f8e0d39deaef - - - - -] (10667) accepted ('172.16.2.240', 40984) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
  2018-05-09 22:08:47.173 10668 INFO nova.metadata.wsgi.server [req-483c7fc7-8094-4a73-83f4-f8e0d39deaef - - - - -] (10668) wsgi starting up on http://0.0.0.0:8775
  2018-05-09 22:08:47.235 10669 INFO nova.metadata.wsgi.server [req-483c7fc7-8094-4a73-83f4-f8e0d39deaef - - - - -] (10669) wsgi starting up on http://0.0.0.0:8775
  2018-05-09 22:09:48.275 10667 DEBUG nova.api.openstack.wsgi [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x8ad4290>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:607
  2018-05-09 22:09:48.279 10667 DEBUG nova.compute.api [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Searching by: {'deleted': False, 'project_id': u'ea02f765bd584e8c813459cb42063f76'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2347
  2018-05-09 22:10:48.301 10667 DEBUG oslo_db.sqlalchemy.engines [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:290
  2018-05-09 22:10:48.321 10667 DEBUG nova.context [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Found 2 cells: 00000000-0000-0000-0000-000000000000(cell0),d34a74ee-bc98-4f36-882e-27b2acb2e510(cell1) load_cells /usr/lib/python2.7/site-packages/nova/context.py:474
  2018-05-09 22:10:48.362 10667 DEBUG oslo_concurrency.lockutils [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2018-05-09 22:10:48.365 10667 DEBUG oslo_concurrency.lockutils [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2018-05-09 22:10:48.369 10667 DEBUG oslo_concurrency.lockutils [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Lock "d34a74ee-bc98-4f36-882e-27b2acb2e510" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2018-05-09 22:10:48.374 10667 DEBUG oslo_concurrency.lockutils [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Lock "d34a74ee-bc98-4f36-882e-27b2acb2e510" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.005s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2018-05-09 22:11:48.326 10667 WARNING nova.context [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Timed out waiting for response from cell 00000000-0000-0000-0000-000000000000: CellTimeout: Timeout waiting for response from cell
  2018-05-09 22:11:48.327 10667 WARNING nova.context [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Timed out waiting for response from cell d34a74ee-bc98-4f36-882e-27b2acb2e510: CellTimeout: Timeout waiting for response from cell
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Unexpected exception in API method: TypeError: 'object' object is not iterable
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 165, in detail
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 308, in _get_servers
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     sort_keys=sort_keys, sort_dirs=sort_dirs)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2462, in get_all
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     context, filters, limit, marker, fields, sort_keys, sort_dirs)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/instance_list.py", line 107, in get_instance_objects_sorted
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     expected_attrs)
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1199, in _make_instance_list
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     for db_inst in db_inst_list:
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/multi_cell_list.py", line 259, in get_records_sorted
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     for i in heapq.merge(*results.values()):
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/heapq.py", line 372, in merge
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi     for itnum, it in enumerate(map(iter, iterables)):
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi TypeError: 'object' object is not iterable
  2018-05-09 22:11:48.328 10667 ERROR nova.api.openstack.wsgi
  2018-05-09 22:11:48.421 10667 INFO nova.api.openstack.wsgi [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'>
  2018-05-09 22:11:48.422 10667 DEBUG nova.api.openstack.wsgi [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1064
  2018-05-09 22:11:48.424 10667 INFO nova.osapi_compute.wsgi.server [req-f7e1bfb0-926c-43ca-8fbd-66bcfe15e409 96ce2e706922453aa600078b866f8a68 ea02f765bd584e8c813459cb42063f76 - default default] 172.16.2.240 "GET /v2.1/servers/detail HTTP/1.1" status: 500 len: 626 time: 181.3375800

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1770175/+subscriptions
References

[Bug 1770175] [NEW] When list the servers, report : 'TypeError: 'object' object is not iterable'
From: litton peng, 2018-05-09