yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #31243
[Bug 1441745] [NEW] Lots of gate failures with "not enough hosts available"
Public bug reported:
Thousands of matches in the last two days:
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gdmFsaWQgaG9zdCB3YXMgZm91bmQuIFRoZXJlIGFyZSBub3QgZW5vdWdoIGhvc3RzIGF2YWlsYWJsZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI4NTA4MTY3MTcwfQ==
The following is from this log file:
http://logs.openstack.org/42/163842/8/check/check-tempest-dsvm-neutron-
full/1f66320/logs/screen-n-cond.txt.gz
For the few I looked at, there is an error in the n-cond log:
2015-04-08 07:20:15.207 WARNING nova.scheduler.utils [req-a21c9875-efe1-407d-b08b-2b05b35b4642 AggregatesAdminTestJSON-325246720 AggregatesAdminTestJSON-279542170] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
return func(*args, **kwargs)
File "/opt/stack/new/nova/nova/scheduler/manager.py", line 86, in select_destinations
filter_properties)
File "/opt/stack/new/nova/nova/scheduler/filter_scheduler.py", line 80, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts
available.
--------------------------
That makes it sound like the problem is that the deployed devstack does
not have enough capacity. But right before that I see:
2015-04-08 07:20:15.014 ERROR nova.conductor.manager [req-a21c9875-efe1-407d-b08b-2b05b35b4642 AggregatesAdminTestJSON-325246720 AggregatesAdminTestJSON-279542170] Instance update attempted for 'availability_zone' on 745aafcf-686d-4cf0-91c7-701e282f6d06
2015-04-08 07:20:15.149 ERROR nova.scheduler.utils [req-a21c9875-efe1-407d-b08b-2b05b35b4642 AggregatesAdminTestJSON-325246720 AggregatesAdminTestJSON-279542170] [instance: 745aafcf-686d-4cf0-91c7-701e282f6d06] Error from last host: devstack-trusty-rax-dfw-1769605.slave.openstack.org (node devstack-trusty-rax-dfw-1769605.slave.openstack.org): [u'Traceback (most recent call last):\n', u' File "/opt/stack/new/nova/nova/compute/manager.py", line 2193, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/new/nova/nova/compute/manager.py", line 2336, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 745aafcf-686d-4cf0-91c7-701e282f6d06 was re-scheduled: u\'u"unexpected update keyword \\\'availability_zone\\\'"\\nTraceback (most recent call last):\\n\\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner\\n return func(*args, **kwargs)\\n\\n File "/opt/stack/new/nova/nova/conductor/manager.py", line 125, in instance_update\\n raise KeyError("unexpected update keyword \\\'%s\\\'" % key)\\n\\nKeyError: u"unexpected update keyword \\\'availability_zone\\\'"\\n\'\n']
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1441745
Title:
Lots of gate failures with "not enough hosts available"
Status in OpenStack Compute (Nova):
New
Bug description:
Thousands of matches in the last two days:
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gdmFsaWQgaG9zdCB3YXMgZm91bmQuIFRoZXJlIGFyZSBub3QgZW5vdWdoIGhvc3RzIGF2YWlsYWJsZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI4NTA4MTY3MTcwfQ==
The following is from this log file:
http://logs.openstack.org/42/163842/8/check/check-tempest-dsvm-
neutron-full/1f66320/logs/screen-n-cond.txt.gz
For the few I looked at, there is an error in the n-cond log:
2015-04-08 07:20:15.207 WARNING nova.scheduler.utils [req-a21c9875-efe1-407d-b08b-2b05b35b4642 AggregatesAdminTestJSON-325246720 AggregatesAdminTestJSON-279542170] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
return func(*args, **kwargs)
File "/opt/stack/new/nova/nova/scheduler/manager.py", line 86, in select_destinations
filter_properties)
File "/opt/stack/new/nova/nova/scheduler/filter_scheduler.py", line 80, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts
available.
--------------------------
That makes it sound like the problem is that the deployed devstack
does not have enough capacity. But right before that I see:
2015-04-08 07:20:15.014 ERROR nova.conductor.manager [req-a21c9875-efe1-407d-b08b-2b05b35b4642 AggregatesAdminTestJSON-325246720 AggregatesAdminTestJSON-279542170] Instance update attempted for 'availability_zone' on 745aafcf-686d-4cf0-91c7-701e282f6d06
2015-04-08 07:20:15.149 ERROR nova.scheduler.utils [req-a21c9875-efe1-407d-b08b-2b05b35b4642 AggregatesAdminTestJSON-325246720 AggregatesAdminTestJSON-279542170] [instance: 745aafcf-686d-4cf0-91c7-701e282f6d06] Error from last host: devstack-trusty-rax-dfw-1769605.slave.openstack.org (node devstack-trusty-rax-dfw-1769605.slave.openstack.org): [u'Traceback (most recent call last):\n', u' File "/opt/stack/new/nova/nova/compute/manager.py", line 2193, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/new/nova/nova/compute/manager.py", line 2336, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 745aafcf-686d-4cf0-91c7-701e282f6d06 was re-scheduled: u\'u"unexpected update keyword \\\'availability_zone\\\'"\\nTraceback (most recent call last):\\n\\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner\\n return func(*args, **kwargs)\\n\\n File "/opt/stack/new/nova/nova/conductor/manager.py", line 125, in instance_update\\n raise KeyError("unexpected update keyword \\\'%s\\\'" % key)\\n\\nKeyError: u"unexpected update keyword \\\'availability_zone\\\'"\\n\'\n']
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1441745/+subscriptions
Follow ups
References