yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1756845] [NEW] Queens Devstack instance creation failure "Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sunil G <sunil.07g@xxxxxxxxx>
Date: Mon, 19 Mar 2018 11:10:30 -0000
Reply-to: Bug 1756845 <1756845@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Devstack Queens cirros instance launch failed with the error "MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a."

Creation of Network, router.. all went well.
Launch instance task failed.


Steps to reproduce
==================
Install Devstack from stable/queens branch and launch instance

Environment
===========
Ubuntu Server 16.04.4 LTS
Devstack stable/queens

stack@stack:/opt/devstack$ git log -1
commit bb0c101c0c7352b0fadf0e13d4c47f59437db454
Merge: 661b186 abe9fd8
Author: Zuul <zuul@xxxxxxxxxxxxxxxxxxxx>
Date:   Tue Mar 13 02:28:44 2018 +0000

    Merge "Cap max microversions for queens" into stable/queens


Logs & Configs
==============

nova-compute.log
-----------------

Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR os_vif [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m#033[01;31mFailed to plug vif VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b')#033[00m: ProcessExecutionError: Unexpected error while running command.
Mar 19 06:29:32 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0
Mar 19 06:29:32 stack nova-compute[6834]: Exit code: 1
Mar 19 06:29:32 stack nova-compute[6834]: Stdout: u''
Mar 19 06:29:32 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n'
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mTraceback (most recent call last):
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/os_vif/__init__.py", line 77, in plug
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    plugin.plug(vif, instance_info)
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 209, in plug
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    self._plug_bridge(vif, instance_info)
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 161, in _plug_bridge
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    linux_net.ensure_bridge(vif.bridge_name)
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    return self.channel.remote_call(name, args, kwargs)
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 202, in remote_call
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    raise exc_type(*result[2])
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mProcessExecutionError: Unexpected error while running command.
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mCommand: brctl setageing qbr41533537-1b 0
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mExit code: 1
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStdout: u''
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStderr: u'set ageing time failed: Numerical result out of range\n'
Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m
Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR nova.virt.libvirt.driver [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to start libvirt guest#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.


Mar 19 06:29:33 stack nova-compute[6834]: #033[01;31mERROR nova.compute.manager [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mInstance failed to spawn#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.
Mar 19 06:29:33 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0
Mar 19 06:29:33 stack nova-compute[6834]: Exit code: 1
Mar 19 06:29:33 stack nova-compute[6834]: Stdout: u''
Mar 19 06:29:33 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n'
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last):
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/compute/manager.py", line 2238, in _build_resources
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    yield resources
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/compute/manager.py", line 2022, in _build_and_run_instance
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    block_device_info=block_device_info)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3047, in spawn
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    destroy_disks_on_failure=True)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5536, in _create_domain_and_network
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    destroy_disks_on_failure)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self.force_reraise()
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    six.reraise(self.type_, self.value, self.tb)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5495, in _create_domain_and_network
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self.plug_vifs(instance, network_info)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 822, in plug_vifs
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self.vif_driver.plug(instance, vif)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 769, in plug
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self._plug_os_vif(instance, vif_obj)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 749, in _plug_os_vif
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    raise exception.InternalError(msg)
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mInternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mCommand: brctl setageing qbr41533537-1b 0
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mExit code: 1
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStdout: u''
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStderr: u'set ageing time failed: Numerical result out of range\n'
Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m


nova-conductor.log
-------------------

Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mError from last host: stack (node stack): [u'Traceback (most recent call last):\n', u'  File "/opt/stack/nova/nova/compute/manager.py", line 1840, in _do_build_and_run_instance\n    filter_properties, request_spec)\n', u'  File "/opt/stack/nova/nova/compute/manager.py", line 2108, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.\nCommand: brctl setageing qbr41533537-1b 0\nExit code: 1\nStdout: u''\nStderr: u'set ageing time failed: Numerical result out of range\\n'\n"]#033[00m
Mar 19 06:29:35 stack nova-conductor[5910]: #033[00;32mDEBUG nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mRescheduling: True#033[00m #033[00;33m{{(pid=6969) build_instances /opt/stack/nova/nova/conductor/manager.py:570}}#033[00m
Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m#033[01;33mFailed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.
Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;33mSetting instance to ERROR state.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.


Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to deallocate networks#033[00m: Unauthorized: Unknown auth type: None
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last):
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/conductor/manager.py", line 366, in _cleanup_allocated_networks
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    context, instance, requested_networks=requested_networks)
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1270, in deallocate_for_instance
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self._unbind_ports(context, ports_to_skip, neutron)
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 527, in _unbind_ports
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    port_client = get_client(context, admin=True)
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 153, in get_client
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    _ADMIN_AUTH = _load_auth_plugin(CONF)
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 76, in _load_auth_plugin
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    raise neutron_client_exc.Unauthorized(message=err_msg)
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mUnauthorized: Unknown auth type: None
Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m
Mar 19 06:29:36 stack devstack@n-api.service[24084]: #033[00;36mINFO nova.api.openstack.requestlog [#033[01;36mNone req-2685a1bf-9098-49f7-8f86-00a01bab5a05 #033[00;36mdemo admin#033[00;36m] #033[01;35m#033[00;36m10.39.18.67 "GET /compute/v2.1" status: 302 len: None microversion: - time: 0.004733#033[00m

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1756845

Title:
  Queens Devstack instance creation failure "Error: Exceeded maximum
  number of retries. Exhausted all hosts available for retrying build
  failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a"

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Devstack Queens cirros instance launch failed with the error "MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a."

  Creation of Network, router.. all went well.
  Launch instance task failed.

  
  Steps to reproduce
  ==================
  Install Devstack from stable/queens branch and launch instance

  Environment
  ===========
  Ubuntu Server 16.04.4 LTS
  Devstack stable/queens

  stack@stack:/opt/devstack$ git log -1
  commit bb0c101c0c7352b0fadf0e13d4c47f59437db454
  Merge: 661b186 abe9fd8
  Author: Zuul <zuul@xxxxxxxxxxxxxxxxxxxx>
  Date:   Tue Mar 13 02:28:44 2018 +0000

      Merge "Cap max microversions for queens" into stable/queens

  
  Logs & Configs
  ==============

  nova-compute.log
  -----------------

  Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR os_vif [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m#033[01;31mFailed to plug vif VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b')#033[00m: ProcessExecutionError: Unexpected error while running command.
  Mar 19 06:29:32 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0
  Mar 19 06:29:32 stack nova-compute[6834]: Exit code: 1
  Mar 19 06:29:32 stack nova-compute[6834]: Stdout: u''
  Mar 19 06:29:32 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n'
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mTraceback (most recent call last):
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/os_vif/__init__.py", line 77, in plug
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    plugin.plug(vif, instance_info)
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 209, in plug
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    self._plug_bridge(vif, instance_info)
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 161, in _plug_bridge
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    linux_net.ensure_bridge(vif.bridge_name)
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    return self.channel.remote_call(name, args, kwargs)
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 202, in remote_call
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m    raise exc_type(*result[2])
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mProcessExecutionError: Unexpected error while running command.
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mCommand: brctl setageing qbr41533537-1b 0
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mExit code: 1
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStdout: u''
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStderr: u'set ageing time failed: Numerical result out of range\n'
  Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m
  Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR nova.virt.libvirt.driver [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to start libvirt guest#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.


  Mar 19 06:29:33 stack nova-compute[6834]: #033[01;31mERROR nova.compute.manager [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mInstance failed to spawn#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.
  Mar 19 06:29:33 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0
  Mar 19 06:29:33 stack nova-compute[6834]: Exit code: 1
  Mar 19 06:29:33 stack nova-compute[6834]: Stdout: u''
  Mar 19 06:29:33 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n'
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last):
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/compute/manager.py", line 2238, in _build_resources
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    yield resources
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/compute/manager.py", line 2022, in _build_and_run_instance
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    block_device_info=block_device_info)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3047, in spawn
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    destroy_disks_on_failure=True)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5536, in _create_domain_and_network
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    destroy_disks_on_failure)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self.force_reraise()
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    six.reraise(self.type_, self.value, self.tb)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5495, in _create_domain_and_network
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self.plug_vifs(instance, network_info)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 822, in plug_vifs
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self.vif_driver.plug(instance, vif)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 769, in plug
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self._plug_os_vif(instance, vif_obj)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 749, in _plug_os_vif
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    raise exception.InternalError(msg)
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mInternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mCommand: brctl setageing qbr41533537-1b 0
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mExit code: 1
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStdout: u''
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStderr: u'set ageing time failed: Numerical result out of range\n'
  Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m

  
  nova-conductor.log
  -------------------

  Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mError from last host: stack (node stack): [u'Traceback (most recent call last):\n', u'  File "/opt/stack/nova/nova/compute/manager.py", line 1840, in _do_build_and_run_instance\n    filter_properties, request_spec)\n', u'  File "/opt/stack/nova/nova/compute/manager.py", line 2108, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.\nCommand: brctl setageing qbr41533537-1b 0\nExit code: 1\nStdout: u''\nStderr: u'set ageing time failed: Numerical result out of range\\n'\n"]#033[00m
  Mar 19 06:29:35 stack nova-conductor[5910]: #033[00;32mDEBUG nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mRescheduling: True#033[00m #033[00;33m{{(pid=6969) build_instances /opt/stack/nova/nova/conductor/manager.py:570}}#033[00m
  Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m#033[01;33mFailed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.
  Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;33mSetting instance to ERROR state.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.



  Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to deallocate networks#033[00m: Unauthorized: Unknown auth type: None
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last):
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/conductor/manager.py", line 366, in _cleanup_allocated_networks
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    context, instance, requested_networks=requested_networks)
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1270, in deallocate_for_instance
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    self._unbind_ports(context, ports_to_skip, neutron)
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 527, in _unbind_ports
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    port_client = get_client(context, admin=True)
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 153, in get_client
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    _ADMIN_AUTH = _load_auth_plugin(CONF)
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 76, in _load_auth_plugin
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m    raise neutron_client_exc.Unauthorized(message=err_msg)
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mUnauthorized: Unknown auth type: None
  Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m
  Mar 19 06:29:36 stack devstack@n-api.service[24084]: #033[00;36mINFO nova.api.openstack.requestlog [#033[01;36mNone req-2685a1bf-9098-49f7-8f86-00a01bab5a05 #033[00;36mdemo admin#033[00;36m] #033[01;35m#033[00;36m10.39.18.67 "GET /compute/v2.1" status: 302 len: None microversion: - time: 0.004733#033[00m

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1756845/+subscriptions
Follow ups

[Bug 1756845] Re: Queens Devstack instance creation failure "Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a"
From: Matt Riedemann, 2018-03-21