yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #85326
[Bug 1907522] Re: test_*_with_qos_min_bw_allocation fails in the nova-multi-cell job with: nova.exception.MigrationPreCheckError: Migration pre-check error: Failed to create port bindings for host <host>
ussuri backport landed.
The cross cell resize feature was finished in ussuri so no need to
backport his fix to train or later.
** Changed in: nova/ussuri
Status: In Progress => Fix Released
** Changed in: nova/ussuri
Status: Fix Released => Fix Committed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1907522
Title:
test_*_with_qos_min_bw_allocation fails in the nova-multi-cell job
with: nova.exception.MigrationPreCheckError: Migration pre-check
error: Failed to create port bindings for host <host>
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) ussuri series:
Fix Committed
Status in OpenStack Compute (nova) victoria series:
Fix Released
Bug description:
Seen in the gate on the master branch, the
test_migrate_with_qos_min_bw_allocation and
test_resize_with_qos_min_bw_allocation tests that were recently added
to tempest [1] are failing in the nova-multi-cell job with:
"ERROR oslo_messaging.rpc.server
nova.exception.MigrationPreCheckError: Migration pre-check error:
Failed to create port bindings for host ubuntu-focal-ovh-
bhs1-0022118086"
with the following traceback in screen-n-super-cond.txt [2]:
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/cross_cell_migrate.py", line 266, in _create_port_bindings
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self._bindings_by_port_id = self.network_api.bind_ports_to_host(
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/network/neutron.py", line 1352, in bind_ports_to_host
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server raise exception.PortBindingFailed(port_id=port_id)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server nova.exception.PortBindingFailed: Binding failed for port d37d5be3-df95-4842-bddf-2058d8e452fc, please check neutron logs for more information.
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 354, in _cold_migrate
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server task.execute()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 26, in wrap
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self.rollback(ex)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self.force_reraise()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server raise value
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 23, in wrap
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server return original(self)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 40, in execute
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server return self._execute()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/migrate.py", line 295, in _execute
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server task.execute()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 26, in wrap
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self.rollback(ex)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self.force_reraise()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server raise value
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 23, in wrap
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server return original(self)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 40, in execute
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server return self._execute()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/cross_cell_migrate.py", line 838, in _execute
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server target_cell_migration = self._prep_resize_at_dest(
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/cross_cell_migrate.py", line 727, in _prep_resize_at_dest
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server target_cell_migration_context = verify_task.execute()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 26, in wrap
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self.rollback(ex)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self.force_reraise()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server raise value
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 23, in wrap
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server return original(self)
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 40, in execute
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server return self._execute()
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/cross_cell_migrate.py", line 327, in _execute
Dec 08 19:11:12.292344 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server self._create_port_bindings()
Dec 08 19:11:12.297758 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/cross_cell_migrate.py", line 269, in _create_port_bindings
Dec 08 19:11:12.297758 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server raise exception.MigrationPreCheckError(reason=_(
Dec 08 19:11:12.297758 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Failed to create port bindings for host ubuntu-focal-ovh-bhs1-0022118086
Dec 08 19:11:12.297758 ubuntu-focal-ovh-bhs1-0022118085 nova-conductor[93226]: ERROR oslo_messaging.rpc.server
and corresponding neutron failure:
"ERROR neutron.plugins.ml2.managers
neutron_lib.exceptions.placement.UnknownResourceProvider: No such
resource provider known by Neutron"
with traceback in screen-q-svc.txt [3] :
Dec 08 19:11:11.521492 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers [req-1d6e98ea-4f98-4a73-a4f7-470561c7e03b req-0389a6b9-c4f4-448e-baa9-8f31bf231926 service neutron] Failed to bind port d37d5be3-df95-4842-bddf-2058d8e452fc on host ubuntu-focal-ovh-bhs1-0022118086 allocated on resource provider c7942897-51a0-56e7-86e1-4727af85911d, because no mechanism driver reports being responsible
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers [req-1d6e98ea-4f98-4a73-a4f7-470561c7e03b req-0389a6b9-c4f4-448e-baa9-8f31bf231926 service neutron] Mechanism driver openvswitch failed in bind_port: neutron_lib.exceptions.placement.UnknownResourceProvider: No such resource provider known by Neutron: c7942897-51a0-56e7-86e1-4727af85911d
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 923, in _bind_port_level
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers drivers=[self._infer_driver_from_allocation(
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 973, in _infer_driver_from_allocation
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers raise place_exc.UnknownResourceProvider(
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers neutron_lib.exceptions.placement.UnknownResourceProvider: No such resource provider known by Neutron: c7942897-51a0-56e7-86e1-4727af85911d
Dec 08 19:11:11.526334 ubuntu-focal-ovh-bhs1-0022118085 neutron-server[85124]: ERROR neutron.plugins.ml2.managers
[1] https://review.opendev.org/c/openstack/tempest/+/694539
[2] https://zuul.opendev.org/t/openstack/build/d7c15477b23e4a28b400c30277b3b3e3/log/controller/logs/screen-n-super-cond.txt?severity=4#4180
[3] https://zuul.opendev.org/t/openstack/build/d7c15477b23e4a28b400c30277b3b3e3/log/controller/logs/screen-q-svc.txt#72367
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1907522/+subscriptions
References
