yahoo-eng-team team mailing list archive

[Jadon Naas <1999540@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

This bug was encountered in IntegrationsQA (also known as SolutionsQA)
testing with Foundation Cloud Engine in the IntegrationsQA lab
environment.

The Tempest tests failed because OpenStack was unable to bind a port to
a virtual machine instance during the test_two_vms_fips Tempest test
from the Neutron Tempest Plugin
(neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin).

Here is the error Tempest reported:
<testcase classname="neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin" id="6bba729b-3fb6-494b-9e1e-82bbd89a1045" name="test_two_vms_fips" time="15.480" timestamp="2022-12-13T05:15:19+0000">
<failure>Traceback (most recent call last): File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 110, in test_two_vms_fips servers, server_ssh_clients, fips = self._create_vms(hyper, avail_zone) File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 75, in _create_vms servers.append(self.create_server( File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/base.py", line 150, in create_server self.wait_for_server_active(server['server'], client=client) File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/base.py", line 527, in wait_for_server_active self.wait_for_server_status( File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/base.py", line 518, in wait_for_server_status waiters.wait_for_server_status(client, server['id'], status, **kwargs) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-6c2956be-9f82-47a0-baae-388878cb5981/repo/tempest/common/waiters.py", line 81, in wait_for_server_status raise exceptions.BuildErrorException(details, server_id=server_id) tempest.exceptions.BuildErrorException: Server 3867e215-f17e-4a29-a5bd-763ad2544a82 failed to build and is in ERROR status Details: Fault: {'code': 500, 'created': '2022-12-13T05:15:33Z', 'message': 'Binding failed for port efbe7ab9-b493-4d0b-a10d-0854f011bc9e, please check neutron logs for more information.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2489, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4329, in spawn\n xml = self._get_guest_xml(context, instance, network_info,\n File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7288, in _get_guest_xml\n network_info_str = str(network_info)\n File "/usr/lib/python3/dist-packages/nova/network/model.py", line 620, in __str__\n return self._sync_wrapper(fn, *args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n File "/usr/lib/python3/dist-packages/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 181, in wait\n return self._exit_event.wait()\n File "/usr/lib/python3/dist-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 221, in main\n result = function(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/utils.py", line 656, in context_wrapper\n return func(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1876, in _allocate_network_async\n raise e\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1854, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1213, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1351, in _update_ports_for_instance\n vif.destroy()\n File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1320, in _update_ports_for_instance\n updated_port = self._update_port(\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 584, in _update_port\n _ensure_no_port_binding_failure(port)\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 293, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\nnova.exception.PortBindingFailed: Binding failed for port efbe7ab9-b493-4d0b-a10d-0854f011bc9e, please check neutron logs for more information.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2312, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2585, in _build_and_run_instance\n raise exception.RescheduledException(\nnova.exception.RescheduledException: Build of instance 3867e215-f17e-4a29-a5bd-763ad2544a82 was re-scheduled: Binding failed for port efbe7ab9-b493-4d0b-a10d-0854f011bc9e, please check neutron logs for more information.\n'}. </failure>
</testcase>

The Tempest test results can be found at https://oil-
jenkins.canonical.com/artifacts/8cf589a3-8f0f-452b-b893-7f99d9d3a6a6/generated/generated/openstack/tempest_result.xml.

I could not find any errors in the Neutron logs for this during this
time. The full artifacts, including logs, for this test run can be
accessed at:

https://oil-
jenkins.canonical.com/artifacts/8cf589a3-8f0f-452b-b893-7f99d9d3a6a6/index.html

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: cdo-qa cdo-tempest foundations-engine

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1999540

Title:
  Tempest test test_two_vms_fips failed due to port binding on instance
  failing

Status in neutron:
  New

Bug description:
  This bug was encountered in IntegrationsQA (also known as SolutionsQA)
  testing with Foundation Cloud Engine in the IntegrationsQA lab
  environment.

  The Tempest tests failed because OpenStack was unable to bind a port
  to a virtual machine instance during the test_two_vms_fips Tempest
  test from the Neutron Tempest Plugin
  (neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin).

  Here is the error Tempest reported:
  <testcase classname="neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin" id="6bba729b-3fb6-494b-9e1e-82bbd89a1045" name="test_two_vms_fips" time="15.480" timestamp="2022-12-13T05:15:19+0000">
  <failure>Traceback (most recent call last): File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 110, in test_two_vms_fips servers, server_ssh_clients, fips = self._create_vms(hyper, avail_zone) File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 75, in _create_vms servers.append(self.create_server( File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/base.py", line 150, in create_server self.wait_for_server_active(server['server'], client=client) File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/base.py", line 527, in wait_for_server_active self.wait_for_server_status( File "/snap/fcbtest/35/lib/python3.10/site-packages/neutron_tempest_plugin/scenario/base.py", line 518, in wait_for_server_status waiters.wait_for_server_status(client, server['id'], status, **kwargs) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-6c2956be-9f82-47a0-baae-388878cb5981/repo/tempest/common/waiters.py", line 81, in wait_for_server_status raise exceptions.BuildErrorException(details, server_id=server_id) tempest.exceptions.BuildErrorException: Server 3867e215-f17e-4a29-a5bd-763ad2544a82 failed to build and is in ERROR status Details: Fault: {'code': 500, 'created': '2022-12-13T05:15:33Z', 'message': 'Binding failed for port efbe7ab9-b493-4d0b-a10d-0854f011bc9e, please check neutron logs for more information.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2489, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4329, in spawn\n xml = self._get_guest_xml(context, instance, network_info,\n File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7288, in _get_guest_xml\n network_info_str = str(network_info)\n File "/usr/lib/python3/dist-packages/nova/network/model.py", line 620, in __str__\n return self._sync_wrapper(fn, *args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n File "/usr/lib/python3/dist-packages/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 181, in wait\n return self._exit_event.wait()\n File "/usr/lib/python3/dist-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 221, in main\n result = function(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/utils.py", line 656, in context_wrapper\n return func(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1876, in _allocate_network_async\n raise e\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1854, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1213, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1351, in _update_ports_for_instance\n vif.destroy()\n File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1320, in _update_ports_for_instance\n updated_port = self._update_port(\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 584, in _update_port\n _ensure_no_port_binding_failure(port)\n File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 293, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\nnova.exception.PortBindingFailed: Binding failed for port efbe7ab9-b493-4d0b-a10d-0854f011bc9e, please check neutron logs for more information.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2312, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2585, in _build_and_run_instance\n raise exception.RescheduledException(\nnova.exception.RescheduledException: Build of instance 3867e215-f17e-4a29-a5bd-763ad2544a82 was re-scheduled: Binding failed for port efbe7ab9-b493-4d0b-a10d-0854f011bc9e, please check neutron logs for more information.\n'}. </failure>
  </testcase>

  The Tempest test results can be found at https://oil-
  jenkins.canonical.com/artifacts/8cf589a3-8f0f-452b-b893-7f99d9d3a6a6/generated/generated/openstack/tempest_result.xml.

  I could not find any errors in the Neutron logs for this during this
  time. The full artifacts, including logs, for this test run can be
  accessed at:

  https://oil-
  jenkins.canonical.com/artifacts/8cf589a3-8f0f-452b-b893-7f99d9d3a6a6/index.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1999540/+subscriptions