yahoo-eng-team team mailing list archive

[OpenStack Infra <2011377@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/877535
Committed: https://opendev.org/openstack/neutron/commit/04d3f889efeef74e8eb8d8bf330f1594913b161a
Submitter: "Zuul (22348)"
Branch:    master

commit 04d3f889efeef74e8eb8d8bf330f1594913b161a
Author: Miro Tomaska <mtomaska@xxxxxxxxxx>
Date:   Wed Mar 15 12:26:12 2023 -0500

    Fix metadata agent intermittent test failures
    
    Metadata agent has been experiencing intermittent failures
    mostly because of test conccurency and how the metadata agent
    code assumes its the only process running on the system and
    operating on the ovnmeta-* namespaces. See comment#1 the
    linked bug for more details. Although I dont like forcing
    --concurrency 1 for this test class, I think that is going
    to be the best solution and any new tests that will be added
    in the future.
    
    Closes-Bug: #2011377
    Change-Id: Ie7f3b496de6b23be5739fbeba10f53602e8b300d


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2011377

Title:
  test_agent_resync_on_non_existing_bridge failing intermittently sp

Status in neutron:
  Fix Released

Bug description:
  Test
  neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge
  is failing intermitelly. Ex [0]

  I just reintroduced this test [1] into the code. The failure is does
  not happen all the time but I can be reproduce it locally with
  --until-failure with multiple concurrency(big hint) and running the
  whole TestMetadaAgent class of tests (another hint). Like this

  `tox -e dsvm-functional --
  neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent
  --until-failure --concurrency 0`

  When the failure happens following exception is found in the logs

  2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
  2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
  2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
  2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     match.run(event, row, updates)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     self.agent.provision_datapath(row.datapath)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     ip2.addr.add_multiple(ipv4_cidrs_to_add)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     privileged.add_ip_addresses(
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     return self.channel.remote_call(name, args, kwargs,
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     raise exc_type(*result[2])
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event

  The thing is this line of code should actually be creating the new
  namespace[2] so not sure why its complaining that namespace was not
  found. I am suspecting there is some race condition or more likely
  some test interferance due to test runner concurrency.

  [0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17
  [1] https://review.opendev.org/c/openstack/neutron/+/875586
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2011377/+subscriptions