yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2011377] [NEW] test_agent_resync_on_non_existing_bridge failing intermittently sp

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Miro Tomaska <2011377@xxxxxxxxxxxxxxxxxx>
Date: Mon, 13 Mar 2023 03:20:43 -0000
Reply-to: Bug 2011377 <2011377@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Test
neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge
is failing intermitelly. Ex [0]

I just reintroduced this test [1] into the code. The failure is does not
happen all the time but I can be reproduce it locally with --until-
failure with multiple concurrency(big hint) and running the whole
TestMetadaAgent class of tests (another hint). Like this

`tox -e dsvm-functional --
neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent
--until-failure --concurrency 0`

When the failure happens following exception is found in the logs

2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     match.run(event, row, updates)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     self.agent.provision_datapath(row.datapath)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     ip2.addr.add_multiple(ipv4_cidrs_to_add)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     privileged.add_ip_addresses(
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     return self.channel.remote_call(name, args, kwargs,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     raise exc_type(*result[2])
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event

The thing is this line of code should actually be creating the new
namespace[2] so not sure why its complaining that namespace was not
found. I am suspecting there is some race condition or more likely some
test interferance due to test runner concurrency.

[0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17
[1] https://review.opendev.org/c/openstack/neutron/+/875586
[2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611

** Affects: neutron
     Importance: Undecided
     Assignee: Miro Tomaska (mtomaska)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Miro Tomaska (mtomaska)

** Description changed:

  Test
  neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge
- is failing intermitelly.
+ is failing intermitelly. Ex [0]
  
- I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this
-  
- `tox -e dsvm-functional -- neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent --until-failure --concurrency 0`
+ I just reintroduced this test [1] into the code. The failure is does not
+ happen all the time but I can be reproduce it locally with --until-
+ failure with multiple concurrency(big hint) and running the whole
+ TestMetadaAgent class of tests (another hint). Like this
+ 
+ `tox -e dsvm-functional --
+ neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent
+ --until-failure --concurrency 0`
  
  When the failure happens following exception is found in the logs
  
  2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
  2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
  2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
  2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     match.run(event, row, updates)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     self.agent.provision_datapath(row.datapath)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     ip2.addr.add_multiple(ipv4_cidrs_to_add)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     privileged.add_ip_addresses(
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     return self.channel.remote_call(name, args, kwargs,
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     raise exc_type(*result[2])
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
- 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event 
+ 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
  
+ The thing is this line of code should actually be creating the new
+ namespace[2] so not sure why its complaining that namespace was not
+ found. I am suspecting there is some race condition or more likely some
+ test interferance due to test runner concurrency.
  
- The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency.
- 
- 
+ [0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17
  [1] https://review.opendev.org/c/openstack/neutron/+/875586
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2011377

Title:
  test_agent_resync_on_non_existing_bridge failing intermittently sp

Status in neutron:
  New

Bug description:
  Test
  neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge
  is failing intermitelly. Ex [0]

  I just reintroduced this test [1] into the code. The failure is does
  not happen all the time but I can be reproduce it locally with
  --until-failure with multiple concurrency(big hint) and running the
  whole TestMetadaAgent class of tests (another hint). Like this

  `tox -e dsvm-functional --
  neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent
  --until-failure --concurrency 0`

  When the failure happens following exception is found in the logs

  2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
  2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
  2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
  2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     match.run(event, row, updates)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     self.agent.provision_datapath(row.datapath)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     ip2.addr.add_multiple(ipv4_cidrs_to_add)
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     privileged.add_ip_addresses(
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     return self.channel.remote_call(name, args, kwargs,
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event     raise exc_type(*result[2])
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
  2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event

  The thing is this line of code should actually be creating the new
  namespace[2] so not sure why its complaining that namespace was not
  found. I am suspecting there is some race condition or more likely
  some test interferance due to test runner concurrency.

  [0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17
  [1] https://review.opendev.org/c/openstack/neutron/+/875586
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2011377/+subscriptions
Follow ups

[Bug 2011377] Re: test_agent_resync_on_non_existing_bridge failing intermittently sp
From: OpenStack Infra, 2023-03-21