yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1998337] [NEW] test_dvr_router_lifecycle_ha_with_snat_with_fips fails occasionally in the gate

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bence Romsics <1998337@xxxxxxxxxxxxxxxxxx>
Date: Wed, 30 Nov 2022 13:54:11 -0000
Reply-to: Bug 1998337 <1998337@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Opening this report to track the following test that fails occasionally
in the gate:

job neutron-functional-with-uwsgi
test neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_router_lifecycle_ha_with_snat_with_fipstesttools

Sample traceback:

ft1.31: neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_router_lifecycle_ha_with_snat_with_fipstesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 208, in test_dvr_router_lifecycle_ha_with_snat_with_fips
    self._dvr_router_lifecycle(enable_ha=True, enable_snat=True)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 626, in _dvr_router_lifecycle
    self._assert_dvr_floating_ips(router, snat_bound_fip=snat_bound_fip,
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 791, in _assert_dvr_floating_ips
    self.assertTrue(fg_port_created_successfully)
  File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

It seems to recur occasionally, for example:

https://675daf3418638bf15806-f7e1f8eddcfdd9404f4b72ab9bb1f324.ssl.cf1.rackcdn.com/865575/1/check/neutron-functional-with-uwsgi/bd983b3/testr_results.html
https://488eb2b76bde124417ee-80e67ec01f194d5b25d665df26ee3378.ssl.cf2.rackcdn.com/839066/18/check/neutron-functional-with-uwsgi/66c7fcc/testr_results.html

There may be more that's similar:

$ logsearch log --project openstack/neutron --result FAILURE --pipeline check --job neutron-functional-with-uwsgi --limit 30 'line 208, in test_dvr_router_lifecycle_ha_with_snat_with_fips'
Builds with matching logs 5/30:
+----------------------------------+---------------------+-----------------------------------+--------+
| uuid                             | finished            | review                            | branch |
+----------------------------------+---------------------+-----------------------------------+--------+
| 1d265722d23548d6930486699202347d | 2022-11-30T13:42:28 | https://review.opendev.org/863881 | master |
| cb2a2d7161764d5f823a09528eedc44c | 2022-11-28T16:47:20 | https://review.opendev.org/865018 | master |
| 66c7fcc56a5347648732bfcb90341ef5 | 2022-11-27T00:55:10 | https://review.opendev.org/839066 | master |
| 85b3b709e9d54718a4f0847da5b4b2df | 2022-11-25T10:00:01 | https://review.opendev.org/865018 | master |
| bd983b367ac441c190e38dcf1fadc87f | 2022-11-24T16:17:06 | https://review.opendev.org/865575 | master |
+----------------------------------+---------------------+-----------------------------------+--------+

** Affects: neutron
     Importance: Medium
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1998337

Title:
  test_dvr_router_lifecycle_ha_with_snat_with_fips fails occasionally in
  the gate

Status in neutron:
  New

Bug description:
  Opening this report to track the following test that fails
  occasionally in the gate:

  job neutron-functional-with-uwsgi
  test neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_router_lifecycle_ha_with_snat_with_fipstesttools

  Sample traceback:

  ft1.31: neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_router_lifecycle_ha_with_snat_with_fipstesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 208, in test_dvr_router_lifecycle_ha_with_snat_with_fips
      self._dvr_router_lifecycle(enable_ha=True, enable_snat=True)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 626, in _dvr_router_lifecycle
      self._assert_dvr_floating_ips(router, snat_bound_fip=snat_bound_fip,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 791, in _assert_dvr_floating_ips
      self.assertTrue(fg_port_created_successfully)
    File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue
      raise self.failureException(msg)
  AssertionError: False is not true

  It seems to recur occasionally, for example:

  https://675daf3418638bf15806-f7e1f8eddcfdd9404f4b72ab9bb1f324.ssl.cf1.rackcdn.com/865575/1/check/neutron-functional-with-uwsgi/bd983b3/testr_results.html
  https://488eb2b76bde124417ee-80e67ec01f194d5b25d665df26ee3378.ssl.cf2.rackcdn.com/839066/18/check/neutron-functional-with-uwsgi/66c7fcc/testr_results.html

  There may be more that's similar:

  $ logsearch log --project openstack/neutron --result FAILURE --pipeline check --job neutron-functional-with-uwsgi --limit 30 'line 208, in test_dvr_router_lifecycle_ha_with_snat_with_fips'
  Builds with matching logs 5/30:
  +----------------------------------+---------------------+-----------------------------------+--------+
  | uuid                             | finished            | review                            | branch |
  +----------------------------------+---------------------+-----------------------------------+--------+
  | 1d265722d23548d6930486699202347d | 2022-11-30T13:42:28 | https://review.opendev.org/863881 | master |
  | cb2a2d7161764d5f823a09528eedc44c | 2022-11-28T16:47:20 | https://review.opendev.org/865018 | master |
  | 66c7fcc56a5347648732bfcb90341ef5 | 2022-11-27T00:55:10 | https://review.opendev.org/839066 | master |
  | 85b3b709e9d54718a4f0847da5b4b2df | 2022-11-25T10:00:01 | https://review.opendev.org/865018 | master |
  | bd983b367ac441c190e38dcf1fadc87f | 2022-11-24T16:17:06 | https://review.opendev.org/865575 | master |
  +----------------------------------+---------------------+-----------------------------------+--------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1998337/+subscriptions