yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2115241] [NEW] test_spawn_rate_limited_metadata_proxy unit test fails randomly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2115241@xxxxxxxxxxxxxxxxxx>
Date: Tue, 24 Jun 2025 07:15:45 -0000
Reply-to: Bug 2115241 <2115241@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

neutron.tests.unit.agent.metadata.test_driver.TestMetadataDriverProcess.test_spawn_rate_limited_metadata_proxy
fails randomly as below:-

ft1.12: neutron.tests.unit.agent.metadata.test_driver.TestMetadataDriverProcess.test_spawn_rate_limited_metadata_proxytesttools.testresult.real._StringException: pythonlogging:'': {{{ERROR [neutron.agent.linux.external_process] haproxy for MetadataPath with uuid host_metadata_proxy not found. The process should not have died}}}
stderr: {{{
/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_messaging/server.py:354: DeprecationWarning: Eventlet usages are deprecated and the removal of Eventlet from OpenStack is planned, for this reason the Eventlet executor is deprecated. Start migrating your stack to the threading executor. Please also start considering removing your internal Eventlet usages. in version '2025.1' and will be removed in version '2026.1'
  debtcollector.deprecate(
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 306, in test_spawn_rate_limited_metadata_proxy
    return self._test_spawn_metadata_proxy(rate_limited=True)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 222, in _test_spawn_metadata_proxy
    agent.metadata_driver.spawn_monitored_metadata_proxy(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/metadata/driver_base.py", line 310, in spawn_monitored_metadata_proxy
    pm.enable(ensure_active=True)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 96, in enable
    if not self.active:
  File "/usr/lib/python3.10/unittest/mock.py", line 2938, in __get__
    return self()
  File "/usr/lib/python3.10/unittest/mock.py", line 1114, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.10/unittest/mock.py", line 1118, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/python3.10/unittest/mock.py", line 1175, in _execute_mock_call
    result = next(effect)
StopIteration

And other as:-
ft1.12: neutron.tests.unit.agent.metadata.test_driver.TestMetadataDriverProcess.test_spawn_rate_limited_metadata_proxytesttools.testresult.real._StringException: pythonlogging:'': {{{ERROR [neutron.agent.linux.external_process] haproxy for MetadataPath with uuid host_metadata_proxy not found. The process should not have died}}}
stderr: {{{
/home/zuul/src/opendev.org/openstack/neutron/.tox/py313/lib/python3.13/site-packages/oslo_messaging/server.py:354: DeprecationWarning: Eventlet usages are deprecated and the removal of Eventlet from OpenStack is planned, for this reason the Eventlet executor is deprecated. Start migrating your stack to the threading executor. Please also start considering removing your internal Eventlet usages. in version '2025.1' and will be removed in version '2026.1'
  debtcollector.deprecate(
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 306, in test_spawn_rate_limited_metadata_proxy
    return self._test_spawn_metadata_proxy(rate_limited=True)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 222, in _test_spawn_metadata_proxy
    agent.metadata_driver.spawn_monitored_metadata_proxy(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        agent.process_monitor,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        bind_address_v6=self.METADATA_DEFAULT_IPV6,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        bind_interface='fake-if')
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/metadata/driver_base.py", line 310, in spawn_monitored_metadata_proxy
    pm.enable(ensure_active=True)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 96, in enable
    if not self.active:
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.13/unittest/mock.py", line 3045, in __get__
    return self()
  File "/usr/local/lib/python3.13/unittest/mock.py", line 1169, in __call__
    return self._mock_call(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/unittest/mock.py", line 1173, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/unittest/mock.py", line 1230, in _execute_mock_call
    result = next(effect)
StopIteration

Builds:-
https://025b8612ea74b6df403c-82c7c26c1c9d82596e9efade16533251.ssl.cf1.rackcdn.com/openstack/8ae8a7c942354bb780425a17cd25e199/testr_results.html
https://9f4bad0085cac2a65c00-b3e1428302d70b58654d7733ebfb21f6.ssl.cf2.rackcdn.com/openstack/7cc99183a92c44b0b05d7ea7215c3146/testr_results.html


Related patch https://review.opendev.org/c/openstack/neutron/+/952220

Seen this twice on 16th and 17th June. Not sure if this already fixed
since than with some patch.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2115241

Title:
  test_spawn_rate_limited_metadata_proxy unit test fails randomly

Status in neutron:
  New

Bug description:
  neutron.tests.unit.agent.metadata.test_driver.TestMetadataDriverProcess.test_spawn_rate_limited_metadata_proxy
  fails randomly as below:-

  ft1.12: neutron.tests.unit.agent.metadata.test_driver.TestMetadataDriverProcess.test_spawn_rate_limited_metadata_proxytesttools.testresult.real._StringException: pythonlogging:'': {{{ERROR [neutron.agent.linux.external_process] haproxy for MetadataPath with uuid host_metadata_proxy not found. The process should not have died}}}
  stderr: {{{
  /home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_messaging/server.py:354: DeprecationWarning: Eventlet usages are deprecated and the removal of Eventlet from OpenStack is planned, for this reason the Eventlet executor is deprecated. Start migrating your stack to the threading executor. Please also start considering removing your internal Eventlet usages. in version '2025.1' and will be removed in version '2026.1'
    debtcollector.deprecate(
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 306, in test_spawn_rate_limited_metadata_proxy
      return self._test_spawn_metadata_proxy(rate_limited=True)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 222, in _test_spawn_metadata_proxy
      agent.metadata_driver.spawn_monitored_metadata_proxy(
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/metadata/driver_base.py", line 310, in spawn_monitored_metadata_proxy
      pm.enable(ensure_active=True)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 96, in enable
      if not self.active:
    File "/usr/lib/python3.10/unittest/mock.py", line 2938, in __get__
      return self()
    File "/usr/lib/python3.10/unittest/mock.py", line 1114, in __call__
      return self._mock_call(*args, **kwargs)
    File "/usr/lib/python3.10/unittest/mock.py", line 1118, in _mock_call
      return self._execute_mock_call(*args, **kwargs)
    File "/usr/lib/python3.10/unittest/mock.py", line 1175, in _execute_mock_call
      result = next(effect)
  StopIteration

  And other as:-
  ft1.12: neutron.tests.unit.agent.metadata.test_driver.TestMetadataDriverProcess.test_spawn_rate_limited_metadata_proxytesttools.testresult.real._StringException: pythonlogging:'': {{{ERROR [neutron.agent.linux.external_process] haproxy for MetadataPath with uuid host_metadata_proxy not found. The process should not have died}}}
  stderr: {{{
  /home/zuul/src/opendev.org/openstack/neutron/.tox/py313/lib/python3.13/site-packages/oslo_messaging/server.py:354: DeprecationWarning: Eventlet usages are deprecated and the removal of Eventlet from OpenStack is planned, for this reason the Eventlet executor is deprecated. Start migrating your stack to the threading executor. Please also start considering removing your internal Eventlet usages. in version '2025.1' and will be removed in version '2026.1'
    debtcollector.deprecate(
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 306, in test_spawn_rate_limited_metadata_proxy
      return self._test_spawn_metadata_proxy(rate_limited=True)
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/metadata/test_driver.py", line 222, in _test_spawn_metadata_proxy
      agent.metadata_driver.spawn_monitored_metadata_proxy(
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
          agent.process_monitor,
          ^^^^^^^^^^^^^^^^^^^^^^
      ...<5 lines>...
          bind_address_v6=self.METADATA_DEFAULT_IPV6,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          bind_interface='fake-if')
          ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/metadata/driver_base.py", line 310, in spawn_monitored_metadata_proxy
      pm.enable(ensure_active=True)
      ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 96, in enable
      if not self.active:
             ^^^^^^^^^^^
    File "/usr/local/lib/python3.13/unittest/mock.py", line 3045, in __get__
      return self()
    File "/usr/local/lib/python3.13/unittest/mock.py", line 1169, in __call__
      return self._mock_call(*args, **kwargs)
             ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/unittest/mock.py", line 1173, in _mock_call
      return self._execute_mock_call(*args, **kwargs)
             ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/unittest/mock.py", line 1230, in _execute_mock_call
      result = next(effect)
  StopIteration

  Builds:-
  https://025b8612ea74b6df403c-82c7c26c1c9d82596e9efade16533251.ssl.cf1.rackcdn.com/openstack/8ae8a7c942354bb780425a17cd25e199/testr_results.html
  https://9f4bad0085cac2a65c00-b3e1428302d70b58654d7733ebfb21f6.ssl.cf2.rackcdn.com/openstack/7cc99183a92c44b0b05d7ea7215c3146/testr_results.html

  
  Related patch https://review.opendev.org/c/openstack/neutron/+/952220

  Seen this twice on 16th and 17th June. Not sure if this already fixed
  since than with some patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2115241/+subscriptions