yahoo-eng-team team mailing list archive

[Slawek Kaplonski <1922684@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: In Progress => Fix Released

** Tags removed: neutron-proactive-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1922684

Title:
  Functional dhcp agent tests fails to spawn metadata proxy

Status in neutron:
  Fix Released

Bug description:
  In case when during spawning of the setup of dhcp agent, there will be
  iptabls error "Another app is currently holding the xtables lock"
  tests like
  neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete
  and
  neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_force_metadata_for_subnet_create_delete
  may fail with error like below:

  2021-03-30 20:57:01.829 61168 DEBUG neutron.agent.linux.dhcp [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Previous DHCP port information: <MagicMock name='DhcpPluginApi().create_dhcp_port()' id='139833378414448'>. Updated DHCP port information: <Mock name='DhcpPluginApi().get_dhcp_port()' id='139833360938464'>. _check_dhcp_port_subnet /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py:1582
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Unable to enable dhcp for 24e1cf2a-a60d-41a9-9666-a38a90117cf9.: TypeError: can not serialize 'MagicMock' object
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/dhcp/agent.py", line 227, in call_driver
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     rv = getattr(driver, action)(**action_kwargs)
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 266, in enable
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     common_utils.wait_until_true(self._enable, timeout=300)
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 707, in wait_until_true
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     while not predicate():
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 278, in _enable
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 1692, in setup
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     if ip_lib.ensure_device_is_ready(interface_name,
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 963, in ensure_device_is_ready
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     if not dev.link.exists or not dev.link.address:
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 500, in exists
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     return privileged.interface_exists(self.name, self._parent.namespace)
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     return self.channel.remote_call(name, args, kwargs)
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 214, in remote_call
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     result = self.send_recv((Message.CALL.value, name, args, kwargs))
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 170, in send_recv
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     self.writer.send((myid, msg))
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 54, in send
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     buf = msgpack.packb(msg, use_bin_type=True,
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent     return Packer(**kwargs).pack(o)
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent   File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
  2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent TypeError: can not serialize 'MagicMock' object

  
  Example of such error https://e9a79c0822aeb557489f-25bbaed08360ac14bb61b985414d524a.ssl.cf2.rackcdn.com/783743/1/check/neutron-functional-with-uwsgi/6f0aa76/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete.txt

  It happens like that because in that test there is no mock of
  neutron.agent.linux.dhcp.DeviceManager._update_dhcp_port() method.
  Workflow of such test is like below:

  1. Call DeviceManager.setup() method
  2. This method as one of the first steps will call _update_dhcp_port which will replace dhcp port prepared for test with some other mock,
  3. During first run "port" variable in DeviceManager.setup() method is correct so all will work fine but if we hit iptables xlock error, setup() method will be called again and
  4. Now "port" local variable is already update by _update_dhcp_port method thus test setup() method will fail as interface_name is now wrong.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1922684/+subscriptions