← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #80485

[Bug 1849726] [NEW] Routed Networks DHCP agents getting ports in unconnected segments

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Using Queens in Ubuntu 18.04, with Routed Provider Networks (network
segments).

DHCP agent RPC Callback in Neutron Server/API is returning DHCP reserved
ports binded to fixed_ips of subnets in other segments, raising the
error copied below when DHCP agent starts.

In our understanding, this situation arises when DHCP agents are stopped
and its network namespace is  removed, but it corresponding port not
deleted. In our case we installed too many DHCP agents in each segment,
so we decided to stop and uninstall several of them, cleaning the Linux
network namespaces but not deleting the orphaned dhcp ports. When we
tried to reinstall an agent in other compute, we hit the error.

---------------
Currently the list of subnets is correctly filtered (https://review.opendev.org/#/c/339365/) and works ok for newly started DHCP agents:

  subnets = [subnet for subnet in subnets
              if subnet['network_id'] not in routed_net_ids or
              subnet['id'] in seg_subnet_ids]

...but the list of ports is not filtered accordingly. We worked around
this issue filtering ports immediatly below subnet filter:

  ports = [ port for port in ports
              if port['fixed_ips'][0]['subnet_id'] in seg_subnet_ids ]

------------------
ERROR oslo_messaging.rpc.server [req-bd6e9240-7844-4e0b-9c13-ee05e24f01a9 - - - - -] Exception during message handling: HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology.
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped
ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 305, in update_dhcp_port
ERROR oslo_messaging.rpc.server     return self._port_action(plugin, context, port, 'update_port')
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 100, in _port_action
ERROR oslo_messaging.rpc.server     return plugin.update_port(context, port['id'], port)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 627, in inner
ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper
ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server     self.force_reraise()
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped
ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1359, in update_port
ERROR oslo_messaging.rpc.server     port)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1345, in update_port
ERROR oslo_messaging.rpc.server     new_port=new_port)
ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_backend_mixin.py", line 783, in update_port
ERROR oslo_messaging.rpc.server     host=host, port_id=old_port['id'])
ERROR oslo_messaging.rpc.server HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1849726

Title:
  Routed Networks DHCP agents getting ports in unconnected segments

Status in neutron:
  New

Bug description:
  Using Queens in Ubuntu 18.04, with Routed Provider Networks (network
  segments).

  DHCP agent RPC Callback in Neutron Server/API is returning DHCP
  reserved ports binded to fixed_ips of subnets in other segments,
  raising the error copied below when DHCP agent starts.

  In our understanding, this situation arises when DHCP agents are
  stopped and its network namespace is  removed, but it corresponding
  port not deleted. In our case we installed too many DHCP agents in
  each segment, so we decided to stop and uninstall several of them,
  cleaning the Linux network namespaces but not deleting the orphaned
  dhcp ports. When we tried to reinstall an agent in other compute, we
  hit the error.

  ---------------
  Currently the list of subnets is correctly filtered (https://review.opendev.org/#/c/339365/) and works ok for newly started DHCP agents:

    subnets = [subnet for subnet in subnets
                if subnet['network_id'] not in routed_net_ids or
                subnet['id'] in seg_subnet_ids]

  ...but the list of ports is not filtered accordingly. We worked around
  this issue filtering ports immediatly below subnet filter:

    ports = [ port for port in ports
                if port['fixed_ips'][0]['subnet_id'] in seg_subnet_ids ]

  ------------------
  ERROR oslo_messaging.rpc.server [req-bd6e9240-7844-4e0b-9c13-ee05e24f01a9 - - - - -] Exception during message handling: HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology.
  ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
  ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
  ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
  ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
  ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 305, in update_dhcp_port
  ERROR oslo_messaging.rpc.server     return self._port_action(plugin, context, port, 'update_port')
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 100, in _port_action
  ERROR oslo_messaging.rpc.server     return plugin.update_port(context, port['id'], port)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 627, in inner
  ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
  ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
  ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
  ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper
  ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
  ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR oslo_messaging.rpc.server     self.force_reraise()
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped
  ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1359, in update_port
  ERROR oslo_messaging.rpc.server     port)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
  ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1345, in update_port
  ERROR oslo_messaging.rpc.server     new_port=new_port)
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_backend_mixin.py", line 783, in update_port
  ERROR oslo_messaging.rpc.server     host=host, port_id=old_port['id'])
  ERROR oslo_messaging.rpc.server HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1849726/+subscriptions
  Thread Previous • Date Previous • • Thread Next