← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #54800

[Bug 1611060] [NEW] KeyError in get_isolated_subnets cause dhcp not work properly

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

DhcpRpcCallback.get_active_networks_info() returns all the networks/subnets/ports in system when dhcp agent restart or resync to improve dhcp agent performance. But it filter subnets with "enable_dhcp=True" when calls get_subnets(), which is inconsistent with what DhcpRpcCallback.get_network_info() does. 
Disable dhcp of certain non-isolated subnets in a network and enable the others. get_active_networks_info() will return subnets which is dhcp enabled only and all ports in the network including router interface of subnets which is dhcp disabled. This will cause KeyError in Dnsmasq.get_isolated_subnets() when iterate router interface ports because certain subnets miss, so that dnsmasq process will not start up. See https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L951.
Besides, routes for neighbor IPv4 subnets in a network would be incomplete due to it.

How to reproduce:
- Create a network and a router
$ neutron net-create net4dhcp
$ neutron router-create router

- Create two non-isolated subnets and enable one's dhcp only
$ neutron subnet-create --name subnet_dhcp_enabled net4dhcp 192.168.1.0/24
$ neutron router-interface-add router subnet=subnet_dhcp_enabled
$ neutron subnet-create --name subnet_dhcp_disabled --disable-dhcp net4dhcp 192.168.2.0/24
$ neutron router-interface-add router subnet=subnet_dhcp_disabled

- Restart the dhcp agent which hosting qdhcp namespace of this network

Error log:
2016-08-08 23:51:39.837 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for a3b25d9d-d1cd-4c29-959f-127894b342ed.
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent Traceback (most recent call last):
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 115, in call_driver
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 213, in enable
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self.spawn_process()
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 425, in spawn_process
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self._spawn_or_reload_process(reload_with_HUP=False)
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 434, in _spawn_or_reload_process
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self._output_config_files()
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 461, in _output_config_files
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self._output_opts_file()
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 756, in _output_opts_file
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     options, subnet_index_map = self._generate_opts_per_subnet()
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 768, in _generate_opts_per_subnet
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     isolated_subnets = self.get_isolated_subnets(self.network)
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 951, in get_isolated_subnets
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     if subnets[alloc.subnet_id].gateway_ip == alloc.ip_address:
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent KeyError: u'26b7ed7d-4116-4b14-a48b-9e034d1d28f2'
2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent

Affected versions:
It can be reproduced in branch master, liberty and mitaka.

** Affects: neutron
     Importance: Undecided
     Assignee: Quan Tian (tianquan23)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Quan Tian (tianquan23)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1611060

Title:
  KeyError in get_isolated_subnets cause dhcp not work properly

Status in neutron:
  In Progress

Bug description:
  DhcpRpcCallback.get_active_networks_info() returns all the networks/subnets/ports in system when dhcp agent restart or resync to improve dhcp agent performance. But it filter subnets with "enable_dhcp=True" when calls get_subnets(), which is inconsistent with what DhcpRpcCallback.get_network_info() does. 
  Disable dhcp of certain non-isolated subnets in a network and enable the others. get_active_networks_info() will return subnets which is dhcp enabled only and all ports in the network including router interface of subnets which is dhcp disabled. This will cause KeyError in Dnsmasq.get_isolated_subnets() when iterate router interface ports because certain subnets miss, so that dnsmasq process will not start up. See https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L951.
  Besides, routes for neighbor IPv4 subnets in a network would be incomplete due to it.

  How to reproduce:
  - Create a network and a router
  $ neutron net-create net4dhcp
  $ neutron router-create router

  - Create two non-isolated subnets and enable one's dhcp only
  $ neutron subnet-create --name subnet_dhcp_enabled net4dhcp 192.168.1.0/24
  $ neutron router-interface-add router subnet=subnet_dhcp_enabled
  $ neutron subnet-create --name subnet_dhcp_disabled --disable-dhcp net4dhcp 192.168.2.0/24
  $ neutron router-interface-add router subnet=subnet_dhcp_disabled

  - Restart the dhcp agent which hosting qdhcp namespace of this network

  Error log:
  2016-08-08 23:51:39.837 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for a3b25d9d-d1cd-4c29-959f-127894b342ed.
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent Traceback (most recent call last):
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 115, in call_driver
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 213, in enable
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self.spawn_process()
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 425, in spawn_process
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self._spawn_or_reload_process(reload_with_HUP=False)
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 434, in _spawn_or_reload_process
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self._output_config_files()
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 461, in _output_config_files
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     self._output_opts_file()
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 756, in _output_opts_file
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     options, subnet_index_map = self._generate_opts_per_subnet()
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 768, in _generate_opts_per_subnet
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     isolated_subnets = self.get_isolated_subnets(self.network)
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 951, in get_isolated_subnets
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent     if subnets[alloc.subnet_id].gateway_ip == alloc.ip_address:
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent KeyError: u'26b7ed7d-4116-4b14-a48b-9e034d1d28f2'
  2016-08-08 23:51:39.837 TRACE neutron.agent.dhcp.agent

  Affected versions:
  It can be reproduced in branch master, liberty and mitaka.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1611060/+subscriptions
  Thread Previous • Date Previous • • Thread Next