yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1341042] Re: using neutron CLI to set a bad gateway result in dnsmasq continuously restart

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: ugvddm <zhengguangou@xxxxxxxxx>
Date: Thu, 17 Jul 2014 15:01:58 -0000
Reply-to: Bug 1341042 <1341042@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
It seems like duplicated https://bugs.launchpad.net/neutron/+bug/1304181

** Changed in: neutron
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1341042

Title:
  using neutron CLI to set a bad gateway result in dnsmasq continuously
  restart

Status in OpenStack Neutron (virtual network service):
  Invalid

Bug description:
  I'm using devstack stable/icehouse, and my neutron version is
  1409da70959496375f1ccccac45457663a918ec8

  I created an internal network not connected to the router using default gateway and dhcp enabled.
  I then created a VM which activated dnsmasq, which offered an IP to this VM.

  If I mis-update the gateway, by going to Horizon, "Networks", "test-net", "subnet-2" "Edit Subnet", and set
    Gateway IP (optional): 10.100.100.100
  or mis-update the gateway using neutron CLI,
  it results in neutron-dhcp-agent continously restarting the dnsmasq for the "private" network.

  Horizon will not allow the user to mis-update the gateway, if I set "force_gateway_on_subnet = True" in /etc/neutron/neutron.conf.
  But this parameter does not affect the behavior of the neutron CLI.

  
  > neutron net-create test-net
  Created a new network:
  +----------------+--------------------------------------+
  | Field          | Value                                |
  +----------------+--------------------------------------+
  | admin_state_up | True                                 |
  | id             | b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a |
  | name           | test-net                             |
  | shared         | False                                |
  | status         | ACTIVE                               |
  | subnets        |                                      |
  | tenant_id      | 8092813be8fd4122a20ee3a6bfe91162     |
  +----------------+--------------------------------------+

  > neutron net-list
  +--------------------------------------+----------+--------------------------------------------------+
  | id                                   | name     | subnets                                          |
  +--------------------------------------+----------+--------------------------------------------------+
  | 0dd5722d-f535-42ec-9257-437c05e4de25 | private  | 81859ee5-4ea5-4e60-ab2a-ba74146d39ba 10.0.0.0/24 |
  | 27c1649d-f6fc-4893-837d-dbc293fc4b80 | public   | 6c1836a1-eb7d-4acb-ad6f-6c394cedced5             |
  | b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a | test-net |                                                  |
  +--------------------------------------+----------+--------------------------------------------------+

  > neutron subnet-create --name subnet-2 test-net --enable_dhcp=True 10.10.150.0/24
  Created a new subnet:
  +------------------+--------------------------------------------------+
  | Field            | Value                                            |
  +------------------+--------------------------------------------------+
  | allocation_pools | {"start": "10.10.150.2", "end": "10.10.150.254"} |
  | cidr             | 10.10.150.0/24                                   |
  | dns_nameservers  |                                                  |
  | enable_dhcp      | True                                             |
  | gateway_ip       | 10.10.150.1                                      |
  | host_routes      |                                                  |
  | id               | 4e462af6-7514-413c-bf55-4ae9a1643ce9             |
  | ip_version       | 4                                                |
  | name             | subnet-2                                         |
  | network_id       | b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a             |
  | tenant_id        | 8092813be8fd4122a20ee3a6bfe91162                 |
  +------------------+--------------------------------------------------+

  > neutron subnet-list
  +--------------------------------------+----------------+----------------+--------------------------------------------------+
  | id                                   | name           | cidr           | allocation_pools                                 |
  +--------------------------------------+----------------+----------------+--------------------------------------------------+
  | 4e462af6-7514-413c-bf55-4ae9a1643ce9 | subnet-2       | 10.10.150.0/24 | {"start": "10.10.150.2", "end": "10.10.150.254"} |
  | 81859ee5-4ea5-4e60-ab2a-ba74146d39ba | private-subnet | 10.0.0.0/24    | {"start": "10.0.0.2", "end": "10.0.0.254"}       |
  +--------------------------------------+----------------+----------------+--------------------------------------------------+

  > ps -ef |grep dnsmasq | grep -v grep
  nobody     995   635  0 16:09 ?        00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
  nobody   12991     1  0 18:21 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap75786806-5a --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/host --addn-hosts=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

  > nova boot --image cirros-0.3.1-x86_64-uec --flavor m1.tiny --nic net-id=b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a vm2
  :
  :

  > nova list
  +--------------------------------------+------+--------+------------+-------------+----------------------+
  | ID                                   | Name | Status | Task State | Power State | Networks             |
  +--------------------------------------+------+--------+------------+-------------+----------------------+
  | d98511f7-452c-4ab6-8af9-d73576714c87 | vm1  | ACTIVE | -          | Running     | private=10.0.0.2     |
  | 5934a236-edfc-4662-85f4-f00bdac1137a | vm2  | ACTIVE | -          | Running     | test-net=10.10.150.2 |
  +--------------------------------------+------+--------+------------+-------------+----------------------+

  
  > ps -ef | grep dnsmasq | grep -v grep
  nobody     995   635  0 16:09 ?        00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
  nobody   12991     1  0 18:21 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap75786806-5a --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/host --addn-hosts=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
  nobody   25284     1  0 19:02 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap41fe641d-7c --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/host --addn-hosts=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/opts --leasefile-ro --dhcp-range=tag0,10.10.150.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

  19:17:42 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
  995
  12991
  25284

  > neutron subnet-update 4e462af6-7514-413c-bf55-4ae9a1643ce9 --gateway_ip=10.100.100.100
  Updated subnet: 4e462af6-7514-413c-bf55-4ae9a1643ce9

  
  19:19:36 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
  995
  25284
  30892

  19:19:49 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
  995
  25284
  30999

  19:19:59 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
  995
  25284
  31137

  
  I get the following output from screen:

  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/dhcp_agent.py", line 129, in call_driver
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     getattr(driver, action)(**action_kwargs)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 165, in enable
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     reuse_existing=True)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 848, in setup
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     self._set_default_route(network, interface_name)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 731, in _set_default_route
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     device.route.add_gateway(subnet.gateway_ip)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 369, in add_gateway
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     self._as_root(*args)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 218, in _as_root
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     kwargs.get('use_root_namespace', False))
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 71, in _as_root
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     namespace)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 82, in _execute
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     root_helper=root_helper)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 76, in execute
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent     raise RuntimeError(m)
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent RuntimeError:
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a', 'ip', 'route', 'replace', 'default', 'via', '10.100.100.100', 'dev', 'tap41fe641d-7c']
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Exit code: 2
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Stdout: ''
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Network is unreachable\n'
  2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1341042/+subscriptions
References

[Bug 1341042] [NEW] using neutron CLI to set a bad gateway result in dnsmasq continuously restart
From: Pauline Yeung, 2014-07-12