yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1294254] [NEW] dhcp cannot find tap device exceptions

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Darragh O'Reilly <1294254@xxxxxxxxxxxxxxxxxx>
Date: Tue, 18 Mar 2014 18:02:12 -0000
Reply-to: Bug 1294254 <1294254@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

This happens in tempest tests a lot:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcImNhbm5vdCBmaW5kIGRldmljZVwiIEFORCBmaWxlbmFtZTpcImxvZ3Mvc2NyZWVuLXEtZGhjcC50eHRcIiBBTkQgYnVpbGRfYnJhbmNoOlwibWFzdGVyXCIgQU5EIHByb2plY3Q6XCJvcGVuc3RhY2svbmV1dHJvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI0MzIwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTUwNTgyMTI3OTF9

The _set_default_route function is called when the driver is called with
'enable' or 'reload_allocations'. It calls the get_dhcp_port rpc which
tries to get the previously created dhcp port for the network and host.
But sometimes that port has been deleted by this time, and then
get_dhcp_port actually creates a new one. This will have a different
uuid than the original, so the tap name will be different and the call
to list the routes on it gives an exception as that tap was never
created.

It is not clear from the code why _set_default_route needs to do this.


Example from call_driver reload_allocations
http://logs.openstack.org/12/59212/12/check/check-tempest-dsvm-neutron-full/e3c77d0/logs/screen-q-dhcp.txt.gz#_2014-03-17_10_16_03_166

[req-024b81f2-eac3-490f-920d-b9fa1f70f51e None] Unable to reload_allocations dhcp for 9edab258-c056-40e3-a340-1c58622380bf.
Traceback (most recent call last):
  File "/opt/stack/new/neutron/neutron/agent/dhcp_agent.py", line 127, in call_driver
    getattr(driver, action)(**action_kwargs)
  File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 400, in reload_allocations
    self.device_manager.update(self.network)
  File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 810, in update
    self._set_default_route(network)
  File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 669, in _set_default_route
    gateway = device.route.get_gateway()
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 388, in get_gateway
    *filters).split('\n')
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 211, in _run
    return self._parent._run(kwargs.get('options', []), self.COMMAND, args)
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 52, in _run
    return self._as_root(options, command, args)
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
    namespace)
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
    root_helper=root_helper)
  File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 76, in execute
    raise RuntimeError(m)
RuntimeError: 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-9edab258-c056-40e3-a340-1c58622380bf', 'ip', 'route', 'list', 'dev', 'tapa01643ce-47']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "tapa01643ce-47"\n'


Example from call_driver 'enable'
http://logs.openstack.org/12/59212/12/check/check-tempest-dsvm-neutron-full/e3c77d0/logs/screen-q-dhcp.txt.gz#_2014-03-17_10_16_03_166

[req-e5cdbe16-076d-4012-80af-7873ff14cdeb None] Unable to enable dhcp for 3236628d-4ccd-4c29-a2b6-ca4752725874.
Traceback (most recent call last):
  File "/opt/stack/new/neutron/neutron/agent/dhcp_agent.py", line 127, in call_driver
    getattr(driver, action)(**action_kwargs)
  File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 166, in enable
    reuse_existing=True)
  File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 803, in setup
    self._set_default_route(network)
  File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 669, in _set_default_route
    gateway = device.route.get_gateway()
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 388, in get_gateway
    *filters).split('\n')
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 211, in _run
    return self._parent._run(kwargs.get('options', []), self.COMMAND, args)
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 52, in _run
    return self._as_root(options, command, args)
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
    namespace)
  File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
    root_helper=root_helper)
  File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 76, in execute
    raise RuntimeError(m)
RuntimeError: 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-3236628d-4ccd-4c29-a2b6-ca4752725874', 'ip', 'route', 'list', 'dev', 'tap88655de5-5f']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "tap88655de5-5f"\n'

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-ipam-dhcp

** Tags added: l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1294254

Title:
  dhcp cannot find tap device exceptions

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  This happens in tempest tests a lot:

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcImNhbm5vdCBmaW5kIGRldmljZVwiIEFORCBmaWxlbmFtZTpcImxvZ3Mvc2NyZWVuLXEtZGhjcC50eHRcIiBBTkQgYnVpbGRfYnJhbmNoOlwibWFzdGVyXCIgQU5EIHByb2plY3Q6XCJvcGVuc3RhY2svbmV1dHJvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI0MzIwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTUwNTgyMTI3OTF9

  The _set_default_route function is called when the driver is called
  with 'enable' or 'reload_allocations'. It calls the get_dhcp_port rpc
  which tries to get the previously created dhcp port for the network
  and host. But sometimes that port has been deleted by this time, and
  then get_dhcp_port actually creates a new one. This will have a
  different uuid than the original, so the tap name will be different
  and the call to list the routes on it gives an exception as that tap
  was never created.

  It is not clear from the code why _set_default_route needs to do this.

  
  Example from call_driver reload_allocations
  http://logs.openstack.org/12/59212/12/check/check-tempest-dsvm-neutron-full/e3c77d0/logs/screen-q-dhcp.txt.gz#_2014-03-17_10_16_03_166

  [req-024b81f2-eac3-490f-920d-b9fa1f70f51e None] Unable to reload_allocations dhcp for 9edab258-c056-40e3-a340-1c58622380bf.
  Traceback (most recent call last):
    File "/opt/stack/new/neutron/neutron/agent/dhcp_agent.py", line 127, in call_driver
      getattr(driver, action)(**action_kwargs)
    File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 400, in reload_allocations
      self.device_manager.update(self.network)
    File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 810, in update
      self._set_default_route(network)
    File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 669, in _set_default_route
      gateway = device.route.get_gateway()
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 388, in get_gateway
      *filters).split('\n')
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 211, in _run
      return self._parent._run(kwargs.get('options', []), self.COMMAND, args)
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 52, in _run
      return self._as_root(options, command, args)
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
      namespace)
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
      root_helper=root_helper)
    File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 76, in execute
      raise RuntimeError(m)
  RuntimeError: 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-9edab258-c056-40e3-a340-1c58622380bf', 'ip', 'route', 'list', 'dev', 'tapa01643ce-47']
  Exit code: 1
  Stdout: ''
  Stderr: 'Cannot find device "tapa01643ce-47"\n'

  
  Example from call_driver 'enable'
  http://logs.openstack.org/12/59212/12/check/check-tempest-dsvm-neutron-full/e3c77d0/logs/screen-q-dhcp.txt.gz#_2014-03-17_10_16_03_166

  [req-e5cdbe16-076d-4012-80af-7873ff14cdeb None] Unable to enable dhcp for 3236628d-4ccd-4c29-a2b6-ca4752725874.
  Traceback (most recent call last):
    File "/opt/stack/new/neutron/neutron/agent/dhcp_agent.py", line 127, in call_driver
      getattr(driver, action)(**action_kwargs)
    File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 166, in enable
      reuse_existing=True)
    File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 803, in setup
      self._set_default_route(network)
    File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 669, in _set_default_route
      gateway = device.route.get_gateway()
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 388, in get_gateway
      *filters).split('\n')
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 211, in _run
      return self._parent._run(kwargs.get('options', []), self.COMMAND, args)
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 52, in _run
      return self._as_root(options, command, args)
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
      namespace)
    File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
      root_helper=root_helper)
    File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 76, in execute
      raise RuntimeError(m)
  RuntimeError: 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-3236628d-4ccd-4c29-a2b6-ca4752725874', 'ip', 'route', 'list', 'dev', 'tap88655de5-5f']
  Exit code: 1
  Stdout: ''
  Stderr: 'Cannot find device "tap88655de5-5f"\n'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1294254/+subscriptions
Follow ups

[Bug 1294254] Re: dhcp cannot find tap device exceptions
From: Chuck Short, 2014-08-07
[Bug 1294254] Re: dhcp cannot find tap device exceptions
From: Chuck Short, 2014-08-07
[Bug 1294254] Re: dhcp cannot find tap device exceptions
From: Thierry Carrez, 2014-06-12
[Bug 1294254] [NEW] dhcp cannot find tap device exceptions
From: Darragh O'Reilly, 2014-03-18
References

[Bug 1294254] [NEW] dhcp cannot find tap device exceptions
From: Darragh O'Reilly, 2014-03-18