yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1579458] [NEW] dhcp port was deleted unexpectedly when running rally list_and_create_ports test

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: SHI Peiqi <shipeiqi@xxxxxxxxxxxxxxxxxxxx>
Date: Sun, 08 May 2016 03:34:40 -0000
Reply-to: Bug 1579458 <1579458@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I tested this test in devstack master: (the content of json is not
related to the bug, so I don't detail the content)

        rally --debug task start
/opt/stack/rally/samples/tasks/scenarios/neutron/create_and_list_ports.json

The test shows it passed successfully on the high level, however in the
q-dhcp.log file， I found same errors:

2016-05-08 09:26:48.918 ^[[01;31mERROR neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[01;31m] ^[[01;35m^[[01;31mExit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "t      ap123bb3c5-ab"
15639 ^[[00m
15640 2016-05-08 09:26:48.919 ^[[01;31mERROR neutron.agent.dhcp.agent [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb2      955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[01;31m] ^[[01;35m^[[01;31mUnable to reload_allocations dhcp for 1a671e08-78b5-45fa-8bfe-f      9651444b242.^[[00m
15641 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00mTraceback (most recent call last):
15642 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", l      ine 113, in call_driver
15643 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    getattr(driver, action)(**action_kwargs)
15644 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", l      ine 477, in reload_allocations
15645 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    self.device_manager.update(self.network, self.interfac      e_name)
15646 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", l      ine 1281, in update
15647 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    self._set_default_route(network, device_name)
15648 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", l      ine 1015, in _set_default_route
15649 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    gateway = device.route.get_gateway()
15650 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 765, in get_gateway
15651 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    route_list_lines = self._run(options, tuple(args)).spl      it('\n')
15652 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 355, in _run
15653 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    return self._parent._run(options, self.COMMAND, args)
15654 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 79, in _run
15655 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    return self._as_root(options, command, args)
15656 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 94, in _as_root
15657 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    log_fail_as_error=self.log_fail_as_error)
15658 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 103, in _execute
15659 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    log_fail_as_error=log_fail_as_error)
15660 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/utils.py",       line 137, in execute
15661 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    raise RuntimeError(msg)
15662 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00mRuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Can      not find device "tap123bb3c5-ab"


This is because the dhcp port was deleted before that:

15624 2016-05-08 09:26:48.579 ^[[00;32mDEBUG neutron.agent.dhcp.agent [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb2      955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mCalling driver for network: 1a671e08-78b5-45fa-8bfe-f9651444b24      2 action: reload_allocations^[[00m ^[[00;33mfrom (pid=32551) call_driver /opt/stack/neutron/neutron/agent/dhcp/agent.py:104^[[00m
15625 2016-05-08 09:26:48.584 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mRunning command (rootwrap daemon): ['ip', 'netns', 'exec', 'qd      hcp-1a671e08-78b5-45fa-8bfe-f9651444b242', 'dhcp_release', 'tap123bb3c5-ab', '10.2.1.2', 'fa:16:3e:f2:f9:f8']^[[00m ^[[00;33mfrom (pid=32      551) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98^[[00m
15626 2016-05-08 09:26:48.649 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mExit code: 0^[[00m ^[[00;33mfrom (pid=32551) execute /opt/stac      k/neutron/neutron/agent/linux/utils.py:139^[[00m
15627 2016-05-08 09:26:48.650 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mRunning command: ['ip', '-o', 'link', 'show', 'br-int']^[[00m       ^[[00;33mfrom (pid=32551) create_process /opt/stack/neutron/neutron/agent/linux/utils.py:82^[[00m
15628 2016-05-08 09:26:48.675 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mExit code: 0^[[00m ^[[00;33mfrom (pid=32551) execute /opt/stac      k/neutron/neutron/agent/linux/utils.py:139^[[00m
15629 2016-05-08 09:26:48.676 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mRunning command (rootwrap daemon): ['ovs-vsctl', '--timeout=10      ', '--oneline', '--format=json', '--', '--if-exists', 'del-port', 'br-int', 'tap123bb3c5-ab']^[[00m ^[[00;33mfrom (pid=32551) execute_roo      twrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98^[[00m

** Affects: neutron
     Importance: Undecided
     Assignee: SHI Peiqi (uestc-shi)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => SHI Peiqi (uestc-shi)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1579458

Title:
  dhcp port was deleted unexpectedly when running rally
  list_and_create_ports test

Status in neutron:
  New

Bug description:
  I tested this test in devstack master: (the content of json is not
  related to the bug, so I don't detail the content)

          rally --debug task start
  /opt/stack/rally/samples/tasks/scenarios/neutron/create_and_list_ports.json

  The test shows it passed successfully on the high level, however in
  the q-dhcp.log file， I found same errors:

  2016-05-08 09:26:48.918 ^[[01;31mERROR neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[01;31m] ^[[01;35m^[[01;31mExit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "t      ap123bb3c5-ab"
  15639 ^[[00m
  15640 2016-05-08 09:26:48.919 ^[[01;31mERROR neutron.agent.dhcp.agent [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb2      955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[01;31m] ^[[01;35m^[[01;31mUnable to reload_allocations dhcp for 1a671e08-78b5-45fa-8bfe-f      9651444b242.^[[00m
  15641 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00mTraceback (most recent call last):
  15642 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", l      ine 113, in call_driver
  15643 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    getattr(driver, action)(**action_kwargs)
  15644 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", l      ine 477, in reload_allocations
  15645 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    self.device_manager.update(self.network, self.interfac      e_name)
  15646 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", l      ine 1281, in update
  15647 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    self._set_default_route(network, device_name)
  15648 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", l      ine 1015, in _set_default_route
  15649 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    gateway = device.route.get_gateway()
  15650 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 765, in get_gateway
  15651 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    route_list_lines = self._run(options, tuple(args)).spl      it('\n')
  15652 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 355, in _run
  15653 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    return self._parent._run(options, self.COMMAND, args)
  15654 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 79, in _run
  15655 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    return self._as_root(options, command, args)
  15656 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 94, in _as_root
  15657 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    log_fail_as_error=self.log_fail_as_error)
  15658 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py",       line 103, in _execute
  15659 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    log_fail_as_error=log_fail_as_error)
  15660 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/utils.py",       line 137, in execute
  15661 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m    raise RuntimeError(msg)
  15662 ^[[01;31m2016-05-08 09:26:48.919 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00mRuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Can      not find device "tap123bb3c5-ab"

  
  This is because the dhcp port was deleted before that:

  15624 2016-05-08 09:26:48.579 ^[[00;32mDEBUG neutron.agent.dhcp.agent [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb2      955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mCalling driver for network: 1a671e08-78b5-45fa-8bfe-f9651444b24      2 action: reload_allocations^[[00m ^[[00;33mfrom (pid=32551) call_driver /opt/stack/neutron/neutron/agent/dhcp/agent.py:104^[[00m
  15625 2016-05-08 09:26:48.584 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mRunning command (rootwrap daemon): ['ip', 'netns', 'exec', 'qd      hcp-1a671e08-78b5-45fa-8bfe-f9651444b242', 'dhcp_release', 'tap123bb3c5-ab', '10.2.1.2', 'fa:16:3e:f2:f9:f8']^[[00m ^[[00;33mfrom (pid=32      551) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98^[[00m
  15626 2016-05-08 09:26:48.649 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mExit code: 0^[[00m ^[[00;33mfrom (pid=32551) execute /opt/stac      k/neutron/neutron/agent/linux/utils.py:139^[[00m
  15627 2016-05-08 09:26:48.650 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mRunning command: ['ip', '-o', 'link', 'show', 'br-int']^[[00m       ^[[00;33mfrom (pid=32551) create_process /opt/stack/neutron/neutron/agent/linux/utils.py:82^[[00m
  15628 2016-05-08 09:26:48.675 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mExit code: 0^[[00m ^[[00;33mfrom (pid=32551) execute /opt/stac      k/neutron/neutron/agent/linux/utils.py:139^[[00m
  15629 2016-05-08 09:26:48.676 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[01;36mreq-bc440ad1-097e-4854-ba9f-b3d6fa3ebc57 ^[[00;36mc_rally_aabb      2955_PDaqOU6i 41c974f05789403894531cd2979bd870^[[00;32m] ^[[01;35m^[[00;32mRunning command (rootwrap daemon): ['ovs-vsctl', '--timeout=10      ', '--oneline', '--format=json', '--', '--if-exists', 'del-port', 'br-int', 'tap123bb3c5-ab']^[[00m ^[[00;33mfrom (pid=32551) execute_roo      twrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98^[[00m

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1579458/+subscriptions
Follow ups

[Bug 1579458] Re: dhcp port was deleted unexpectedly when running rally list_and_create_ports test
From: Armando Migliaccio, 2017-02-01