yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1435012] [NEW] Exception when removing last network port from external router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gal Sagie <gal.sagie@xxxxxxxxxx>
Date: Sun, 22 Mar 2015 14:20:07 -0000
Reply-to: Bug 1435012 <1435012@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Steps to reproduce:

Single node devstack setup.

Start devstack with ML2 and ovs,l2 population drivers.
Enable DVR in L3/L2 and Service plugin (with DVR_SNAT)

1. Create Network with an instance on it
2. Create Router attached to external network
3. Add the network to the router

(Two new interfaces should be added to the router, the network and an
SNAT interface

4. Delete the two added interfaces from the router (or just the network
one)

The following exception is shown in the L3 Agent:

2015-03-22 16:07:54.613 ERROR neutron.agent.linux.utils [-]
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-81176deb-cc2b-4d29-903e-0a1a2fe207d6', 'ip', '-4', 'route', 'del', 'default', 'via', '10.1.0.8', 'dev', 'qr-a3c4575d-5f', 'table', '167837697']
Exit code: 1
Stdin:
Stdout:
Stderr: Cannot find device "qr-a3c4575d-5f"

2015-03-22 16:07:54.614 ERROR neutron.agent.l3.dvr_router [-] DVR: removed snat failed
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Traceback (most recent call last):
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/l3/dvr_router.py", line 282, in _snat_redirect_remove
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     ns_ipd.route.delete_gateway(gateway, table=snat_idx)
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 409, in delete_gateway
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     self._as_root([ip_version], tuple(args))
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     use_root_namespace=use_root_namespace)
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=self.log_fail_as_error)
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=log_fail_as_error)
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 135, in execute
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     raise RuntimeError(m)
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router RuntimeError:
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-81176deb-cc2b-4d29-903e-0a1a2fe207d6', 'ip', '-4', 'route', 'del', 'default', 'via', '10.1.0.8', 'dev', 'qr-a3c4575d-5f', 'table', '167837697']
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Exit code: 1
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdin:
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdout:
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stderr: Cannot find device "qr-a3c4575d-5f"
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router
2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router

Assumption:
The code tries to delete the default route (for the SNAT) inside the router namespace, how ever the namespace
is already deleted (since this is only connected network)

** Affects: neutron
     Importance: Undecided
     Assignee: Gal Sagie (gal-sagie)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Gal Sagie (gal-sagie)

** Description changed:

  Steps to reproduce:
  
  Single node devstack setup.
  
  Start devstack with ML2 and ovs,l2 population drivers.
  Enable DVR in L3/L2 and Service plugin (with DVR_SNAT)
  
- 
  1. Create Network with an instance on it
- 2. Create Router attached to external router
+ 2. Create Router attached to external network
  3. Add the network to the router
  
  (Two new interfaces should be added to the router, the network and an
  SNAT interface
  
  4. Delete the two added interfaces from the router (or just the network
  one)
  
  The following exception is shown in the L3 Agent:
  
- 2015-03-22 16:07:54.613 ERROR neutron.agent.linux.utils [-] 
+ 2015-03-22 16:07:54.613 ERROR neutron.agent.linux.utils [-]
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-81176deb-cc2b-4d29-903e-0a1a2fe207d6', 'ip', '-4', 'route', 'del', 'default', 'via', '10.1.0.8', 'dev', 'qr-a3c4575d-5f', 'table', '167837697']
  Exit code: 1
- Stdin: 
- Stdout: 
+ Stdin:
+ Stdout:
  Stderr: Cannot find device "qr-a3c4575d-5f"
  
  2015-03-22 16:07:54.614 ERROR neutron.agent.l3.dvr_router [-] DVR: removed snat failed
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Traceback (most recent call last):
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/l3/dvr_router.py", line 282, in _snat_redirect_remove
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     ns_ipd.route.delete_gateway(gateway, table=snat_idx)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 409, in delete_gateway
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     self._as_root([ip_version], tuple(args))
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     use_root_namespace=use_root_namespace)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=self.log_fail_as_error)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=log_fail_as_error)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 135, in execute
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     raise RuntimeError(m)
- 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router RuntimeError: 
+ 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router RuntimeError:
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-81176deb-cc2b-4d29-903e-0a1a2fe207d6', 'ip', '-4', 'route', 'del', 'default', 'via', '10.1.0.8', 'dev', 'qr-a3c4575d-5f', 'table', '167837697']
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Exit code: 1
- 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdin: 
- 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdout: 
+ 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdin:
+ 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdout:
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stderr: Cannot find device "qr-a3c4575d-5f"
- 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router 
- 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router 
- 
+ 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router
+ 2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router
  
  Assumption:
  The code tries to delete the default route (for the SNAT) inside the router namespace, how ever the namespace
  is already deleted (since this is only connected network)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1435012

Title:
  Exception when removing last network port from external router

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Steps to reproduce:

  Single node devstack setup.

  Start devstack with ML2 and ovs,l2 population drivers.
  Enable DVR in L3/L2 and Service plugin (with DVR_SNAT)

  1. Create Network with an instance on it
  2. Create Router attached to external network
  3. Add the network to the router

  (Two new interfaces should be added to the router, the network and an
  SNAT interface

  4. Delete the two added interfaces from the router (or just the
  network one)

  The following exception is shown in the L3 Agent:

  2015-03-22 16:07:54.613 ERROR neutron.agent.linux.utils [-]
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-81176deb-cc2b-4d29-903e-0a1a2fe207d6', 'ip', '-4', 'route', 'del', 'default', 'via', '10.1.0.8', 'dev', 'qr-a3c4575d-5f', 'table', '167837697']
  Exit code: 1
  Stdin:
  Stdout:
  Stderr: Cannot find device "qr-a3c4575d-5f"

  2015-03-22 16:07:54.614 ERROR neutron.agent.l3.dvr_router [-] DVR: removed snat failed
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Traceback (most recent call last):
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/l3/dvr_router.py", line 282, in _snat_redirect_remove
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     ns_ipd.route.delete_gateway(gateway, table=snat_idx)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 409, in delete_gateway
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     self._as_root([ip_version], tuple(args))
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     use_root_namespace=use_root_namespace)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=self.log_fail_as_error)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     log_fail_as_error=log_fail_as_error)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 135, in execute
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router     raise RuntimeError(m)
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router RuntimeError:
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-81176deb-cc2b-4d29-903e-0a1a2fe207d6', 'ip', '-4', 'route', 'del', 'default', 'via', '10.1.0.8', 'dev', 'qr-a3c4575d-5f', 'table', '167837697']
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Exit code: 1
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdin:
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stdout:
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router Stderr: Cannot find device "qr-a3c4575d-5f"
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router
  2015-03-22 16:07:54.614 TRACE neutron.agent.l3.dvr_router

  Assumption:
  The code tries to delete the default route (for the SNAT) inside the router namespace, how ever the namespace
  is already deleted (since this is only connected network)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1435012/+subscriptions
Follow ups

[Bug 1435012] Re: Exception when removing last network port from external router
From: Thierry Carrez, 2015-06-24
[Bug 1435012] [NEW] Exception when removing last network port from external router
From: Gal Sagie, 2015-03-22
References

[Bug 1435012] [NEW] Exception when removing last network port from external router
From: Gal Sagie, 2015-03-22