← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #08934

[Bug 1271958] [NEW] nova compute fail to remove instance with port if network is broken

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

If user was capable to create broken network configuration, instance
become undeletable.  Reason why user can create broken networking is
under investigation (current hypothesis: if network (neutron) created in
one tennant and instance in other, and user is admin in both tenants, it
cause broken configuration).

But such instance deletetion cause trace on nova-compute:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
    **args)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
    payload)
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
    pass
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
    function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
    e, sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792, in terminate_instance
    do_terminate_instance(instance, bdms)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784, in do_terminate_instance
    reservations=reservations)
  File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
    rv = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757, in _delete_instance
    user_id=user_id)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729, in _delete_instance
    self._shutdown_instance(context, db_inst, bdms)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639, in _shutdown_instance
    network_info = self._get_instance_nw_info(context, instance)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876, in _get_instance_nw_info
    instance)
  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 455, in get_instance_nw_info
    result = self._get_instance_nw_info(context, instance, networks)  
  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 463, in _get_instance_nw_info
    nw_info = self._build_network_info_model(context, instance, networks)
  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1009, in _build_network_info_model
    subnets)
  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 962, in _nw_info_build_network
    label=network_name,
UnboundLocalError: local variable 'network_name' referenced before assignment


The reason is following code :

    def _nw_info_build_network(self, port, networks, subnets):
        # NOTE(danms): This loop can't fail to find a network since we
        # filtered ports to only the ones matching networks in our parent
        for net in networks:
            if port['network_id'] == net['id']:
                network_name = net['name']
                break

(if no net found network_name become undefined).

Following patch should allow instance deletion in case of networking
problems:

diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
index a41924d..8a44f99 100644
--- a/nova/network/neutronv2/api.py
+++ b/nova/network/neutronv2/api.py
@@ -939,6 +939,8 @@ class API(base.Base):
             if port['network_id'] == net['id']:
                 network_name = net['name']
                 break
+        else:
+            network_name = ""

         bridge = None
         ovs_interfaceid = None

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "Fix deletion of instances with broken networking"
   https://bugs.launchpad.net/bugs/1271958/+attachment/3955123/+files/nova-compute-fix-broken-net-instance-deletion.patch

** Description changed:

  If user was capable to create broken network configuration, instance
  become undeletable.  Reason why user can create broken networking is
  under investigation (current hypothesis: if network (neutron) created in
  one tennant and instance in other, and user is admin in both tenants, it
  cause broken configuration).
  
  But such instance deletetion cause trace on nova-compute:
  
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     **args)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     payload)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     pass
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792, in terminate_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     do_terminate_instance(instance, bdms)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784, in do_terminate_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     reservations=reservations)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     rv = f(*args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757, in _delete_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     user_id=user_id)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729, in _delete_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, db_inst, bdms)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639, in _shutdown_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876, in _get_instance_nw_info
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     instance)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 455, in get_instance_nw_info
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 463, in _get_instance_nw_info
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1009, in _build_network_info_model
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     subnets)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 962, in _nw_info_build_network
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp     label=network_name,
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp UnboundLocalError: local variable 'network_name' referenced before assignment
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
+ Traceback (most recent call last):
+   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
+     **args)
+   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
+     result = getattr(proxyobj, method)(ctxt, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
+     return function(self, context, *args, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
+     payload)
+   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
+     return f(self, context, *args, **kw)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
+     pass
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
+     return function(self, context, *args, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
+     function(self, context, *args, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
+     e, sys.exc_info())
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
+     return function(self, context, *args, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792, in terminate_instance
+     do_terminate_instance(instance, bdms)
+   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
+     return f(*args, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784, in do_terminate_instance
+     reservations=reservations)
+   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
+     rv = f(*args, **kwargs)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757, in _delete_instance
+     user_id=user_id)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729, in _delete_instance
+     self._shutdown_instance(context, db_inst, bdms)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639, in _shutdown_instance
+     network_info = self._get_instance_nw_info(context, instance)
+   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876, in _get_instance_nw_info
+     instance)
+   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 455, in get_instance_nw_info
+     result = self._get_instance_nw_info(context, instance, networks)  
+   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 463, in _get_instance_nw_info
+     nw_info = self._build_network_info_model(context, instance, networks)
+   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1009, in _build_network_info_model
+     subnets)
+   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 962, in _nw_info_build_network
+     label=network_name,
+ UnboundLocalError: local variable 'network_name' referenced before assignment
  
  
- The reason is following code:
+ The reason is following code :
  
-     def _nw_info_build_network(self, port, networks, subnets):
-         # NOTE(danms): This loop can't fail to find a network since we
-         # filtered ports to only the ones matching networks in our parent
-         for net in networks:
-             if port['network_id'] == net['id']:
-                 network_name = net['name']
-                 break
+     def _nw_info_build_network(self, port, networks, subnets):
+         # NOTE(danms): This loop can't fail to find a network since we
+         # filtered ports to only the ones matching networks in our parent
+         for net in networks:
+             if port['network_id'] == net['id']:
+                 network_name = net['name']
+                 break
  
  (if no net found network_name become undefined).
  
  Following patch should allow instance deletion in case of networking
  problems:
  
  diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
  index a41924d..8a44f99 100644
  --- a/nova/network/neutronv2/api.py
  +++ b/nova/network/neutronv2/api.py
  @@ -939,6 +939,8 @@ class API(base.Base):
-              if port['network_id'] == net['id']:
-                  network_name = net['name']
-                  break
+              if port['network_id'] == net['id']:
+                  network_name = net['name']
+                  break
  +        else:
  +            network_name = ""
-  
-          bridge = None
-          ovs_interfaceid = None
+ 
+          bridge = None
+          ovs_interfaceid = None

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1271958

Title:
  nova compute fail to remove instance with port if network is broken

Status in OpenStack Compute (Nova):
  New

Bug description:
  If user was capable to create broken network configuration, instance
  become undeletable.  Reason why user can create broken networking is
  under investigation (current hypothesis: if network (neutron) created
  in one tennant and instance in other, and user is admin in both
  tenants, it cause broken configuration).

  But such instance deletetion cause trace on nova-compute:

  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
      **args)
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
      result = getattr(proxyobj, method)(ctxt, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
      payload)
    File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
      return f(self, context, *args, **kw)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
      pass
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
      function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
      e, sys.exc_info())
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792, in terminate_instance
      do_terminate_instance(instance, bdms)
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784, in do_terminate_instance
      reservations=reservations)
    File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
      rv = f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757, in _delete_instance
      user_id=user_id)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729, in _delete_instance
      self._shutdown_instance(context, db_inst, bdms)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639, in _shutdown_instance
      network_info = self._get_instance_nw_info(context, instance)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876, in _get_instance_nw_info
      instance)
    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 455, in get_instance_nw_info
      result = self._get_instance_nw_info(context, instance, networks)  
    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 463, in _get_instance_nw_info
      nw_info = self._build_network_info_model(context, instance, networks)
    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1009, in _build_network_info_model
      subnets)
    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 962, in _nw_info_build_network
      label=network_name,
  UnboundLocalError: local variable 'network_name' referenced before assignment

  
  The reason is following code :

      def _nw_info_build_network(self, port, networks, subnets):
          # NOTE(danms): This loop can't fail to find a network since we
          # filtered ports to only the ones matching networks in our parent
          for net in networks:
              if port['network_id'] == net['id']:
                  network_name = net['name']
                  break

  (if no net found network_name become undefined).

  Following patch should allow instance deletion in case of networking
  problems:

  diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
  index a41924d..8a44f99 100644
  --- a/nova/network/neutronv2/api.py
  +++ b/nova/network/neutronv2/api.py
  @@ -939,6 +939,8 @@ class API(base.Base):
               if port['network_id'] == net['id']:
                   network_name = net['name']
                   break
  +        else:
  +            network_name = ""

           bridge = None
           ovs_interfaceid = None

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1271958/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next