yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1365606] Re: Network deallocation can fail if a network has been deleted

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Thu, 18 Dec 2014 19:58:53 -0000
Reply-to: Bug 1365606 <1365606@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1365606

Title:
  Network deallocation can fail if a network has been deleted

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  The call to get_instance_nw_info fails with an error if a network is
  deleted during the deallocation. Networks are not supposed to be able
  to be deleted if they have fixed ips in use, but there is a race where
  a network can be deleted while an allocation is still in process. This
  is make many times worse by the fact that get_instance_nw_info makes
  3*networks + 1 (db + rpc) calls. This should be converted to a) get
  everything in a single db request, b) handle networks not existing
  gracefully.

  The traceback from get_nw_info failing can look like:

  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 242, in deallocate_fixed_ip
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     address, instance=instance)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 931, in deallocate_fixed_ip
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     instance_uuid)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 395, in _do_trigger_security_group_members_refresh_for_instance
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     None, None)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 602, in get_instance_nw_info
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     rxtx_factor, host)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 625, in build_network_info_model
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     instance_host)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 701, in _get_subnets_from_network
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     network['project_id'], network['uuid'], vif.uuid)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/nova_ipam_lib.py", line 46, in get_subnets_by_net_id
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     n = network_obj.Network.get_by_uuid(context.elevated(), net_id)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 110, in wrapper
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     args, kwargs)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 425, in object_class_action
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     objver=objver, args=args, kwargs=kwargs)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     wait_for_reply=True, timeout=timeout)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher     raise result
  2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher NetworkNotFoundForUUID_Remote: Network could not be found for uuid 3af0a6e6-59f8-4d7e-90ec-b5b866f578f8

  or:

  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 242, in deallocate_fixed_ip
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     address, instance=instance)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 931, in deallocate_fixed_ip
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     instance_uuid)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 395, in _do_trigger_security_group_members_refresh_for_instance
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     None, None)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 598, in get_instance_nw_info
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     network = self._get_network_by_id(context, vif.network_id)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1445, in _get_network_by_id
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     project_only='allow_none')
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 110, in wrapper
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     args, kwargs)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 425, in object_class_action
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     objver=objver, args=args, kwargs=kwargs)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     wait_for_reply=True, timeout=timeout)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher     raise result
  2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher NetworkNotFound_Remote: Network 23 could not be found.

  depending on where in the call stack the race occurs.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1365606/+subscriptions
References

[Bug 1365606] [NEW] Network deallocation can fail if a network has been deleted
From: Vish Ishaya, 2014-09-04