yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1285158] Re: deallocating network not updating database

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1285158@xxxxxxxxxxxxxxxxxx>
Date: Mon, 22 Sep 2014 13:09:35 -0000
Reply-to: Bug 1285158 <1285158@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/havana
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1285158

Title:
  deallocating network not updating database

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) havana series:
  New

Bug description:
  An instance that fails to spawn (say due to corrupt image download) on
  a compute node will call 'deallocate_for_instance' but does not update
  the database to remove the networking information.

  from compute manager log for instance failing to spawn:
  AUDIT nova.compute.manager [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Starting instance...
  ...
  AUDIT nova.compute.claims [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Claim successful
  ...
  DEBUG nova.network.neutronv2.api [-] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Successfully created port: 38b34c17-e228-4d79-9248-c642a42959a8 _create_port /usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py:172
  ...
  2014-02-11 13:20:25.257 24500 DEBUG nova.network.api [-] Updating cache with info: [VIF({'ovs_interfaceid': None, 'network': Network({'bridge': None, 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.2'})], 'version': 4, 'meta': {'dhcp_server': u'10.100.0.3'}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/16', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'10540146451709'}, 'id': u'09030eb3-bca3-4df4-a5f9-721b6bd5d599', 'label': u'private'}), 'devname': u'tap38b34c17-e2', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:22:7c:aa', 'type': u'other', 'id': u'38b34c17-e228-4d79-9248-c642a42959a8', 'qbg_params': None})] update_instance_cache_with_nw_info /usr/lib/python2.7/dist-packages/nova/network/api.py:72
  ...
  ERROR nova.compute.manager [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Instance failed to spawn
  TRACE nova.compute.manager [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] IOError: [Errno 32] Corrupt image download. Checksum was 65327d2b03e53805a3354233b09aee62 expected 82d98abd651173e8c3e74b02d811f8a1
  AUDIT nova.compute.manager [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Terminating instance
  DEBUG nova.compute.manager [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Deallocating network for instance _deallocate_network /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1518


  When the instance successfully spawns on another compute node
  AUDIT nova.compute.manager [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Starting instance...
  ...
  AUDIT nova.compute.claims [req-165b21e5-f727-4d2e-98f3-1a5f0039595e 10546644733724 10540146451709] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Claim successful
  ...
  DEBUG nova.network.neutronv2.api [-] [instance: 6b28052e-3488-404d-9626-bb42f51ae98f] Successfully created port: 59f61b81-3314-4bdd-b455-2611af6653c2 _create_port /usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py:172
  ...
  2014-02-11 13:21:20.491 63160 DEBUG nova.network.api [-] Updating cache with info: [VIF({'ovs_interfaceid': None, 'network': Network({'bridge': None, 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': u'fixed', 'floating_ips': [], 'address': u'10.100.0.2'})], 'version': 4, 'meta': {u'dhcp_server': u'10.100.0.3'}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/16', 'gateway': IP({'meta': {}, 'version': 4, 'type': u'gateway', 'address': u'10.100.0.1'})})], 'meta': {u'injected': False, u'tenant_id': u'10540146451709'}, 'id': u'09030eb3-bca3-4df4-a5f9-721b6bd5d599', 'label': u'private'}), 'devname': u'tap38b34c17-e2', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:22:7c:aa', 'type': u'other', 'id': u'38b34c17-e228-4d79-9248-c642a42959a8', 'qbg_params': None}), VIF({'ovs_interfaceid': None, 'network': Network({'bridge': None, 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.4'})], 'version': 4, 'meta': {'dhcp_server': u'10.100.0.3'}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/16', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'10540146451709'}, 'id': u'09030eb3-bca3-4df4-a5f9-721b6bd5d599', 'label': u'private'}), 'devname': u'tap59f61b81-33', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:51:5d:bf', 'type': u'other', 'id': u'59f61b81-3314-4bdd-b455-2611af6653c2', 'qbg_params': None})] update_instance_cache_with_nw_info /usr/lib/python2.7/dist-packages/nova/network/api.py:72

  So what we are seeing here is that the fixed IP address (10.100.0.2) allocated during the spawn of the instance
  on the first compute node still exists when the port is allocated on the second compute node. Even though there was a succesful call to deallocate?
  Leading to the allocation of multiple fixed IP's.

   - the association of a floating IP to the fixed IP fails with:

  WARNING nova.api.openstack.compute.contrib.floating_ips [req-994a5ce9-fc42-4328-9eac-a05096fa25c3 10546644733724 10540146451709] multiple fixed_ips exist, using the first: 10.100.0.2
  ERROR nova.api.openstack.compute.contrib.floating_ips [req-994a5ce9-fc42-4328-9eac-a05096fa25c3 10546644733724 10540146451709] Error. Unable to associate floating ip
  TRACE nova.api.openstack.compute.contrib.floating_ips Traceback (most recent call last):
  TRACE nova.api.openstack.compute.contrib.floating_ips   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 255, in _add_floating_ip
  TRACE nova.api.openstack.compute.contrib.floating_ips     fixed_address=fixed_address)
  TRACE nova.api.openstack.compute.contrib.floating_ips   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 50, in wrapper
  TRACE nova.api.openstack.compute.contrib.floating_ips     res = f(self, context, *args, **kwargs)
  nova.api.openstack.compute.contrib.floating_ips   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 649, in associate_floating_ip
  nova.api.openstack.compute.contrib.floating_ips     fixed_address)
  nova.api.openstack.compute.contrib.floating_ips   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 634, in _get_port_id_by_fixed_address
  TRACE nova.api.openstack.compute.contrib.floating_ips     raise exception.FixedIpNotFoundForAddress(address=address)
  TRACE nova.api.openstack.compute.contrib.floating_ips FixedIpNotFoundForAddress: Fixed ip not found for address 10.100.0.2.

  Looking through the code, it seems that the call to
  _deallocate_network only removes the network allocation from the
  neutron side. It does not then update the database to reflect the
  deallocation.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1285158/+subscriptions
References

[Bug 1285158] [NEW] deallocating network not updating database
From: moorryan, 2014-02-26