yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1826472] Re: addFloatingIp action can fail if re-associating a floating IP to an instance in another cell

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1826472@xxxxxxxxxxxxxxxxxx>
Date: Fri, 13 Sep 2019 15:20:06 -0000
Reply-to: Bug 1826472 <1826472@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/656594
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=481cb5ce04e220c26e5772f4253d63e212adca45
Submitter: Zuul
Branch:    master

commit 481cb5ce04e220c26e5772f4253d63e212adca45
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Tue Apr 30 17:45:32 2019 -0400

    Find instance in another cell during floating IP re-association
    
    When associating a floating IP to instance A but it was already
    associated with instance B, we try to refresh the info cache on
    instance B. The problem is the context is targeted to the cell
    for instance A and instance B might be in another cell, so we'll
    get an InstanceNotFound error trying to lookup instance B.
    
    This change tries to find the instance in another cell using its
    instance mapping, and makes the code a bit more graceful if
    instance B is deleted.
    
    Change-Id: I71790afd0784d98050ccd7cc0e046321da249cbe
    Closes-Bug: #1826472


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1826472

Title:
  addFloatingIp action can fail if re-associating a floating IP to an
  instance in another cell

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Ran into this with the new nova-multi-cell job:

  http://logs.openstack.org/22/655222/3/check/nova-multi-
  cell/c1ba7e0/controller/logs/screen-n-api.txt.gz#_Apr_25_20_35_15_249786

  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips [None req-3b6da167-cd81-4f88-a27b-05876cf4e6f4 tempest-FloatingIPsAssociationTestJSON-110201909 tempest-FloatingIPsAssociationTestJSON-110201909] Unable to associate floating IP 172.24.5.15 to fixed IP 10.1.0.10 for instance be0ea845-25df-4528-8fcf-c27835f41636. Error: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.: nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last):
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 266, in _add_floating_ip
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     fixed_address=fixed_address)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/network/base_api.py", line 83, in wrapper
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     res = f(self, context, *args, **kwargs)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2401, in associate_floating_ip
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     orig_instance_uuid)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     result = fn(cls, context, *args, **kwargs)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/objects/instance.py", line 505, in get_by_uuid
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     use_slave=use_slave)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 213, in wrapper
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(*args, **kwargs)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/objects/instance.py", line 497, in _db_instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     columns_to_join=columns_to_join)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/api.py", line 758, in instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 171, in wrapper
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(*args, **kwargs)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 258, in wrapped
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(context, *args, **kwargs)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1827, in instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     columns_to_join=columns_to_join)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1836, in _instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     raise exception.InstanceNotFound(instance_id=uuid)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips 

  Due to this old code:

  https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400

  https://review.opendev.org/#/c/33054/

  The idea there is if you're associating a floating IP from one server
  to another, we try to refresh the network info cache on the instance
  that previously had the floating IP so it's not stale and you have 2
  instances reporting they have the same floating IP until the cache is
  cleaned up.

  The problem is in a multi-cell environment the servers could be in
  different cells and the context is targeted at the cell we're
  assigning the floating IP *to* but will fail to lookup the instance in
  the other cell that the floating IP is coming *from* (the original
  instance).

  This results in a 400 error in the API (I'm surprised it's not
  returning a 404 or 500 actually):

  https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/api/openstack/compute/floating_ips.py#L282

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1826472/+subscriptions
References

[Bug 1826472] [NEW] addFloatingIp action can fail if re-associating a floating IP to an instance in another cell
From: Matt Riedemann, 2019-04-25