yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1826472] [NEW] addFloatingIp action can fail if re-associating a floating IP to an instance in another cell

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 25 Apr 2019 22:11:16 -0000
Reply-to: Bug 1826472 <1826472@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Ran into this with the new nova-multi-cell job:

http://logs.openstack.org/22/655222/3/check/nova-multi-
cell/c1ba7e0/controller/logs/screen-n-api.txt.gz#_Apr_25_20_35_15_249786

Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips [None req-3b6da167-cd81-4f88-a27b-05876cf4e6f4 tempest-FloatingIPsAssociationTestJSON-110201909 tempest-FloatingIPsAssociationTestJSON-110201909] Unable to associate floating IP 172.24.5.15 to fixed IP 10.1.0.10 for instance be0ea845-25df-4528-8fcf-c27835f41636. Error: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.: nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last):
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 266, in _add_floating_ip
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     fixed_address=fixed_address)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/network/base_api.py", line 83, in wrapper
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     res = f(self, context, *args, **kwargs)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2401, in associate_floating_ip
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     orig_instance_uuid)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     result = fn(cls, context, *args, **kwargs)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/objects/instance.py", line 505, in get_by_uuid
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     use_slave=use_slave)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 213, in wrapper
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(*args, **kwargs)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/objects/instance.py", line 497, in _db_instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     columns_to_join=columns_to_join)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/api.py", line 758, in instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 171, in wrapper
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(*args, **kwargs)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 258, in wrapped
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(context, *args, **kwargs)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1827, in instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     columns_to_join=columns_to_join)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1836, in _instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     raise exception.InstanceNotFound(instance_id=uuid)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips 

Due to this old code:

https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400

https://review.opendev.org/#/c/33054/

The idea there is if you're associating a floating IP from one server to
another, we try to refresh the network info cache on the instance that
previously had the floating IP so it's not stale and you have 2
instances reporting they have the same floating IP until the cache is
cleaned up.

The problem is in a multi-cell environment the servers could be in
different cells and the context is targeted at the cell we're assigning
the floating IP *to* but will fail to lookup the instance in the other
cell that the floating IP is coming *from* (the original instance).

This results in a 400 error in the API (I'm surprised it's not returning
a 404 or 500 actually):

https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/api/openstack/compute/floating_ips.py#L282

** Affects: nova
     Importance: Medium
         Status: Triaged


** Tags: api cells neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1826472

Title:
  addFloatingIp action can fail if re-associating a floating IP to an
  instance in another cell

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Ran into this with the new nova-multi-cell job:

  http://logs.openstack.org/22/655222/3/check/nova-multi-
  cell/c1ba7e0/controller/logs/screen-n-api.txt.gz#_Apr_25_20_35_15_249786

  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips [None req-3b6da167-cd81-4f88-a27b-05876cf4e6f4 tempest-FloatingIPsAssociationTestJSON-110201909 tempest-FloatingIPsAssociationTestJSON-110201909] Unable to associate floating IP 172.24.5.15 to fixed IP 10.1.0.10 for instance be0ea845-25df-4528-8fcf-c27835f41636. Error: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.: nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last):
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 266, in _add_floating_ip
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     fixed_address=fixed_address)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/network/base_api.py", line 83, in wrapper
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     res = f(self, context, *args, **kwargs)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2401, in associate_floating_ip
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     orig_instance_uuid)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     result = fn(cls, context, *args, **kwargs)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/objects/instance.py", line 505, in get_by_uuid
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     use_slave=use_slave)
  Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 213, in wrapper
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(*args, **kwargs)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/objects/instance.py", line 497, in _db_instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     columns_to_join=columns_to_join)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/api.py", line 758, in instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 171, in wrapper
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(*args, **kwargs)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 258, in wrapped
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     return f(context, *args, **kwargs)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1827, in instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     columns_to_join=columns_to_join)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1836, in _instance_get_by_uuid
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips     raise exception.InstanceNotFound(instance_id=uuid)
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
  Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 devstack@n-api.service[19571]: ERROR nova.api.openstack.compute.floating_ips 

  Due to this old code:

  https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400

  https://review.opendev.org/#/c/33054/

  The idea there is if you're associating a floating IP from one server
  to another, we try to refresh the network info cache on the instance
  that previously had the floating IP so it's not stale and you have 2
  instances reporting they have the same floating IP until the cache is
  cleaned up.

  The problem is in a multi-cell environment the servers could be in
  different cells and the context is targeted at the cell we're
  assigning the floating IP *to* but will fail to lookup the instance in
  the other cell that the floating IP is coming *from* (the original
  instance).

  This results in a 400 error in the API (I'm surprised it's not
  returning a 404 or 500 actually):

  https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/api/openstack/compute/floating_ips.py#L282

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1826472/+subscriptions
Follow ups

[Bug 1826472] Re: addFloatingIp action can fail if re-associating a floating IP to an instance in another cell
From: OpenStack Infra, 2019-09-13