← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1493714] [NEW] nova allows booting two instances with the same neutron port in parallel

 

Public bug reported:

It seems that to reproduce the problem we need a multi node deployment
with at least two nova-compute service.

To reproduce it do the following:
1) create a neutron port
2) boot two instances in parallel with that port
Sometimes both instances become ACTIVE in nova which is clearly wrong.


vagrant@controller:~/devstack$ neutron net-list
+--------------------------------------+---------+----------------------------------------------------------+
| id                                   | name    | subnets                                                  |
+--------------------------------------+---------+----------------------------------------------------------+
| fc257a00-d3bf-47e6-b91f-e2cef985c414 | public  | a610715c-614f-492c-8810-f51e03c5d383 2001:db8::/64       |
|                                      |         | a923871f-90ad-4354-935d-db24861a5890 172.24.4.0/24       |
| 7a057b12-0e69-4c31-859e-098263abeeba | private | 04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe fda4:10b7:acaa::/64 |
|                                      |         | ee70023c-f273-471a-8b84-cb25bb64fcf9 10.0.0.0/24         |
+--------------------------------------+---------+----------------------------------------------------------+
vagrant@controller:~/devstack$ neutron port-create 7a057b12-0e69-4c31-859e-098263abeeba
Created a new port:
+-----------------------+-------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                       |
+-----------------------+-------------------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                                        |
| allowed_address_pairs |                                                                                                             |
| binding:host_id       |                                                                                                             |
| binding:profile       | {}                                                                                                          |
| binding:vif_details   | {}                                                                                                          |
| binding:vif_type      | unbound                                                                                                     |
| binding:vnic_type     | normal                                                                                                      |
| device_id             |                                                                                                             |
| device_owner          |                                                                                                             |
| fixed_ips             | {"subnet_id": "ee70023c-f273-471a-8b84-cb25bb64fcf9", "ip_address": "10.0.0.4"}                             |
|                       | {"subnet_id": "04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe", "ip_address": "fda4:10b7:acaa:0:f816:3eff:fe0c:285d"} |
| id                    | f2da8f78-8ae4-49f0-bca0-5820588d33ea                                                                        |
| mac_address           | fa:16:3e:0c:28:5d                                                                                           |
| name                  |                                                                                                             |
| network_id            | 7a057b12-0e69-4c31-859e-098263abeeba                                                                        |
| port_security_enabled | True                                                                                                        |
| security_groups       | 73853e74-a6c7-4b71-ba45-5b82b5e1ad81                                                                        |
| status                | DOWN                                                                                                        |
| tenant_id             | 16f8c1dbfa2d472da0d1335b8a70aee0                                                                            |
+-----------------------+-------------------------------------------------------------------------------------------------------------+
vagrant@controller:~/devstack$ nova boot --image cirros-0.3.4-x86_64-uec --flavor 42 --nic port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm1 & nova boot --image cirros-0.3.4-x86_64-uec --flavor 42 --nic port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm2 &
[1] 18785
[2] 18786

vagrant@controller:~/devstack$ nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| ID                                   | Name | Status | Task State | Power State | Networks                                               |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| 24e67346-18fe-4cfa-8b01-f017c915a111 | vm1  | ACTIVE | -          | Running     | private=fda4:10b7:acaa:0:f816:3eff:fe0c:285d, 10.0.0.4 |
| d90292a2-5b8f-4566-a621-defdf7aa0246 | vm2  | ACTIVE | -          | Running     |                                                        |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+

Based on the code we have an atomicity problem on the neutron REST API:
At https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L611 nova will check if the device_id of the neutron port are empty or not. Then nova sets the current instance_uuid to the device_id later at https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L692

So it is possible that two nova-compute processes check the port status
before one of them sets the device_id, so as a result both nova-compute
will think that the port are free to use and the slower nova-compute
will overwrite the device_id of the port.

** Affects: nova
     Importance: Undecided
     Assignee: Balazs Gibizer (balazs-gibizer)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1493714

Title:
  nova allows booting two instances with the same neutron port in
  parallel

Status in OpenStack Compute (nova):
  New

Bug description:
  It seems that to reproduce the problem we need a multi node deployment
  with at least two nova-compute service.

  To reproduce it do the following:
  1) create a neutron port
  2) boot two instances in parallel with that port
  Sometimes both instances become ACTIVE in nova which is clearly wrong.

  
  vagrant@controller:~/devstack$ neutron net-list
  +--------------------------------------+---------+----------------------------------------------------------+
  | id                                   | name    | subnets                                                  |
  +--------------------------------------+---------+----------------------------------------------------------+
  | fc257a00-d3bf-47e6-b91f-e2cef985c414 | public  | a610715c-614f-492c-8810-f51e03c5d383 2001:db8::/64       |
  |                                      |         | a923871f-90ad-4354-935d-db24861a5890 172.24.4.0/24       |
  | 7a057b12-0e69-4c31-859e-098263abeeba | private | 04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe fda4:10b7:acaa::/64 |
  |                                      |         | ee70023c-f273-471a-8b84-cb25bb64fcf9 10.0.0.0/24         |
  +--------------------------------------+---------+----------------------------------------------------------+
  vagrant@controller:~/devstack$ neutron port-create 7a057b12-0e69-4c31-859e-098263abeeba
  Created a new port:
  +-----------------------+-------------------------------------------------------------------------------------------------------------+
  | Field                 | Value                                                                                                       |
  +-----------------------+-------------------------------------------------------------------------------------------------------------+
  | admin_state_up        | True                                                                                                        |
  | allowed_address_pairs |                                                                                                             |
  | binding:host_id       |                                                                                                             |
  | binding:profile       | {}                                                                                                          |
  | binding:vif_details   | {}                                                                                                          |
  | binding:vif_type      | unbound                                                                                                     |
  | binding:vnic_type     | normal                                                                                                      |
  | device_id             |                                                                                                             |
  | device_owner          |                                                                                                             |
  | fixed_ips             | {"subnet_id": "ee70023c-f273-471a-8b84-cb25bb64fcf9", "ip_address": "10.0.0.4"}                             |
  |                       | {"subnet_id": "04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe", "ip_address": "fda4:10b7:acaa:0:f816:3eff:fe0c:285d"} |
  | id                    | f2da8f78-8ae4-49f0-bca0-5820588d33ea                                                                        |
  | mac_address           | fa:16:3e:0c:28:5d                                                                                           |
  | name                  |                                                                                                             |
  | network_id            | 7a057b12-0e69-4c31-859e-098263abeeba                                                                        |
  | port_security_enabled | True                                                                                                        |
  | security_groups       | 73853e74-a6c7-4b71-ba45-5b82b5e1ad81                                                                        |
  | status                | DOWN                                                                                                        |
  | tenant_id             | 16f8c1dbfa2d472da0d1335b8a70aee0                                                                            |
  +-----------------------+-------------------------------------------------------------------------------------------------------------+
  vagrant@controller:~/devstack$ nova boot --image cirros-0.3.4-x86_64-uec --flavor 42 --nic port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm1 & nova boot --image cirros-0.3.4-x86_64-uec --flavor 42 --nic port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm2 &
  [1] 18785
  [2] 18786

  vagrant@controller:~/devstack$ nova list
  +--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
  | ID                                   | Name | Status | Task State | Power State | Networks                                               |
  +--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
  | 24e67346-18fe-4cfa-8b01-f017c915a111 | vm1  | ACTIVE | -          | Running     | private=fda4:10b7:acaa:0:f816:3eff:fe0c:285d, 10.0.0.4 |
  | d90292a2-5b8f-4566-a621-defdf7aa0246 | vm2  | ACTIVE | -          | Running     |                                                        |
  +--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+

  Based on the code we have an atomicity problem on the neutron REST API:
  At https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L611 nova will check if the device_id of the neutron port are empty or not. Then nova sets the current instance_uuid to the device_id later at https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L692

  So it is possible that two nova-compute processes check the port
  status before one of them sets the device_id, so as a result both
  nova-compute will think that the port are free to use and the slower
  nova-compute will overwrite the device_id of the port.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1493714/+subscriptions


Follow ups