yahoo-eng-team team mailing list archive

[OpenStack Infra <1535551@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/285048
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6281fddbcb4c471b6b06e24d3faa2990e040f3d2
Submitter: Jenkins
Branch:    master

commit 6281fddbcb4c471b6b06e24d3faa2990e040f3d2
Author: Lujin Luo <luo.lujin@xxxxxxxxxxxxxx>
Date:   Tue Jun 21 14:23:33 2016 +0900

    Add a unique key to port_id in routerports table
    
    If multiple commands to add router interfaces to different routers
    by the same port are executed concurrently, then all the commands
    would show success.
    
    However, there are three issues:
    1. Only one router interface is actually added by the port
    2. Multiple router ports records are stored in routerports table
    3. The port table is updated multiple times and eventually the
    last-arrived command would truly take effect
    
    This patch adds a unique key to port_id in routerport table,
    so that only the first-arrived command will insert router port
    record and all later requests would raise exceptions.
    
    Besides, port.device_id and port.device_owner in port table
    needs to be updated again after routerport record is inserted.
    Otherwise, in race condition the port table will store the router
    information from last-arrived request. However, in routerport table,
    only the first-arrived request's router information is inserted.
    
    Change-Id: I15be35689ec59ac02ed34abe5862fa4580c8587c
    Closes-Bug: #1535551


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1535551

Title:
  One port can be added as multiple routers' interfaces if commands are
  executed at the same time

Status in neutron:
  Fix Released

Bug description:
  I have three controller nodes and the Neutron servers on these
  controllers are set behind Pacemaker and HAProxy to realize
  active/active HA using DevStack. MariaDB Galera cluster is used as my
  database backend.I am using the latest codes.

  If one port is added as multiple routers' interfaces, the expected result is that only API request is executed successfully and the port is associated to one router. Other API requests would recieve error message like
  PortInUseClient: Unable to complete operation on port d2c97788-61d7-489a-8b20-7a6e8e39a217 for network 496de8cf-4284-41d7-ad6b-7dd5f232dc21. Port already has an attached device 1b316d80-f5d8-4477-88df-54b376c4c8cd.

  Besides, in routerports database, only one record of port is allowed
  to exist. However, if we run two commands to add one port as two
  different routers' interfaces at the same time. Both of the commands
  would show execution succeed. The truth is two records that the port
  is associated to both routers are listed in routerports database.

  How to reproduce

  Step 1: Create two routers
  $ neutron router-create router-1
  $ neutron router-create router-2

  Step 2: Create an internal network
  $ neutron net-create net1

  Step 3: Add a subnet to the network
  $ neutron subnet-create --name subnet1 net1 192.166.100.0/24

  Step 4: Create a port in the network
  $ neutron port-create --name port1 net1

  Step 5: Add this port as two routers' interfaces at the same time
  On controller1:
  $ neutron router-interface-add router-1 port=port1
  on controller2:
  $ neutron router-interface-add router-2 port=port1

  Both commands would return success, as shown
  http://paste.openstack.org/show/483840/

  Step 6: Check port list on both routers
  The result is shown http://paste.openstack.org/show/483843/

  As we can see, only one router is successfully associated to the port

  Step 7: Check routerports database
  http://paste.openstack.org/show/483842/

  where '99276755-236a-44b7-bf97-b2234d97028b' is the port_id of the
  port we created in Step 4.

  To sum up, we have two issues here
  a) Only one API request is executed successfully, but both commands return success
  b) Routerports database is updated twice and we need to delete the older record.

  Related source codes is [1]

  [1]
  https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L535

  ------------------------Update on 2016/6/15----------------------------
  If we have operator-1 who is trying to add port_1 as router_1's interface while at the same time operator-2 is trying to add port_2 as router_2's interface. However, operator-2 miss-typed "port-2" to "port-1". Without this unique key, both commands will return Success. Operator-2 would hardly realize that he/she did a wrong command. What is worse is that, if router_1 truly added port_1 as interface, and router_2 did not. If we perform interface_delete command on router_2, port_1 is deleted and router_1 (which truly has the interface of port_1) will lose interface.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1535551/+subscriptions