yahoo-eng-team team mailing list archive

[OpenStack Infra <2009055@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/881943
Committed: https://opendev.org/openstack/neutron/commit/c0af5b3b5ea89d3147adf1054625f29d5b01b309
Submitter: "Zuul (22348)"
Branch:    master

commit c0af5b3b5ea89d3147adf1054625f29d5b01b309
Author: Felix Huettner <felix.huettner@mail.schwarz>
Date:   Wed Mar 1 16:14:18 2023 +0100

    Reduce lock contention on subnets
    
    in [1] a lock was introduced with the goal of preventing subnets from
    being deleted while ports are being created in them in parallel.
    This was acheived by aquiring an exclusive lock on the row of the
    subnet in the Subnet table when adding/modifying a port or deleting
    the subnet.
    
    However as this was a exclusive lock it also prevented concurrent port
    modifications on the same subnet from happening. This can cause
    performance issues on environment with large shared subnets (e.g. a
    large external subnet).
    
    To reduce the lock contention for this case we split the lock in two
    parts:
    
    * For normal port operations we will aquire a shared lock on the
      row of the subnet. This allows multiple such operations to happen in
      parallel.
    * For deleting a subnet we will aquire an exclusive lock on the row of
      the subnet. This lock can not be aquired when there is any shared
      lock currently on the row.
    
    With this we maintain the same locking level as before, but reduce the
    amount of lock contention happening and thereby improve throughput.
    
    The performance improvement can be measured using rally test [2].
    (improving from 21 to 18 seconds).
    Alternatively it can be tested using 250 parallel curl calls to create a
    port in the same network. This improves from 113s to 42s.
    
    [1]: https://review.opendev.org/c/openstack/neutron/+/713045
    [2]: https://github.com/openstack/rally-openstack/blob/master/samples/tasks/scenarios/neutron/create-and-delete-ports.json
    
    Closes-Bug: #2009055
    Change-Id: I31b1a9c2f986f59fee0da265acebbd88d2f8e4f8


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2009055

Title:
  Performance issue when creating lots of ports

Status in neutron:
  Fix Released

Bug description:
  When creating ~250 ports on a single subnet we currently take around
  140 seconds.

  This duration increased significantly with the introduction of a fix
  for https://bugs.launchpad.net/neutron/+bug/1865891 that exclusively
  locks a subnet when a port is created/updated in there.

  This was observed in devstack wih ovn as a backend.

  Reproducer:

  1. Create a network and a subnet
  2. Run
  ```
  for i in `seq 1 250`; do  curl -g http://10.1.0.129:9696/networking/v2.0/ports -H "Content-Type: application/json" -H "X-Auth-Token: sometoken" -X POST -d '{"port": {"admin_state_up": true, "name": "test", "network_id": "somenetwork"}}' & done
  time wait
  ```

  We see this causing issues on highly frequented subnets (e.g. the ones
  hosting our public ips) as the requests take long and die because of
  our 20 second database max_statement_time.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2009055/+subscriptions