yahoo-eng-team team mailing list archive

[Frode Nordahl <1890432@xxxxxxxxxxxxxxxxxx>]

Adding upstream Neutron project to this LP.

The lock contention arises from the update of the metadata_port:
https://github.com/openstack/neutron/blob/24590a334fff0ed1cb513b0f496be965bc9309d4/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2111
https://github.com/openstack/neutron/blob/24590a334fff0ed1cb513b0f496be965bc9309d4/neutron/db/ipam_backend_mixin.py#L653-L680

Updating the fixed_ips field of the metadata_port will make Neutron
attempt to lock all the subnets involved regardless of
update_metadata_port only changing one subnet or all of them.

I wonder how the OVS driver dealt with this, as it would have the exact
same issue.

Perhaps the only option is for Neutron to gracefully ignore a
update_metadata_port failure at subnet creation and update the metadata
port at a later time in one of its maintenance jobs.

** Also affects: neutron
   Importance: Undecided
       Status: New

** Changed in: charm-neutron-api
       Status: Triaged => Invalid

** Changed in: charm-neutron-api
   Importance: High => Undecided

** Changed in: charm-neutron-api
     Assignee: Frode Nordahl (fnordahl) => (unassigned)

** Changed in: neutron
       Status: New => In Progress

** Changed in: neutron
     Assignee: (unassigned) => Frode Nordahl (fnordahl)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1890432

Title:
  Create subnet is failing under high load with OVN

Status in OpenStack neutron-api charm:
  Invalid
Status in neutron:
  In Progress

Bug description:
  Under a high concurrency level create subnet is starting to fail.
  (12-14% failure rate) The bundle is OVN / Ussuri.

  neutronclient.common.exceptions.Conflict: Unable to complete operation
  on subnet  This subnet is being modified by another concurrent
  operation.

  Stacktrace: https://pastebin.ubuntu.com/p/sQ5CqD6NyS/
  Rally task:

  {% set flavor_name = flavor_name or "m1.medium" %}
  {% set image_name = image_name or "bionic-kvm" %}

  ---
    NeutronNetworks.create_and_delete_subnets:
      -
        args:
          network_create_args: {}
          subnet_create_args: {}
          subnet_cidr_start: "1.1.0.0/30"
          subnets_per_network: 2
        runner:
          type: "constant"
          times: 100
          concurrency: 10
        context:
          network: {}
          users:
            tenants: 30
            users_per_tenant: 1
          quotas:
            neutron:
              network: -1
              subnet: -1

  Concurrency level set to 1 instead of 10 is not triggering the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-api/+bug/1890432/+subscriptions