yahoo-eng-team team mailing list archive

[OpenStack Infra <1926787@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/790060
Committed: https://opendev.org/openstack/neutron/commit/e135a8221dba3beac4047ca9351bdfe600fcf01a
Submitter: "Zuul (22348)"
Branch:    master

commit e135a8221dba3beac4047ca9351bdfe600fcf01a
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu May 6 12:48:05 2021 +0000

    New Quota driver ``DbQuotaNoLockDriver``
    
    This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
    per (resource, project_id) but retrieves the instant (resource,
    project_id) usage and the current (resource, project_id) reservations.
    If the requested number of resources fit the available quota, a new
    ``Reservation`` register is created with the amount of units requested.
    
    All those operations are done inside a DB transaction context. That
    means the amount of resources and reservations is guaranteed inside
    this transaction (depending on the DB backend isolation level defined)
    and the new reservation created will not clash with other DB transation.
    That will guarantee the number of resources and instant reservations
    never exceed the quota limits defined for this (resource, project_id).
    
    NOTES:
    - This change tries to be as unobtrusive as possible. The new driver
      uses the same ``DbQuotaDriver`` dabatase tables (except for
      ``QuotaUsage``) and the same Quota engine API, located in
      ``neutron.quota``. However, the Quota engine resources implements some
      particular API actions like "dirty", that are not used in the new
      driver.
    - The Pecan Quota enforcement hooks,
      ``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
      "resync", "mark_resources_dirty" or "set_resources_dirty", that has
      no meaning in the new driver.
    - The isolation between the Quota engine and the Pecan hook, and the
      driver itself is not clearly defined. A refactor of the Quota engine,
      Quota service, Quota drivers and a common API between the driver and
      the engine is needed.
    - If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
      ``TrackedResource`` will be joined in a single class. This resource
      class will have a count method (countable) or a hard dependency on a
      database table (tracked resource). The only difference will be the
      "count" method implementation.
    
    Closes-Bug: #1926787
    
    Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926787

Title:
  [DB] Neutron quota request implementation can end in a lock status

Status in neutron:
  Fix Released

Bug description:
  Neutron quota request implementation can end in a DB lock status. The
  quota is assigned per resource (port, network, security group, etc.)
  and per project. When a request is done, a DB lock is set for this
  (resource, project) tuple. This lock in the DB engine to lock this
  tuple in all workers of all API servers.

  That implies there is a a bottleneck when a high number of requests
  arrive to the API at the same time. If the number of requests exceeds
  the number of resources processes, the DB locked transactions will
  increase indefinitely. This can be seen in the DB executing:

    $ mysql -e "show processlist;" | egrep "reservations|quotausages"

  The query used by Neutron to lock this (resource, project) tuple is:

      UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
        AND quotausages.resource = <resource_type>

  An improved quota system should be implemented that allow parallel
  resource request and avoids this DB lock status.

  NOTE: please check [2][3]. "Neutron does not enforce quotas in such a
  way that a quota violation like this could never occur". That means
  even with this restrictive DB locking method, resource overcommit is
  possible.

  [1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150
  [2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2
  [3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5

  Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926787/+subscriptions