yahoo-eng-team team mailing list archive

[Rodolfo Alonso <1926787@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Neutron quota request implementation can end in a DB lock status. The
quota is assigned per resource (port, network, security group, etc.) and
per project. When a request is done, a DB lock is set for this
(resource, project) tuple. This lock in the DB engine to lock this tuple
in all workers of all API servers.

That implies there is a a bottleneck when a high number of requests
arrive to the API at the same time. If the number of requests exceeds
the number of resources processes, the DB locked transactions will
increase indefinitely. This can be seen in the DB executing:

  $ mysql -e "show processlist;" | egrep "reservations|quotausages"

The query used by Neutron to lock this (resource, project) tuple is:

    UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
      AND quotausages.resource = <resource_type>

An improved quota system should be implemented that allow parallel
resource request and avoids this DB lock status.

NOTE: please check [2][3]. "Neutron does not enforce quotas in such a
way that a quota violation like this could never occur". That means even
with this restrictive DB locking method, resource overcommit is
possible.

[1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150
[2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2
[3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661

** Affects: neutron
     Importance: Wishlist
         Status: New

** Changed in: neutron
   Importance: Undecided => Wishlist

** Description changed:

  Neutron quota request implementation can end in a DB lock status. The
  quota is assigned per resource (port, network, security group, etc.) and
  per project. When a request is done, a DB lock is set for this
  (resource, project) tuple. This lock in the DB engine to lock this tuple
  in all workers of all API servers.
  
  That implies there is a a bottleneck when a high number of requests
  arrive to the API at the same time. If the number of requests exceeds
  the number of resources processes, the DB locked transactions will
  increase indefinitely. This can be seen in the DB executing:
  
-   $ mysql -e "show processlist;" | egrep "reservations|quotausages"
- 
+   $ mysql -e "show processlist;" | egrep "reservations|quotausages"
  
  The query used by Neutron to lock this (resource, project) tuple is:
  
-     UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
-       AND quotausages.resource = <resource_type>
+     UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
+       AND quotausages.resource = <resource_type>
  
- 
- An improved quota system should be implemented that allow parallel resource request and avoids this DB lock status.
+ An improved quota system should be implemented that allow parallel
+ resource request and avoids this DB lock status.
  
  NOTE: please check [2][3]. "Neutron does not enforce quotas in such a
  way that a quota violation like this could never occur". That means even
  with this restrictive DB locking method, resource overcommit is
  possible.
  
- 
  [1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150
  [2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2
  [3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5
+ 
+ Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926787

Title:
  [DB] Neutron quota request implementation can end in a lock status

Status in neutron:
  New

Bug description:
  Neutron quota request implementation can end in a DB lock status. The
  quota is assigned per resource (port, network, security group, etc.)
  and per project. When a request is done, a DB lock is set for this
  (resource, project) tuple. This lock in the DB engine to lock this
  tuple in all workers of all API servers.

  That implies there is a a bottleneck when a high number of requests
  arrive to the API at the same time. If the number of requests exceeds
  the number of resources processes, the DB locked transactions will
  increase indefinitely. This can be seen in the DB executing:

    $ mysql -e "show processlist;" | egrep "reservations|quotausages"

  The query used by Neutron to lock this (resource, project) tuple is:

      UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
        AND quotausages.resource = <resource_type>

  An improved quota system should be implemented that allow parallel
  resource request and avoids this DB lock status.

  NOTE: please check [2][3]. "Neutron does not enforce quotas in such a
  way that a quota violation like this could never occur". That means
  even with this restrictive DB locking method, resource overcommit is
  possible.

  [1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150
  [2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2
  [3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5

  Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926787/+subscriptions