yahoo-eng-team team mailing list archive

[OpenStack Infra <1942179@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/807335
Committed: https://opendev.org/openstack/neutron/commit/e610a5eb9e71aa2549fb11e2139370d227787da2
Submitter: "Zuul (22348)"
Branch:    master

commit e610a5eb9e71aa2549fb11e2139370d227787da2
Author: Slawek Kaplonski <skaplons@xxxxxxxxxx>
Date:   Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware
    
    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.
    
    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.
    
    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1942179

Title:
  neutron api worker leaks memory when processing requests to not
  existing controllers

Status in neutron:
  Fix Released
Status in OpenStack Security Advisory:
  Confirmed

Bug description:
  Authorized cloud user may do API requests to neutron to not existing
  endpoints, like e.g.:

  curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept:
  application/json" -H "User-Agent: openstacksdk/0.59.0
  keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth-
  Token: $token"

  and each such request will increase memory consumption of the neutron-
  api worker process.

  What I did was:

  * start neutron server with just one api worker (easier to calculate
  memory consumption but it would be the same leak in case of more
  workers too). Memory consumption was:

  sudo pmap 212436 | tail -n 1
   total           183736K

  * now run command like:

  $ i=1; while [ $i -lt 2000 ]; do echo "Request $i"; curl -g -i -X GET
  http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H
  "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-
  requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: $token" 2>1
  >/dev/null; i=$(( i+1 )); sleep 0.01; done

  * check memory consumption of the same api worker now:

  sudo pmap 212436 | tail -n 1
   total           457896K

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1942179/+subscriptions