yahoo-eng-team team mailing list archive

[Edward Hope-Morley <2121607@xxxxxxxxxxxxxxxxxx>]

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/yoga
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2121607

Title:
  [SRU] Nova-api showing latency after upgrading to Caracal

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive yoga series:
  New
Status in OpenStack Compute (nova):
  Confirmed
Status in python-attrs package in Ubuntu:
  In Progress
Status in python-attrs source package in Jammy:
  In Progress

Bug description:
  [ Impact ]

  * A resource leak bug in 21.2.0 of python-attrs (<https://github.com/python-attrs/attrs/issues/826>)
   caused performance issues when an application created many classes with the same name.

  * This caused a performance regression in nova through its dependency on python-jsonschema
    * The latency of API calls to nova increases over time

  [ Test Plan ]

   To reproduce the issue and confirm the fix:

  * Create a minimal openstack deployment, using e.g. devstack
  (<https://docs.openstack.org/devstack/latest/>).

  * Run a script which makes many calls to the nova API

   ---Script to call API---
   #!/bin/bash
   # --- Get a fresh token (requires openrc sourced first) ---
   get_token() {
       openstack token issue -f value -c id
   }

   OS_TOKEN=$(get_token)
   echo "Using token: $OS_TOKEN"

   # --- Send requests at 10 per second ---
   COUNT=0
   while true; do
       COUNT=$((COUNT + 1))
       STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
           -H "X-Auth-Token: $OS_TOKEN" \
           -H "Accept: application/json" \
           "$NOVA_URL/servers/detail")

       echo "$(date +'%F %T') [$COUNT] HTTP $STATUS"

       if [ "$STATUS" = "401" ]; then
           echo "[$(date)] Got 401 → refreshing token..."
           OS_TOKEN=$(get_token)
           continue # retry next loop with fresh token
       fi
       # sleep 0.1 # 0.1 sec → 10 per second
   done
   ---/Script to call API---

  * Simultaneously monitor the response time of API calls using the
  openstack CLI

   ---Script to monitor response time---
   #!/bin/bash

   # --- Again openrc must be sourced first ---
   os_quota() {
       openstack quota list --compute --timing --format=json
   }

   headers() {
       os_quota | grep -o -P '/(\w|\d){32}(/d)?' | tr '\n' ','
   }

   time_api() {
       os_quota |
           awk 'BEGIN { FS=","; ORS=","} /quota/ {print $2} END { printf "\n" }'
   }

   OUTPUT="api_calls_$(date +%Y-%m-%dT%H_%M_%S)"

   echo -n "# time," >>"$OUTPUT"
   headers >>"$OUTPUT"
   echo >>"$OUTPUT"

   i=0
   while true; do
       echo -n "$(date +'%F %T')," >>"$OUTPUT"
       time_api >>"$OUTPUT"
       echo -n "."
       if ((i > 80)); then
           i=0
           echo
       fi
       sleep 5
   done

  ---/Script to monitor response time---

  * Observe that the response time increases over time with python-attrs 21.2.0
    * I did this over a period of about 20 hrs.

  * Log in to the server hosting the nova API and upgrade python-attrs
  to 21.4.0

  * Repeat the above steps and observe that the response time remains
  stable over time.

  [ Where problems could occur ]

  * Other packages depend on python-attrs as a library and so could be
  affected.

  * From 21.2.0 to 21.4.0 there was one backward incompatible change change of which could potentailly cause issues:
    * <https://github.com/python-attrs/attrs/blob/21.4.0/CHANGELOG.rst>
      """
      When using @define, converters are now run by default when setting an attribute on an instance -- additionally to validators. I.e. the new default is on_setattr=[attrs.setters.convert, attrs.setters.validate].

      This is unfortunately a breaking change, but it was an oversight, impossible to raise a DeprecationWarning about, and it's better to fix it now while the APIs are very fresh with few users.
      """
    * Since there are no version pins of python-attrs excluding this version it is unlikely to break other packages.
    * It is possible that some users could be using the python3-attr package in their own code, but it is much more likely that
      for user code, attr would be installed via pip, either directly or within a virtualenv.

  * The version in noble is 23.2.0 which already includes all these
  changes and there doesn't seem to be any associated issues.

  [ Other Info ]

  * This SRU proposal bumps to 21.4.0 instead of 21.3.0 because 21.4.0 is a bug fix
    release for a regression in 21.3.0.

  * This is the first ubuntu specific modification to this package, but
  it is a fairly minor one and doesn't need to be carried forward to any
  other series.

  * A possible lighter weight alternative is to simply cherry pick the
  bugfix commit <https://github.com/python-
  attrs/attrs/commit/38580632ceac1cd6e477db71e1d190a4130beed4>

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2121607/+subscriptions