yahoo-eng-team team mailing list archive

[Johannes Kulik <1930406@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When doing volume-attachemnts, nova-api does an RPC-call (with a
long_rpc_timeout) into nova-compute to reserve_block_device_name(). This
takes a lock on the instance. If a volume-attachment was in progress,
which also takes the instance-lock, nova-api's RPC-call needs to wait.

Having RPC-calls in nova-api, that can take a long time, will block the
process handling the request. If a project does a lot of volume-
attachments (e.g. for a k8s workload > 10 attachments per instance),
this could starve out other users of nova-api by occupying all available
processes.

When running nova-api with eventlet, a small number of processes can
handle a lot of requests in parallel and some blocking rpc-calls don't
matter too much.

When switching to uWSGI, the number of processes would have to be
increased drastically to accommodate for that - unless it's possible to
map those requests to threads and use a high number of threads instead.

What's the recommended way to run nova-api on uWSGI to handle this? Low
number of processes with high number of threads to mimic eventlet?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1930406

Title:
  parallel volume-attachment requests might starve out nova-api for
  others

Status in OpenStack Compute (nova):
  New

Bug description:
  When doing volume-attachemnts, nova-api does an RPC-call (with a
  long_rpc_timeout) into nova-compute to reserve_block_device_name().
  This takes a lock on the instance. If a volume-attachment was in
  progress, which also takes the instance-lock, nova-api's RPC-call
  needs to wait.

  Having RPC-calls in nova-api, that can take a long time, will block
  the process handling the request. If a project does a lot of volume-
  attachments (e.g. for a k8s workload > 10 attachments per instance),
  this could starve out other users of nova-api by occupying all
  available processes.

  When running nova-api with eventlet, a small number of processes can
  handle a lot of requests in parallel and some blocking rpc-calls don't
  matter too much.

  When switching to uWSGI, the number of processes would have to be
  increased drastically to accommodate for that - unless it's possible
  to map those requests to threads and use a high number of threads
  instead.

  What's the recommended way to run nova-api on uWSGI to handle this?
  Low number of processes with high number of threads to mimic eventlet?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1930406/+subscriptions