yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Public bug reported:

This is split out from bug 1807044 where nova-api sometimes takes more
than 60 seconds to start on slow CI nodes which causes devstack to
timeout and fail.

Specifically, every nova.compute.api.API constructs a
SchedulerReportClient, which grabs an in-memory lock per API worker
during init:

Dec 05 20:14:27.694593 ubuntu-xenial-ovh-bhs1-0000959981
devstack@n-api.service[23459]: DEBUG oslo_concurrency.lockutils [None
req-dfdfad07-2ff4-43ed-9f67-2acd59687e0c None None] Lock
"placement_client" released by
"nova.scheduler.client.report._create_client" :: held 0.006s
{{(pid=23462) inner /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/lockutils.py:339}}

We could probably be smarter about either making that a singleton in the
API or only init on first access since most of the API extensions aren't
going to even use that SchedulerReportClient.

There are at least 30 extensions in nova-api that create an API class,
and there are 2 workers in devstack jobs, and each API class constructs
that report client which has a lock during init, so it will have a
snowball effect on startup.

Furthermore, with this change:

https://review.openstack.org/#/c/615641/

The NOTE in the API is no longer true:

https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/compute/api.py#L256

So the API likely just needs to add it's own lazy-load behavior for that
client.

** Affects: nova
     Importance: Medium
     Assignee: Matt Riedemann (mriedem)
         Status: Triaged


** Tags: api performance

** Changed in: nova
     Assignee: (unassigned) => Matt Riedemann (mriedem)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1807219

Title:
  SchedulerReporClient init slows down nova-api startup

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  This is split out from bug 1807044 where nova-api sometimes takes more
  than 60 seconds to start on slow CI nodes which causes devstack to
  timeout and fail.

  Specifically, every nova.compute.api.API constructs a
  SchedulerReportClient, which grabs an in-memory lock per API worker
  during init:

  Dec 05 20:14:27.694593 ubuntu-xenial-ovh-bhs1-0000959981
  devstack@n-api.service[23459]: DEBUG oslo_concurrency.lockutils [None
  req-dfdfad07-2ff4-43ed-9f67-2acd59687e0c None None] Lock
  "placement_client" released by
  "nova.scheduler.client.report._create_client" :: held 0.006s
  {{(pid=23462) inner /usr/local/lib/python2.7/dist-
  packages/oslo_concurrency/lockutils.py:339}}

  We could probably be smarter about either making that a singleton in
  the API or only init on first access since most of the API extensions
  aren't going to even use that SchedulerReportClient.

  There are at least 30 extensions in nova-api that create an API class,
  and there are 2 workers in devstack jobs, and each API class
  constructs that report client which has a lock during init, so it will
  have a snowball effect on startup.

  Furthermore, with this change:

  https://review.openstack.org/#/c/615641/

  The NOTE in the API is no longer true:

  https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/compute/api.py#L256

  So the API likely just needs to add it's own lazy-load behavior for
  that client.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1807219/+subscriptions