yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2012530] [NEW] nova-scheduler will crash at startup if placement is not up

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dan Smith <2012530@xxxxxxxxxxxxxxxxxx>
Date: Wed, 22 Mar 2023 15:58:41 -0000
Reply-to: Bug 2012530 <2012530@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

This is the same problem as https://bugs.launchpad.net/nova/+bug/1846820
but for scheduler. Because we initialize our placement client during
manager init, we will crash (and loop) on startup if keystone or
placement are down. Example trace:

Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova.scheduler.client.report [None req-edf51111-6f86-4910-a458-72decae8e451 None None] Failed to initialize placement client (is keystone available?): openstack.exceptions.NotSupported: The placement service for 192.168.122.154:RegionOne exists but does not have any supported versions.
Mar 22 15:54:39 jammy nova-scheduler[119746]: CRITICAL nova [None req-edf51111-6f86-4910-a458-72decae8e451 None None] Unhandled error: openstack.exceptions.NotSupported: The placement service for 192.168.122.154:RegionOne exists but does not have any supported versions.
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova Traceback (most recent call last):
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/usr/local/bin/nova-scheduler", line 10, in <module>
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     sys.exit(main())
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     server = service.Service.create(
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 252, in create
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     service_obj = cls(host, binary, topic, manager,
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 116, in __init__
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/manager.py", line 70, in __init__
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     self.placement_client = report.report_client_singleton()
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/client/report.py", line 91, in report_client_singleton
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     PLACEMENTCLIENT = SchedulerReportClient()
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/client/report.py", line 234, in __init__
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     self._client = self._create_client()
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/client/report.py", line 277, in _create_client
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     client = self._adapter or utils.get_sdk_adapter('placement')
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/utils.py", line 984, in get_sdk_adapter
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     return getattr(conn, service_type)
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/usr/local/lib/python3.10/dist-packages/openstack/service_description.py", line 87, in __get__
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     proxy = self._make_proxy(instance)
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/usr/local/lib/python3.10/dist-packages/openstack/service_description.py", line 266, in _make_proxy
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     raise exceptions.NotSupported(
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova openstack.exceptions.NotSupported: The placement service for 192.168.122.154:RegionOne exists but does not have any supported versions.
Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2012530

Title:
  nova-scheduler will crash at startup if placement is not up

Status in OpenStack Compute (nova):
  New

Bug description:
  This is the same problem as
  https://bugs.launchpad.net/nova/+bug/1846820 but for scheduler.
  Because we initialize our placement client during manager init, we
  will crash (and loop) on startup if keystone or placement are down.
  Example trace:

  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova.scheduler.client.report [None req-edf51111-6f86-4910-a458-72decae8e451 None None] Failed to initialize placement client (is keystone available?): openstack.exceptions.NotSupported: The placement service for 192.168.122.154:RegionOne exists but does not have any supported versions.
  Mar 22 15:54:39 jammy nova-scheduler[119746]: CRITICAL nova [None req-edf51111-6f86-4910-a458-72decae8e451 None None] Unhandled error: openstack.exceptions.NotSupported: The placement service for 192.168.122.154:RegionOne exists but does not have any supported versions.
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova Traceback (most recent call last):
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/usr/local/bin/nova-scheduler", line 10, in <module>
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     sys.exit(main())
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     server = service.Service.create(
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 252, in create
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     service_obj = cls(host, binary, topic, manager,
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 116, in __init__
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/manager.py", line 70, in __init__
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     self.placement_client = report.report_client_singleton()
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/client/report.py", line 91, in report_client_singleton
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     PLACEMENTCLIENT = SchedulerReportClient()
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/client/report.py", line 234, in __init__
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     self._client = self._create_client()
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/scheduler/client/report.py", line 277, in _create_client
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     client = self._adapter or utils.get_sdk_adapter('placement')
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/opt/stack/nova/nova/utils.py", line 984, in get_sdk_adapter
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     return getattr(conn, service_type)
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/usr/local/lib/python3.10/dist-packages/openstack/service_description.py", line 87, in __get__
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     proxy = self._make_proxy(instance)
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova   File "/usr/local/lib/python3.10/dist-packages/openstack/service_description.py", line 266, in _make_proxy
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova     raise exceptions.NotSupported(
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova openstack.exceptions.NotSupported: The placement service for 192.168.122.154:RegionOne exists but does not have any supported versions.
  Mar 22 15:54:39 jammy nova-scheduler[119746]: ERROR nova

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2012530/+subscriptions
Follow ups

[Bug 2012530] Re: nova-scheduler will crash at startup if placement is not up
From: OpenStack Infra, 2023-04-10