← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #89751

[Bug 1990809] [NEW] multinode setup, devstack scheduler fails to start after controller restart

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

In multinode devstack setup nova scheduler fails to start after reboot


Steps to reproduce
==================

1 - deploy multinode devstack
    https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

2 - Verify all compute nodes are listed and setup is working as expected
    $ openstack compute service list
    
    create vm, assign floating IP and access VM

3 - Restart compute nodes, and controller node
    $ sudo init 6

4 - Once controller and all other nodes are rebooted, check whether all nova services are running
    $ openstack compute service list
  
    $ sudo systemctl status devstack@n-*


Expected result
===============
$ sudo systemctl status devstack@n-*

All services should be running


$ openstack compute service list

openstack cmds should run without a issue,


Actual result
=============
nova-schduler fails to start with error:
============================================
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova     self._init_plugins(extensions)
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova 
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: INFO oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells because its interval is negative
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: WARNING stevedore.named [-] Could not load filter_scheduler
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: CRITICAL nova [-] Unhandled error: stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova Traceback (most recent call last):
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/bin/nova-scheduler", line 10, in <module>
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     sys.exit(main())
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     server = service.Service.create(binary='nova-scheduler',
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 252, in create
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     service_obj = cls(host, binary, topic, manager,
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 116, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/scheduler/manager.py", line 60, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self.driver = driver.DriverManager(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova 

============================================

$ openstack compute service list
HttpException: 500: Server Error for url: http://22.0.2.5/compute/v2.1/os-services, Internal Server Error

$ sudo systemctl status devstack@n-sch
● devstack@n-sch.service - Devstack devstack@n-sch.service
     Loaded: loaded (/etc/systemd/system/devstack@n-sch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-09-26 05:09:16 UTC; 3min 34s ago
    Process: 11226 ExecStart=/usr/local/bin/nova-scheduler --config-file /etc/nova/nova.conf (code=exited, status=1/FAILURE)
   Main PID: 11226 (code=exited, status=1/FAILURE)

Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova 
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service: Failed with result 'exit-code'.


Environment
===========
Devstack multinode setup

1 controller
2 computes

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova-sch logs"
   https://bugs.launchpad.net/bugs/1990809/+attachment/5618985/+files/sch.log

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1990809

Title:
  multinode setup, devstack scheduler fails to start after controller
  restart

Status in OpenStack Compute (nova):
  New

Bug description:
  In multinode devstack setup nova scheduler fails to start after reboot


  Steps to reproduce
  ==================

  1 - deploy multinode devstack
      https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

  2 - Verify all compute nodes are listed and setup is working as expected
      $ openstack compute service list
      
      create vm, assign floating IP and access VM

  3 - Restart compute nodes, and controller node
      $ sudo init 6

  4 - Once controller and all other nodes are rebooted, check whether all nova services are running
      $ openstack compute service list
    
      $ sudo systemctl status devstack@n-*


  Expected result
  ===============
  $ sudo systemctl status devstack@n-*

  All services should be running

  
  $ openstack compute service list

  openstack cmds should run without a issue,


  Actual result
  =============
  nova-schduler fails to start with error:
  ============================================
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova     self._init_plugins(extensions)
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova 
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: INFO oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells because its interval is negative
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: WARNING stevedore.named [-] Could not load filter_scheduler
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: CRITICAL nova [-] Unhandled error: stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova Traceback (most recent call last):
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/bin/nova-scheduler", line 10, in <module>
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     sys.exit(main())
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     server = service.Service.create(binary='nova-scheduler',
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 252, in create
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     service_obj = cls(host, binary, topic, manager,
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 116, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/scheduler/manager.py", line 60, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self.driver = driver.DriverManager(
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     super(DriverManager, self).__init__(
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self._init_plugins(extensions)
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova 

  ============================================

  $ openstack compute service list
  HttpException: 500: Server Error for url: http://22.0.2.5/compute/v2.1/os-services, Internal Server Error

  $ sudo systemctl status devstack@n-sch
  ● devstack@n-sch.service - Devstack devstack@n-sch.service
       Loaded: loaded (/etc/systemd/system/devstack@n-sch.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Mon 2022-09-26 05:09:16 UTC; 3min 34s ago
      Process: 11226 ExecStart=/usr/local/bin/nova-scheduler --config-file /etc/nova/nova.conf (code=exited, status=1/FAILURE)
     Main PID: 11226 (code=exited, status=1/FAILURE)

  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     super(DriverManager, self).__init__(
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self._init_plugins(extensions)
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova 
  Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service: Main process exited, code=exited, status=1/FAILURE
  Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service: Failed with result 'exit-code'.

  
  Environment
  ===========
  Devstack multinode setup

  1 controller
  2 computes

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1990809/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next