yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1990809] Re: multinode setup, devstack scheduler fails to start after controller restart

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <1990809@xxxxxxxxxxxxxxxxxx>
Date: Tue, 27 Sep 2022 15:28:12 -0000
Reply-to: Bug 1990809 <1990809@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Looks to me not a Nova issue, maybe just a devstack issue or a
configuration problem. Moving it then to devstack.

** Also affects: devstack
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1990809

Title:
  multinode setup, devstack scheduler fails to start after controller
  restart

Status in devstack:
  New
Status in OpenStack Compute (nova):
  Invalid

Bug description:
  In multinode devstack setup nova scheduler fails to start after reboot


  Steps to reproduce
  ==================

  1 - deploy multinode devstack
      https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

  2 - Verify all compute nodes are listed and setup is working as expected
      $ openstack compute service list
      
      create vm, assign floating IP and access VM

  3 - Restart compute nodes, and controller node
      $ sudo init 6

  4 - Once controller and all other nodes are rebooted, check whether all nova services are running
      $ openstack compute service list
    
      $ sudo systemctl status devstack@n-*


  Expected result
  ===============
  $ sudo systemctl status devstack@n-*

  All services should be running

  
  $ openstack compute service list

  openstack cmds should run without a issue,


  Actual result
  =============
  nova-schduler fails to start with error:
  ============================================
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova     self._init_plugins(extensions)
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova 
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: INFO oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells because its interval is negative
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: WARNING stevedore.named [-] Could not load filter_scheduler
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: CRITICAL nova [-] Unhandled error: stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova Traceback (most recent call last):
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/bin/nova-scheduler", line 10, in <module>
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     sys.exit(main())
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     server = service.Service.create(binary='nova-scheduler',
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 252, in create
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     service_obj = cls(host, binary, topic, manager,
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 116, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/opt/stack/nova/nova/scheduler/manager.py", line 60, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self.driver = driver.DriverManager(
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     super(DriverManager, self).__init__(
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self._init_plugins(extensions)
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova 

  ============================================

  $ openstack compute service list
  HttpException: 500: Server Error for url: http://22.0.2.5/compute/v2.1/os-services, Internal Server Error

  $ sudo systemctl status devstack@n-sch
  ● devstack@n-sch.service - Devstack devstack@n-sch.service
       Loaded: loaded (/etc/systemd/system/devstack@n-sch.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Mon 2022-09-26 05:09:16 UTC; 3min 34s ago
      Process: 11226 ExecStart=/usr/local/bin/nova-scheduler --config-file /etc/nova/nova.conf (code=exited, status=1/FAILURE)
     Main PID: 11226 (code=exited, status=1/FAILURE)

  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     super(DriverManager, self).__init__(
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     self._init_plugins(extensions)
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova     raise NoMatches('No %r driver found, looking for %r' %
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
  Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova 
  Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service: Main process exited, code=exited, status=1/FAILURE
  Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service: Failed with result 'exit-code'.

  
  Environment
  ===========
  Devstack multinode setup

  1 controller
  2 computes

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1990809/+subscriptions
References

[Bug 1990809] [NEW] multinode setup, devstack scheduler fails to start after controller restart
From: Amit Uniyal, 2022-09-26