yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1851917] [NEW] Nova Schedular and Nova Conductor restart on repeat

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Devesh <1851917@xxxxxxxxxxxxxxxxxx>
Date: Sat, 09 Nov 2019 09:51:44 -0000
Reply-to: Bug 1851917 <1851917@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

--------------------------Enviornment------------------------------------------------------------

Controller - IP A 10.1.1.100
Compute - IP A 10.1.1.110

Release - Openstack Stein

--------------------------------Error -----------------------------------------------------------
root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
● openstack-nova-api.service - OpenStack Nova API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
 Main PID: 1213 (nova-api)
   CGroup: /system.slice/openstack-nova-api.service
           ├─ 1213 /usr/bin/python2 /usr/bin/nova-api
           ├─17185 /usr/bin/python2 /usr/bin/nova-api
           └─17186 /usr/bin/python2 /usr/bin/nova-api

Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server...
Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately.
Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server.

● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled)
   Active: activating (start) since Sat 2019-11-09 16:42:25 +07; 15s ago
 Main PID: 17141 (nova-scheduler)
   CGroup: /system.slice/openstack-nova-scheduler.service
           └─17141 /usr/bin/python2 /usr/bin/nova-scheduler

Nov 09 16:42:25 overcloud-controller systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart.
Nov 09 16:42:25 overcloud-controller systemd[1]: Stopped OpenStack Nova Scheduler Server.
Nov 09 16:42:25 overcloud-controller systemd[1]: Starting OpenStack Nova Scheduler Server...

● openstack-nova-conductor.service - OpenStack Nova Conductor Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled)
   Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 9s ago
 Main PID: 17156 (nova-conductor)
   CGroup: /system.slice/openstack-nova-conductor.service
           └─17156 /usr/bin/python2 /usr/bin/nova-conductor

Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart.
Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server.
Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server...

● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
 Main PID: 1210 (nova-novncproxy)
   CGroup: /system.slice/openstack-nova-novncproxy.service
           └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/

Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server.
Hint: Some lines were ellipsized, use -l to show in full.

-----------------------------------Expected Result ----------------------------------------
root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
● openstack-nova-api.service - OpenStack Nova API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
 Main PID: 1213 (nova-api)
   CGroup: /system.slice/openstack-nova-api.service
           ├─ 1213 /usr/bin/python2 /usr/bin/nova-api
           ├─17193 /usr/bin/python2 /usr/bin/nova-api
           └─17194 /usr/bin/python2 /usr/bin/nova-api

Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server...
Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately.
Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server.

● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-09 16:42:42 +07; 1s ago
 Main PID: 17141 (nova-scheduler)
   CGroup: /system.slice/openstack-nova-scheduler.service

Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: first_packet = self.connection._read_packet()
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, ...packet
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: packet.check_error()
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in ..._error
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: err.raise_mysql_exception(self._data)
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise...eption
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: raise errorclass(errno, errval)
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.comp...local_
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Removing descriptor: 5
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Exception AssertionError: AssertionError('do not call blocking functions f...gnored

● openstack-nova-conductor.service - OpenStack Nova Conductor Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled)
   Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 11s ago
 Main PID: 17156 (nova-conductor)
   CGroup: /system.slice/openstack-nova-conductor.service
           └─17156 /usr/bin/python2 /usr/bin/nova-conductor

Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart.
Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server.
Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server...

● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
 Main PID: 1210 (nova-novncproxy)
   CGroup: /system.slice/openstack-nova-novncproxy.service
           └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/

Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server.
Hint: Some lines were ellipsized, use -l to show in full.

-----------------------------------NOVA SCHEDULAR
LOGS-------------------------------------

2019-11-09 16:44:29.120 17464 ERROR oslo_service.service Traceback (most recent call last):
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     service.start()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 166, in start
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     ctxt, self.host, self.binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result = fn(cls, context, *args, **kwargs)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 334, in get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     host, binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return IMPL.service_get_by_host_and_binary(context, host, binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return f(context, *args, **kwargs)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 508, in service_get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     filter_by(binary=binary).\
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2979, in first
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     ret = list(self[0:1])
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return list(res)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return self._execute_and_instances(context)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result = conn.execute(querycontext.statement, self._params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return meth(self, multiparams, params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return connection._execute_clauseelement(self, multiparams, params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     distilled_params,
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     e, statement, parameters, cursor, context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     util.raise_from_cause(newraise, exc_info)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     cursor, statement, parameters, context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     cursor.execute(statement, parameters)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result = self._query(query)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     conn.query(q)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in _read_query_result
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result.read()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     first_packet = self.connection._read_packet()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     packet.check_error()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     err.raise_mysql_exception(self._data)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     raise errorclass(errno, errval)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %(deleted_1)s AND services.host = %(host_1)s AND services.`binary` = %(binary_1)s \n LIMIT %(param_1)s'] [parameters: {u'host_1': 'overcloud-controller', u'param_1': 1, u'deleted_1': 0, u'binary_1': 'nova-scheduler'}] (Background on this error at: http://sqlalche.me/e/f405)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service 


--------------------------------------------Steps Taken--------------------------------------

Restart Rabbit mq

Created NOVA DB Again, New privileges provided.

Restarted nova services

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1851917

Title:
  Nova Schedular and Nova Conductor restart on repeat

Status in OpenStack Compute (nova):
  New

Bug description:
  --------------------------Enviornment------------------------------------------------------------

  Controller - IP A 10.1.1.100
  Compute - IP A 10.1.1.110

  Release - Openstack Stein

  --------------------------------Error -----------------------------------------------------------
  root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
  ● openstack-nova-api.service - OpenStack Nova API Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
   Main PID: 1213 (nova-api)
     CGroup: /system.slice/openstack-nova-api.service
             ├─ 1213 /usr/bin/python2 /usr/bin/nova-api
             ├─17185 /usr/bin/python2 /usr/bin/nova-api
             └─17186 /usr/bin/python2 /usr/bin/nova-api

  Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server...
  Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately.
  Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
  Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server.

  ● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled)
     Active: activating (start) since Sat 2019-11-09 16:42:25 +07; 15s ago
   Main PID: 17141 (nova-scheduler)
     CGroup: /system.slice/openstack-nova-scheduler.service
             └─17141 /usr/bin/python2 /usr/bin/nova-scheduler

  Nov 09 16:42:25 overcloud-controller systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart.
  Nov 09 16:42:25 overcloud-controller systemd[1]: Stopped OpenStack Nova Scheduler Server.
  Nov 09 16:42:25 overcloud-controller systemd[1]: Starting OpenStack Nova Scheduler Server...

  ● openstack-nova-conductor.service - OpenStack Nova Conductor Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled)
     Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 9s ago
   Main PID: 17156 (nova-conductor)
     CGroup: /system.slice/openstack-nova-conductor.service
             └─17156 /usr/bin/python2 /usr/bin/nova-conductor

  Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart.
  Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server.
  Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server...

  ● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
   Main PID: 1210 (nova-novncproxy)
     CGroup: /system.slice/openstack-nova-novncproxy.service
             └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/

  Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server.
  Hint: Some lines were ellipsized, use -l to show in full.

  -----------------------------------Expected Result ----------------------------------------
  root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
  ● openstack-nova-api.service - OpenStack Nova API Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
   Main PID: 1213 (nova-api)
     CGroup: /system.slice/openstack-nova-api.service
             ├─ 1213 /usr/bin/python2 /usr/bin/nova-api
             ├─17193 /usr/bin/python2 /usr/bin/nova-api
             └─17194 /usr/bin/python2 /usr/bin/nova-api

  Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server...
  Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately.
  Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
  Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server.

  ● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2019-11-09 16:42:42 +07; 1s ago
   Main PID: 17141 (nova-scheduler)
     CGroup: /system.slice/openstack-nova-scheduler.service

  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: first_packet = self.connection._read_packet()
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, ...packet
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: packet.check_error()
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in ..._error
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: err.raise_mysql_exception(self._data)
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise...eption
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: raise errorclass(errno, errval)
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.comp...local_
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Removing descriptor: 5
  Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Exception AssertionError: AssertionError('do not call blocking functions f...gnored

  ● openstack-nova-conductor.service - OpenStack Nova Conductor Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled)
     Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 11s ago
   Main PID: 17156 (nova-conductor)
     CGroup: /system.slice/openstack-nova-conductor.service
             └─17156 /usr/bin/python2 /usr/bin/nova-conductor

  Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart.
  Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server.
  Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server...

  ● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
   Main PID: 1210 (nova-novncproxy)
     CGroup: /system.slice/openstack-nova-novncproxy.service
             └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/

  Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server.
  Hint: Some lines were ellipsized, use -l to show in full.

  -----------------------------------NOVA SCHEDULAR
  LOGS-------------------------------------

  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service Traceback (most recent call last):
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     service.start()
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 166, in start
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     ctxt, self.host, self.binary)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result = fn(cls, context, *args, **kwargs)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 334, in get_by_host_and_binary
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     host, binary)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return IMPL.service_get_by_host_and_binary(context, host, binary)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return f(context, *args, **kwargs)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 508, in service_get_by_host_and_binary
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     filter_by(binary=binary).\
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2979, in first
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     ret = list(self[0:1])
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return list(res)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return self._execute_and_instances(context)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result = conn.execute(querycontext.statement, self._params)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return meth(self, multiparams, params)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     return connection._execute_clauseelement(self, multiparams, params)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     distilled_params,
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     e, statement, parameters, cursor, context
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     util.raise_from_cause(newraise, exc_info)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     cursor, statement, parameters, context
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     cursor.execute(statement, parameters)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result = self._query(query)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     conn.query(q)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in _read_query_result
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     result.read()
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     first_packet = self.connection._read_packet()
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     packet.check_error()
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     err.raise_mysql_exception(self._data)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service     raise errorclass(errno, errval)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %(deleted_1)s AND services.host = %(host_1)s AND services.`binary` = %(binary_1)s \n LIMIT %(param_1)s'] [parameters: {u'host_1': 'overcloud-controller', u'param_1': 1, u'deleted_1': 0, u'binary_1': 'nova-scheduler'}] (Background on this error at: http://sqlalche.me/e/f405)
  2019-11-09 16:44:29.120 17464 ERROR oslo_service.service 

  
  --------------------------------------------Steps Taken--------------------------------------

  Restart Rabbit mq

  Created NOVA DB Again, New privileges provided.

  Restarted nova services

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1851917/+subscriptions
Follow ups

[Bug 1851917] Re: Nova Schedular and Nova Conductor restart on repeat
From: Launchpad Bug Tracker, 2020-01-11