yahoo-eng-team team mailing list archive

[OpenStack Infra <1781286@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/686047
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f1ad0b13e8f8296739f75b11e257a6fef2cc538c
Submitter: Zuul
Branch:    master

commit f1ad0b13e8f8296739f75b11e257a6fef2cc538c
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Tue Oct 1 16:52:46 2019 -0400

    Set Instance AZ from Selection AZ during build reschedule
    
    This builds on change Ia50c5f4dd2204f1cafa669097d1e744479c4d8c8
    to use the Selection.availability_zone value when rescheduling
    during initial server create so that the cell conductor does not
    have to make an up-call to the aggregates table in the API DB
    which will fail if the cell conductor is not configured to use
    the API DB.
    
    The functional test added in I62179d6b93ea1a23c4906477ee19b422bfcb72a2
    is updated to show the failure is gone and we get the AZ from the
    Selection object during the reschedule.
    
    For the case that the availability_zone field is not in the Selection
    object, test_build_reschedule_get_az_error still covers that.
    
    Change-Id: I1f1c25cb4de924a1d6c3a979b758efd736bdbff0
    Partial-Bug: #1781286


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1781286

Title:
  CantStartEngineError in cell conductor during reschedule -
  get_host_availability_zone up-call

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  In a stable/queens devstack environment with multiple PowerVM compute
  nodes, everytime I see this in devstack@n-cond-cell1.service logs:

  Jul 11 15:48:57 myhostname nova-conductor[3796]: DEBUG
  nova.conductor.manager [None req-af22375c-f920-4747-bd2f-0de80ee69465
  admin admin] Rescheduling: True {{(pid=4108) build_instances
  /opt/stack/nova/nova/conductor/manager.py:571}}

  it is shortly thereafter followed by:

  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server [None req-af22375c-f920-4747-bd2f-0de80ee69465 admin admin] Exception during message handling: CantStartEngineError: No sql_connection parameter is established
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/conductor/manager.py", line 652, in build_instances
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     host.service_host))
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/availability_zones.py", line 95, in get_host_availability_zone
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     key='availability_zone')
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     result = fn(cls, context, *args, **kwargs)
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/objects/aggregate.py", line 541, in get_by_host
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     _get_by_host_from_db(context, host, key=key)]
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 987, in wrapper
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     with self._transaction_scope(context):
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     return self.gen.next()
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1037, in _transaction_scope
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     context=context) as resource:
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     return self.gen.next()
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 640, in _session
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     bind=self.connection, mode=self.mode)
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 404, in _create_session
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     self._start()
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _start
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     engine_args, maker_args)
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server     "No sql_connection parameter is established")
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server CantStartEngineError: No sql_connection parameter is established
  Jul 11 15:48:57 myhostname nova-conductor[3796]: ERROR oslo_messaging.rpc.server

  The nova_cell1.conf does have [database]connection set:

  [database]
  connection = mysql+pymysql://root:mysql@127.0.0.1/nova_cell1?charset=utf8

  This may be related to https://bugs.launchpad.net/nova/+bug/1736946 ,
  though that is supposedly fixed in stable/queens and the trace is
  different, hence the new defect.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1781286/+subscriptions