yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1736946] Re: Conductor: fails to clean up networking resources

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 19 Dec 2017 01:55:02 -0000
Reply-to: Bug 1736946 <1736946@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This was regressed in newton: https://review.openstack.org/#/c/364005/

** Changed in: nova
   Importance: Undecided => High

** Also affects: nova/newton
   Importance: Undecided
       Status: New

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Also affects: nova/pike
   Importance: Undecided
       Status: New

** Tags added: cells

** Changed in: nova/newton
       Status: New => Confirmed

** Changed in: nova/pike
       Status: New => Confirmed

** Changed in: nova/newton
   Importance: Undecided => High

** Changed in: nova/pike
   Importance: Undecided => High

** Changed in: nova/ocata
   Importance: Undecided => High

** Changed in: nova/ocata
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1736946

Title:
  Conductor: fails to clean up networking resources due to
  _destroy_build_request CantStartEngineError

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) newton series:
  Confirmed
Status in OpenStack Compute (nova) ocata series:
  Confirmed
Status in OpenStack Compute (nova) pike series:
  Confirmed

Bug description:
  If libvirt fails to deploy instance - for example due to problematic
  vif type being passed. The conductor will fail to clean up resources.
  This fails with the exception below. This is due to the fact that the
  cell mapping was not invoked.

  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00mTraceback (most recent call last):
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    res = self.dispatcher.dispatch(message)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    return self._do_dispatch(endpoint, method, ctxt, args)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    result = func(ctxt, **new_args)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/opt/stack/nova/nova/conductor/manager.py", line 559, in build_instances
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    self._destroy_build_request(context, instance)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/opt/stack/nova/nova/conductor/manager.py", line 477, in _destroy_build_request
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    context, instance.uuid)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    result = fn(cls, context, *args, **kwargs)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/opt/stack/nova/nova/objects/build_request.py", line 176, in get_by_instance_uuid
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    db_req = cls._get_by_instance_uuid_from_db(context, instance_uuid)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 983, in wrapper
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    with self._transaction_scope(context):
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    return self.gen.next()
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1033, in _transaction_scope
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    context=context) as resource:
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    return self.gen.next()
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    bind=self.connection, mode=self.mode)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    self._start()
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    engine_args, maker_args)
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 511, in _setup_for_connection
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m    "No sql_connection parameter is established")
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00mCantStartEngineError: No sql_connection parameter is established
  Dec  7 09:12:50 utu1604template nova-conductor[22761]: ERROR oslo_messaging.rpc.server #033[01;35m#033[00m

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1736946/+subscriptions
References

[Bug 1736946] [NEW] Conductor: fails to clean up networking resources
From: Gary Kotton, 2017-12-07