yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1873387] [NEW] Unable to delete instance because cyborg.get_client() failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Brin Zhang <1873387@xxxxxxxxxxxxxxxxxx>
Date: Fri, 17 Apr 2020 04:22:59 -0000
Reply-to: Bug 1873387 <1873387@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When our cloud platform is not deployed or the Cyborg service is removed
for some reason, we should be able to delete the instance correctly.

Today, if there is no Cyborg service, or depolyed before and removed
later, if we want to cleanup the instance with 'accel:device_profile' in
it's flavor when the server build failed, or the user want to delete an
older instance contains 'accel:device_profile',  that will be terminated
and deleted because we did not handle the exception at [1].

[1]
https://opendev.org/openstack/nova/src/branch/master/nova/compute/utils.py#L1559

Part of more details as below:

2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
on2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     return self.request(url, 'GET', 
**kwargs)
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
on2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     return self.session.request(url,
 method, **kwargs)
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
on2.7/site-packages/keystoneauth1/session.py", line 706, in request
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     **endpoint_filter)
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
on2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     return auth.get_endpoint(self, *
*kwargs)
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "build/bdist.linux-x86_64/egg
/nova/context.py", line 79, in get_endpoint
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
on2.7/site-packages/keystoneauth1/access/service_catalog.py", line 400, in url_for
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     endpoint_id=endpoint_id).url
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
on2.7/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     raise exceptions.EndpointNotFoun
d(msg)
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f] EndpointNotFound: ['internal', 'publ
ic'] endpoint for accelerator service in RegionTwo region not found
2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]

** Affects: nova
     Importance: Medium
     Assignee: Brin Zhang (zhangbailin)
         Status: Confirmed


** Tags: cyborg

** Changed in: nova
   Importance: Undecided => Medium

** Changed in: nova
     Assignee: (unassigned) => Brin Zhang (zhangbailin)

** Tags added: cyborg

** Changed in: nova
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1873387

Title:
  Unable to delete instance because cyborg.get_client() failed

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  When our cloud platform is not deployed or the Cyborg service is
  removed for some reason, we should be able to delete the instance
  correctly.

  Today, if there is no Cyborg service, or depolyed before and removed
  later, if we want to cleanup the instance with 'accel:device_profile'
  in it's flavor when the server build failed, or the user want to
  delete an older instance contains 'accel:device_profile',  that will
  be terminated and deleted because we did not handle the exception at
  [1].

  [1]
  https://opendev.org/openstack/nova/src/branch/master/nova/compute/utils.py#L1559

  Part of more details as below:

  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
  on2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     return self.request(url, 'GET', 
  **kwargs)
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
  on2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     return self.session.request(url,
   method, **kwargs)
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
  on2.7/site-packages/keystoneauth1/session.py", line 706, in request
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     **endpoint_filter)
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
  on2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     return auth.get_endpoint(self, *
  *kwargs)
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "build/bdist.linux-x86_64/egg
  /nova/context.py", line 79, in get_endpoint
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
  on2.7/site-packages/keystoneauth1/access/service_catalog.py", line 400, in url_for
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     endpoint_id=endpoint_id).url
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]   File "/var/lib/kolla/venv/lib/pyth
  on2.7/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]     raise exceptions.EndpointNotFoun
  d(msg)
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f] EndpointNotFound: ['internal', 'publ
  ic'] endpoint for accelerator service in RegionTwo region not found
  2020-04-17 03:26:58.868 7 ERROR nova.compute.manager [instance: d378d6a4-2cf5-4076-8bc0-82544f25f34f]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1873387/+subscriptions
Follow ups

[Bug 1873387] Re: Unable to delete instance because cyborg.get_client() failed
From: Balazs Gibizer, 2020-04-17