yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2081174] [NEW] Handle EndpointNotFound in nova notifier

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2081174@xxxxxxxxxxxxxxxxxx>
Date: Thu, 19 Sep 2024 11:41:52 -0000
Reply-to: Bug 2081174 <2081174@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

When nova endpoint for endpoint_type(public/internal/admin) is not
exist, following traceback is raised:-

2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'name': 'network-changed', 'server_uuid': '3c634df2-eb78-4f49-bb01-ae1c546411af', 'tag': 'feaa6ca6-7c33-4778-a33f-cd065112cc99'}]: keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in regionOne region not found
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova Traceback (most recent call last):
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/neutron/notifiers/nova.py", line 282, in send_events
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     response = novaclient.server_external_events.create(
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/v2/server_external_events.py", line 38, in create
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self._create('/os-server-external-events', body, 'events',
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/base.py", line 363, in _create
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp, body = self.api.client.post(url, body=body)
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 401, in post
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/client.py", line 69, in request
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp, body = super(SessionClient, self).request(url,
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 554, in request
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 257, in request
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 811, in request
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     base_url = self.get_endpoint(auth, allow=allow,
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1243, in get_endpoint
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return auth.get_endpoint(self, **kwargs)
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     endpoint_data = self.get_endpoint_data(
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 275, in get_endpoint_data
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     endpoint_data = service_catalog.endpoint_data_for(
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     raise exceptions.EndpointNotFound(msg)
2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in regionOne region not found


Even if the endpoint is created later after the notfication failure, nova notification keeps on failing until the token expires(default token_expiration is 3600 seconds) as service_catalog as part of token is not refreshed.

workaround is to restart neutron, it's good to handle the exception, log
it and invalidate the session as it's useless, and then when endpoint
actually exist new token with up to date service catalog get's created
and nova notification will succeed.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2081174

Title:
  Handle EndpointNotFound in nova notifier

Status in neutron:
  New

Bug description:
  When nova endpoint for endpoint_type(public/internal/admin) is not
  exist, following traceback is raised:-

  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'name': 'network-changed', 'server_uuid': '3c634df2-eb78-4f49-bb01-ae1c546411af', 'tag': 'feaa6ca6-7c33-4778-a33f-cd065112cc99'}]: keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in regionOne region not found
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova Traceback (most recent call last):
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/neutron/notifiers/nova.py", line 282, in send_events
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     response = novaclient.server_external_events.create(
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/v2/server_external_events.py", line 38, in create
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self._create('/os-server-external-events', body, 'events',
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/base.py", line 363, in _create
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp, body = self.api.client.post(url, body=body)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 401, in post
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/client.py", line 69, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp, body = super(SessionClient, self).request(url,
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 554, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 257, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 811, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     base_url = self.get_endpoint(auth, allow=allow,
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1243, in get_endpoint
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return auth.get_endpoint(self, **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     endpoint_data = self.get_endpoint_data(
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 275, in get_endpoint_data
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     endpoint_data = service_catalog.endpoint_data_for(
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     raise exceptions.EndpointNotFound(msg)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in regionOne region not found

  
  Even if the endpoint is created later after the notfication failure, nova notification keeps on failing until the token expires(default token_expiration is 3600 seconds) as service_catalog as part of token is not refreshed.

  workaround is to restart neutron, it's good to handle the exception,
  log it and invalidate the session as it's useless, and then when
  endpoint actually exist new token with up to date service catalog
  get's created and nova notification will succeed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2081174/+subscriptions
Follow ups

[Bug 2081174] Re: Handle EndpointNotFound in nova notifier
From: OpenStack Infra, 2024-09-19