yahoo-eng-team team mailing list archive

[OpenStack Infra <2081174@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/929920
Committed: https://opendev.org/openstack/neutron/commit/7d1a20ed4d458c6682a52679b71b6bc8dea20d07
Submitter: "Zuul (22348)"
Branch:    master

commit 7d1a20ed4d458c6682a52679b71b6bc8dea20d07
Author: yatinkarel <ykarel@xxxxxxxxxx>
Date:   Thu Sep 19 18:32:11 2024 +0530

    Handle EndpointNotFound in nova notifier
    
    Currently if the nova endpoint do not exist
    exception is raised. Even the endpoint gets created
    notification keeps on failing until the session
    expires.
    If the endpoint not exist the session is not useful
    so marking it as invalid, this will ensure if endpoint is
    created later the notification do not fail.
    
    Closes-Bug: #2081174
    Change-Id: I1f7fd1d1371ca0a3c4edb409cffd2177d44a1f23


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2081174

Title:
  Handle EndpointNotFound in nova notifier

Status in neutron:
  Fix Released

Bug description:
  When nova endpoint for endpoint_type(public/internal/admin) is not
  exist, following traceback is raised:-

  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'name': 'network-changed', 'server_uuid': '3c634df2-eb78-4f49-bb01-ae1c546411af', 'tag': 'feaa6ca6-7c33-4778-a33f-cd065112cc99'}]: keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in regionOne region not found
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova Traceback (most recent call last):
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/neutron/notifiers/nova.py", line 282, in send_events
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     response = novaclient.server_external_events.create(
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/v2/server_external_events.py", line 38, in create
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self._create('/os-server-external-events', body, 'events',
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/base.py", line 363, in _create
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp, body = self.api.client.post(url, body=body)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 401, in post
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/novaclient/client.py", line 69, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp, body = super(SessionClient, self).request(url,
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 554, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 257, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 811, in request
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     base_url = self.get_endpoint(auth, allow=allow,
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1243, in get_endpoint
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     return auth.get_endpoint(self, **kwargs)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     endpoint_data = self.get_endpoint_data(
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 275, in get_endpoint_data
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     endpoint_data = service_catalog.endpoint_data_for(
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova   File "/usr/lib/python3.9/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova     raise exceptions.EndpointNotFound(msg)
  2024-09-18 13:19:38.182 15 ERROR neutron.notifiers.nova keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in regionOne region not found

  
  Even if the endpoint is created later after the notfication failure, nova notification keeps on failing until the token expires(default token_expiration is 3600 seconds) as service_catalog as part of token is not refreshed.

  workaround is to restart neutron, it's good to handle the exception,
  log it and invalidate the session as it's useless, and then when
  endpoint actually exist new token with up to date service catalog
  get's created and nova notification will succeed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2081174/+subscriptions