yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1846820] [NEW] nova-conductor may crash during deploy due to haproxy-keystone 504

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Radosław Piliszek <1846820@xxxxxxxxxxxxxxxxxx>
Date: Fri, 04 Oct 2019 19:29:06 -0000
Reply-to: Bug 1846820 <1846820@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

keystone was busy (behind haproxy)

nova-conductor:
2019-10-04 15:39:17.103 6 CRITICAL nova [-] Unhandled error: GatewayTimeout: Gateway Timeout (HTTP 504)
2019-10-04 15:39:17.103 6 ERROR nova Traceback (most recent call last):
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/bin/nova-conductor", line 10, in <module>
2019-10-04 15:39:17.103 6 ERROR nova     sys.exit(main())
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cmd/conductor.py", line 44, in main
2019-10-04 15:39:17.103 6 ERROR nova     topic=rpcapi.RPC_TOPIC)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 257, in create
2019-10-04 15:39:17.103 6 ERROR nova     periodic_interval_max=periodic_interval_max)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 129, in __init__
2019-10-04 15:39:17.103 6 ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 117, in __init__
2019-10-04 15:39:17.103 6 ERROR nova     self.compute_task_mgr = ComputeTaskManager()
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 243, in __init__
2019-10-04 15:39:17.103 6 ERROR nova     self.report_client = report.SchedulerReportClient()
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 186, in __init__
2019-10-04 15:39:17.103 6 ERROR nova     self._client = self._create_client()
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 229, in _create_client
2019-10-04 15:39:17.103 6 ERROR nova     client = self._adapter or utils.get_sdk_adapter('placement')
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/utils.py", line 1039, in get_sdk_adapter
2019-10-04 15:39:17.103 6 ERROR nova     return getattr(conn, service_type)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack/service_description.py", line 92, in __get__
2019-10-04 15:39:17.103 6 ERROR nova     endpoint = proxy_mod.Proxy.get_endpoint(proxy)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 282, in get_endpoint
2019-10-04 15:39:17.103 6 ERROR nova     return self.session.get_endpoint(auth or self.auth, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1200, in get_endpoint
2019-10-04 15:39:17.103 6 ERROR nova     return auth.get_endpoint(self, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2019-10-04 15:39:17.103 6 ERROR nova     allow_version_hack=allow_version_hack, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2019-10-04 15:39:17.103 6 ERROR nova     service_catalog = self.get_access(session).service_catalog
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-10-04 15:39:17.103 6 ERROR nova     self.auth_ref = self.get_auth_ref(session)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2019-10-04 15:39:17.103 6 ERROR nova     return self._plugin.get_auth_ref(session, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
2019-10-04 15:39:17.103 6 ERROR nova     authenticated=False, log=False, **rkwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1106, in post
2019-10-04 15:39:17.103 6 ERROR nova     return self.request(url, 'POST', **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 943, in request
2019-10-04 15:39:17.103 6 ERROR nova     raise exceptions.from_response(resp, method, url)
2019-10-04 15:39:17.103 6 ERROR nova GatewayTimeout: Gateway Timeout (HTTP 504)
2019-10-04 15:39:17.103 6 ERROR nova

** Affects: kolla-ansible
     Importance: High
         Status: Triaged

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: keystone nova-conductor

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1846820

Title:
  nova-conductor may crash during deploy due to haproxy-keystone 504

Status in kolla-ansible:
  Triaged
Status in OpenStack Compute (nova):
  New

Bug description:
  keystone was busy (behind haproxy)

  nova-conductor:
  2019-10-04 15:39:17.103 6 CRITICAL nova [-] Unhandled error: GatewayTimeout: Gateway Timeout (HTTP 504)
  2019-10-04 15:39:17.103 6 ERROR nova Traceback (most recent call last):
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/bin/nova-conductor", line 10, in <module>
  2019-10-04 15:39:17.103 6 ERROR nova     sys.exit(main())
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cmd/conductor.py", line 44, in main
  2019-10-04 15:39:17.103 6 ERROR nova     topic=rpcapi.RPC_TOPIC)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 257, in create
  2019-10-04 15:39:17.103 6 ERROR nova     periodic_interval_max=periodic_interval_max)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 129, in __init__
  2019-10-04 15:39:17.103 6 ERROR nova     self.manager = manager_class(host=self.host, *args, **kwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 117, in __init__
  2019-10-04 15:39:17.103 6 ERROR nova     self.compute_task_mgr = ComputeTaskManager()
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 243, in __init__
  2019-10-04 15:39:17.103 6 ERROR nova     self.report_client = report.SchedulerReportClient()
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 186, in __init__
  2019-10-04 15:39:17.103 6 ERROR nova     self._client = self._create_client()
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 229, in _create_client
  2019-10-04 15:39:17.103 6 ERROR nova     client = self._adapter or utils.get_sdk_adapter('placement')
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/utils.py", line 1039, in get_sdk_adapter
  2019-10-04 15:39:17.103 6 ERROR nova     return getattr(conn, service_type)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack/service_description.py", line 92, in __get__
  2019-10-04 15:39:17.103 6 ERROR nova     endpoint = proxy_mod.Proxy.get_endpoint(proxy)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 282, in get_endpoint
  2019-10-04 15:39:17.103 6 ERROR nova     return self.session.get_endpoint(auth or self.auth, **kwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1200, in get_endpoint
  2019-10-04 15:39:17.103 6 ERROR nova     return auth.get_endpoint(self, **kwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
  2019-10-04 15:39:17.103 6 ERROR nova     allow_version_hack=allow_version_hack, **kwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
  2019-10-04 15:39:17.103 6 ERROR nova     service_catalog = self.get_access(session).service_catalog
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
  2019-10-04 15:39:17.103 6 ERROR nova     self.auth_ref = self.get_auth_ref(session)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
  2019-10-04 15:39:17.103 6 ERROR nova     return self._plugin.get_auth_ref(session, **kwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
  2019-10-04 15:39:17.103 6 ERROR nova     authenticated=False, log=False, **rkwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1106, in post
  2019-10-04 15:39:17.103 6 ERROR nova     return self.request(url, 'POST', **kwargs)
  2019-10-04 15:39:17.103 6 ERROR nova   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 943, in request
  2019-10-04 15:39:17.103 6 ERROR nova     raise exceptions.from_response(resp, method, url)
  2019-10-04 15:39:17.103 6 ERROR nova GatewayTimeout: Gateway Timeout (HTTP 504)
  2019-10-04 15:39:17.103 6 ERROR nova

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1846820/+subscriptions
Follow ups

[Bug 1846820] Fix included in openstack/nova xena-eom
From: OpenStack Infra, 2024-03-05
[Bug 1846820] Fix included in openstack/nova 25.1.1
From: OpenStack Infra, 2023-04-27
[Bug 1846820] Re: nova-conductor may crash during deploy due to haproxy-keystone 504
From: Maksim Malchuk, 2023-04-18
[Bug 1846820] Re: nova-conductor may crash during deploy due to haproxy-keystone 504
From: Radosław Piliszek, 2022-09-23
[Bug 1846820] Re: nova-conductor may crash during deploy due to haproxy-keystone 504
From: melanie witt, 2022-09-22
[Bug 1846820] Related fix merged to nova (master)
From: OpenStack Infra, 2022-08-19