yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1716344] [NEW] Nova-API sometimes uses Keystone's public endpoint

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jens Offenbach <wolle5050@xxxxxx>
Date: Mon, 11 Sep 2017 08:40:40 -0000
Reply-to: Bug 1716344 <1716344@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have setup a fresh HA deployment of OpenStack Pike on Ubuntu 16.04. I
recognized in the logs that Nova sometimes fails during vm creation with
the following exception:

2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Traceback (most recent call last):
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/identity.py", line 42, in verify_project_id
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     raise_exc=False)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 845, in get
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     return self.request(url, 'GET', **kwargs)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     return wrapped(*args, **kwargs)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     resp = send(**kwargs)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 765, in _send_request
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     raise exceptions.SSLError(msg)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity

Keystone's public endpoint should only visible to external clients. All
internal OpenStack services should use the internalURL for
authentication purposes. I think my configuration is correct. The
"auth_url" point to Keystone's internal URL, whereas "auth_uri" points
to Keystone's public endpoint. The strange thing is, that sometimes
after a service restart, Nova uses the Keystone's internal URL and
sometimes the Keystone's public URL. I want to avoid https based
communication for the internal cloud services.

$ openstack endpoint list | grep keystone
| 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone     | identity        | True    | internal  | http://os-identity.mycompany.com:5000/v3                        |
| 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone     | identity        | True    | public    | https://os-cloud.mycompany.com:5000/v3                          |
| f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone     | identity        | True    | admin     | http://os-identity.mycompany.com:35357/v3                       |

################
nova.conf
################
...
[keystone_authtoken]
auth_type = password
auth_uri = http://os-cloud.mycompany.com:5000
auth_url = http://os-identity:35357
memcached_servers = os-memcache:11211
password = novapass
project_domain_name = default
project_name = service
user_domain_name = default
username = nova
...

Using the option "insecure = True" is a workaround to avoid that Nova
sometimes fails when the service uses Keystone's public https endpoint.

Can someone please have a look?

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  I have setup a fresh HA deployment of OpenStack Pike on Ubuntu 16.04. I
  recognized in the logs that Nova sometimes fails during vm creation with
  the following exception:
  
- 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.materna.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
+ 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Traceback (most recent call last):
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/identity.py", line 42, in verify_project_id
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     raise_exc=False)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 845, in get
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     return self.request(url, 'GET', **kwargs)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     return wrapped(*args, **kwargs)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     resp = send(**kwargs)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 765, in _send_request
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     raise exceptions.SSLError(msg)
- 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.materna.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
- 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity 
+ 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
+ 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity
  
  Keystone's public endpoint should only visible to external clients. All
  internal OpenStack services should use the internalURL for
  authentication purposes. I think my configuration is correct. The
  "auth_url" point to Keystone's internal URL, whereas "auth_uri" points
  to Keystone's public endpoint. The strange thing is, that sometimes
  after a service restart, Nova uses the Keystone's internal URL and
  sometimes the Keystone's public URL. I want to avoid https based
  communication for the internal cloud services.
  
  $ openstack endpoint list | grep keystone
- | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone     | identity        | True    | internal  | http://os-identity.materna.com:5000/v3                        |
- | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone     | identity        | True    | public    | https://os-cloud.materna.com:5000/v3                          |
- | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone     | identity        | True    | admin     | http://os-identity.materna.com:35357/v3                       |
+ | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone     | identity        | True    | internal  | http://os-identity.mycompany.com:5000/v3                        |
+ | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone     | identity        | True    | public    | https://os-cloud.mycompany.com:5000/v3                          |
+ | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone     | identity        | True    | admin     | http://os-identity.mycompany.com:35357/v3                       |
  
  ################
  nova.conf
  ################
  ...
  [keystone_authtoken]
  auth_type = password
- auth_uri = http://os-cloud.materna.com:5000
+ auth_uri = http://os-cloud.mycompany.com:5000
  auth_url = http://os-identity:35357
  memcached_servers = os-memcache:11211
  password = novapass
  project_domain_name = default
  project_name = service
  user_domain_name = default
  username = nova
  ...
  
  Using the option "insecure = True" is a workaround to avoid that Nova
  sometimes fails when the service uses Keystone's public https endpoint.
  
  Can someone please have a look?

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1716344

Title:
  Nova-API sometimes uses Keystone's public endpoint

Status in OpenStack Compute (nova):
  New

Bug description:
  I have setup a fresh HA deployment of OpenStack Pike on Ubuntu 16.04.
  I recognized in the logs that Nova sometimes fails during vm creation
  with the following exception:

  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Traceback (most recent call last):
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/identity.py", line 42, in verify_project_id
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     raise_exc=False)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 845, in get
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     return self.request(url, 'GET', **kwargs)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     return wrapped(*args, **kwargs)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     resp = send(**kwargs)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 765, in _send_request
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity     raise exceptions.SSLError(msg)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
  2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity

  Keystone's public endpoint should only visible to external clients.
  All internal OpenStack services should use the internalURL for
  authentication purposes. I think my configuration is correct. The
  "auth_url" point to Keystone's internal URL, whereas "auth_uri" points
  to Keystone's public endpoint. The strange thing is, that sometimes
  after a service restart, Nova uses the Keystone's internal URL and
  sometimes the Keystone's public URL. I want to avoid https based
  communication for the internal cloud services.

  $ openstack endpoint list | grep keystone
  | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone     | identity        | True    | internal  | http://os-identity.mycompany.com:5000/v3                        |
  | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone     | identity        | True    | public    | https://os-cloud.mycompany.com:5000/v3                          |
  | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone     | identity        | True    | admin     | http://os-identity.mycompany.com:35357/v3                       |

  ################
  nova.conf
  ################
  ...
  [keystone_authtoken]
  auth_type = password
  auth_uri = http://os-cloud.mycompany.com:5000
  auth_url = http://os-identity:35357
  memcached_servers = os-memcache:11211
  password = novapass
  project_domain_name = default
  project_name = service
  user_domain_name = default
  username = nova
  ...

  Using the option "insecure = True" is a workaround to avoid that Nova
  sometimes fails when the service uses Keystone's public https
  endpoint.

  Can someone please have a look?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1716344/+subscriptions
Follow ups

[Bug 1716344] Re: Nova-API uses Keystone's public endpoint for project id verification
From: OpenStack Infra, 2017-12-05
[Bug 1716344] Re: Nova-API uses Keystone's public endpoint for project id verification
From: Matt Riedemann, 2017-11-30