yahoo-eng-team team mailing list archive

[OpenStack Infra <1752152@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/557508
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=20eaaee2334957eb8739ecca524a1c4aa9f246e9
Submitter: Zuul
Branch:    master

commit 20eaaee2334957eb8739ecca524a1c4aa9f246e9
Author: Eric Fried <efried@xxxxxxxxxx>
Date:   Wed Mar 28 15:45:26 2018 -0500

    Use ksa session for cinder microversion check
    
    [1] added a method to validate availability of a desired version of the
    cinder API.  This method called into
    cinderclient.client.get_highest_client_server_version to
    (unsurprisingly) discover the highest available version to compare
    against.  However, that routine uses a raw requests.get to access the
    version document from the server.  This breaks when the endpoint URL is
    using HTTPS, because nothing sets up the cert info for that call.
    
    With this change, we work around the issue by duplicating the logic from
    get_highest_client_server_version, but doing the version discovery via
    the same keystoneauth1 session that's configured for use with the client
    itself, thus inheriting any SSL configuration as appropriate.
    
    [1] https://review.openstack.org/#/c/469579/
    
    Change-Id: I4de355195281009a5979710d7f14ae8ea11d10e0
    Closes-Bug: #1752152


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1752152

Title:
  Attach Volume Fails with secure call to cinder

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  In Progress
Status in python-cinderclient:
  Invalid

Bug description:
  It is found that when cinder endpoint is configured to use https,
  attach volume flow fails with the stack trace seen below (seen in nova
  api log) because it fails to make a secure call from nova to cinder.
  Secure calls perform certificate validation and in this particular
  flow, certificate validation is completely skipped

  File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3971, in attach_volume
  2018-02-27 08:16:51.338 1324 ERROR      cinder.is_microversion_supported(context, '3.44')
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 138, in is_microversion_supported
  2018-02-27 08:16:51.338 1324 ERROR      _check_microversion(url, microversion)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 86, in _check_microversion
  2018-02-27 08:16:51.338 1324 ERROR      max_api_version = cinder_client.get_highest_client_server_version(url)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 126, in get_highest_client_server_version
  2018-02-27 08:16:51.338 1324 ERROR      min_server, max_server = get_server_version(url)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 109, in get_server_version
  2018-02-27 08:16:51.338 1324 ERROR      response = requests.get(version_url)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/requests/api.py", line 72, in get
  2018-02-27 08:16:51.338 1324 ERROR      return request('get', url, params=params, **kwargs)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
  2018-02-27 08:16:51.338 1324 ERROR      return session.request(method=method, url=url, **kwargs)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 502, in request
  2018-02-27 08:16:51.338 1324 ERROR      resp = self.send(prep, **send_kwargs)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 612, in send
  2018-02-27 08:16:51.338 1324 ERROR      r = adapter.send(request, **kwargs)
  2018-02-27 08:16:51.338 1324 ERROR    File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 504, in send
  2018-02-27 08:16:51.338 1324 ERROR      raise ConnectionError(e, request=request)
  2018-02-27 08:16:51.338 1324 ERROR  ConnectionError: HTTPSConnectionPool(host='ipx-x-x-x.xxx.xxx.xxx.com', port=9000): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",),))

  This is a regression bug introduced as part of changeset
  https://review.openstack.org/#/c/469579/, which was merged way back in
  June 2017. As part of this changeset, a new function namely
  _check_microversion was introduced, which then makes a cinderclient
  call , which finally makes a cinder https REST api call without
  passing the certificate. This leads to the problem listed above.

  https://github.com/openstack/nova/blob/stable/queens/nova/volume/cinder.py#L75
  https://github.com/openstack/nova/blob/stable/queens/nova/volume/cinder.py#L86

  https://github.com/openstack/python-cinderclient/blob/stable/queens/cinderclient/client.py#L126
  https://github.com/openstack/python-cinderclient/blob/stable/queens/cinderclient/client.py#L109

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1752152/+subscriptions