yahoo-eng-team team mailing list archive

[OpenStack Infra <1728732@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/685181
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1e907602e37fb55bbe5a20164db6d074f87369af
Submitter: Zuul
Branch:    master

commit 1e907602e37fb55bbe5a20164db6d074f87369af
Author: Eric Fried <openstack@xxxxxxxx>
Date:   Thu Sep 26 16:52:12 2019 -0500

    Allow versioned discovery unauthenticated
    
    Make routes to the versioned discovery documents (/v2, /v2.1) go through
    paste pipelines that don't require authentication, while leaving their
    sub-URLs (/v2.1/servers etc) requiring authentication.
    
    To make this work, our URLMap matcher gets support for a very
    rudimentary wildcard syntax, whereby api-paste.ini can differentiate
    between {/v2.1, /v2.1/} and /v2.1/$anything_else. The former points to
    the unauthenticated discovery app pipeline; the latter points to the
    existing "real API" pipeline. Similar for legacy v2.
    
    This entails a slight behavior change: requests to /v2 and /v2.1 used to
    302 redirect to /v2/ and /v2.1/, respectively. Now they just work.
    
    Change-Id: Id47515017982850b167d5c637d93b96ae00ba793
    Closes-Bug: #1845530
    Closes-Bug: #1728732


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1728732

Title:
  OpenStack nova service responds with an erroneous httpd redirect to a
  "GET,version_controller,show" request.

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description:

  When a client, eg. the OpenStack dashboard, makes a nova service
  request of "http://controller:8774/v2.1"; (show controller versions),
  it receives a redirect to "http://controller:8774/v2.1/";. This is
  erroneous for at least the following two reasons:

  1. If, for security reasons, you place the nova service behind an SSL
  termination proxy, the redirect generated as follows:

        from : https://controller:proxy_port/v2.1

        to     : http://controller:proxy_port/v2.1/

  is invalid because the proxy_port requires encrypted traffic and the
  replacement URL is using the wrong protocol (http). The request fails
  on the client side with "Unable to establish connection to
  http://controller:proxy_port/v2.1/: ('Connection aborted.',
  BadStatusLine("''",))".

  2. Even if we are not using a proxy server, the nova service is effectively complaining about a missing trailing forward slash ("/"), telling the client to reissue the same request but with the missing character. This creates unnecessary network traffic (the redirect plus a second request) and additional server load (two requests instead of one). It should be noted that "http://controller:8774/v2.1"; is the endpoint specification recommended in the OpenStack nova installation guides for the ocata, and pike releases. This will result in unnecessary traffic and load on many installations, which will go unnoticed because the request eventually works.
  Solution:

  Replace the first ROUTE_LIST entry (and associated comments) in
  nova.api.openstack.compute.routes, changing it from:

      # NOTE: This is a redirection from '' to '/'. The request to the '/v2.1'
      # or '/2.0' without the ending '/' will get a response with status code
      # '302' returned.
      ('', '/'),

  to:

      # The following 3 lines replaces a redirect specification that caused additional network traffic and load. See bug #xxxxx.
      ('', {
          'GET': [version_controller, 'show']

  I've applied/tested a fix/workaround here: https://github.com/hep-
  gc/nova/commit/b9c27bf29f7042cf637b58c87d6a9b2f3a9b78b6

  To recreate:
  1. Install Openstack (ocata/pike) as per https://docs.openstack.org/pike/install/
  2. Monitor network traffic (tcpdump) on client.
  3. Login to the dashboard, and view compute->project->overview

  To see "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))" error:
  4. Install HAProxy.
  5. Serve the nova public endpoint via the SSL termination proxy server. Our HAProxy configuration for this is as follows:
    frontend nova_public
      bind controller_fqdn:18774 ssl crt /etc/letsencrypt/live/controller_fqdn/web_crt_key.pem
      reqadd X-Forwarded-Proto:\ https
      default_backend nova_internal

    backend nova_internal
      redirect scheme https code 301 if !{ ssl_fc }
      server controller controller:8774 check
  6. Redefine the nova public endpoint in the sql database:
    mysql -ukeystone -p
    connect keystone;
    update endpoint set url="https://otter.heprc.uvic.ca:18774/v2.1"; where id="xxxxxxx"
  7. Dashboard will display "Unable to retrieve usage data" red flag each time the project overview page is displayed, and the http error log will report the connection failure.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1728732/+subscriptions