yahoo-eng-team team mailing list archive

[C Leavett-Brown <crlb@xxxxxxx>]

Public bug reported:

Description:

When a client, eg. the OpenStack dashboard, makes a nova service request
of "http://controller:8774/v2.1"; (show controller versions), it receives
a redirect to "http://controller:8774/v2.1/";. This is erroneous for at
least the following two reasons:

1. If, for security reasons, you place the nova service behind an SSL
termination proxy, the redirect generated as follows:

      from : https://controller:proxy_port/v2.1

      to     : http://controller:proxy_port/v2.1/

is invalid because the proxy_port requires encrypted traffic and the
replacement URL is using the wrong protocol (http). The request fails on
the client side with "Unable to establish connection to
http://controller:proxy_port/v2.1/: ('Connection aborted.',
BadStatusLine("''",))".

2. Even if we are not using a proxy server, the nova service is effectively complaining about a missing trailing forward slash ("/"), telling the client to reissue the same request but with the missing character. This creates unnecessary network traffic (the redirect plus a second request) and additional server load (two requests instead of one). It should be noted that "http://controller:8774/v2.1"; is the endpoint specification recommended in the OpenStack nova installation guides for the ocata, and pike releases. This will result in unnecessary traffic and load on many installations, which will go unnoticed because the request eventually works.
Solution:

Replace the first ROUTE_LIST entry (and associated comments) in
nova.api.openstack.compute.routes, changing it from:

    # NOTE: This is a redirection from '' to '/'. The request to the '/v2.1'
    # or '/2.0' without the ending '/' will get a response with status code
    # '302' returned.
    ('', '/'),

to:

    # The following 3 lines replaces a redirect specification that caused additional network traffic and load. See bug #xxxxx.
    ('', {
        'GET': [version_controller, 'show']

I've applied/tested a fix/workaround here: https://github.com/hep-
gc/nova/commit/b9c27bf29f7042cf637b58c87d6a9b2f3a9b78b6

To recreate:
1. Install Openstack (ocata/pike) as per https://docs.openstack.org/pike/install/
2. Monitor network traffic (tcpdump) on client.
3. Login to the dashboard, and view compute->project->overview

To see "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))" error:
4. Install HAProxy.
5. Serve the nova public endpoint via the SSL termination proxy server. Our HAProxy configuration for this is as follows:
  frontend nova_public
    bind controller_fqdn:18774 ssl crt /etc/letsencrypt/live/controller_fqdn/web_crt_key.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend nova_internal

  backend nova_internal
    redirect scheme https code 301 if !{ ssl_fc }
    server controller controller:8774 check
6. Redefine the nova public endpoint in the sql database:
  mysql -ukeystone -p
  connect keystone;
  update endpoint set url="https://otter.heprc.uvic.ca:18774/v2.1"; where id="xxxxxxx"
7. Dashboard will display "Unable to retrieve usage data" red flag each time the project overview page is displayed, and the http error log will report the connection failure.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1728732

Title:
  OpenStack nova service responds with an erroneous httpd redirect to a
  "GET,version_controller,show" request.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:

  When a client, eg. the OpenStack dashboard, makes a nova service
  request of "http://controller:8774/v2.1"; (show controller versions),
  it receives a redirect to "http://controller:8774/v2.1/";. This is
  erroneous for at least the following two reasons:

  1. If, for security reasons, you place the nova service behind an SSL
  termination proxy, the redirect generated as follows:

        from : https://controller:proxy_port/v2.1

        to     : http://controller:proxy_port/v2.1/

  is invalid because the proxy_port requires encrypted traffic and the
  replacement URL is using the wrong protocol (http). The request fails
  on the client side with "Unable to establish connection to
  http://controller:proxy_port/v2.1/: ('Connection aborted.',
  BadStatusLine("''",))".

  2. Even if we are not using a proxy server, the nova service is effectively complaining about a missing trailing forward slash ("/"), telling the client to reissue the same request but with the missing character. This creates unnecessary network traffic (the redirect plus a second request) and additional server load (two requests instead of one). It should be noted that "http://controller:8774/v2.1"; is the endpoint specification recommended in the OpenStack nova installation guides for the ocata, and pike releases. This will result in unnecessary traffic and load on many installations, which will go unnoticed because the request eventually works.
  Solution:

  Replace the first ROUTE_LIST entry (and associated comments) in
  nova.api.openstack.compute.routes, changing it from:

      # NOTE: This is a redirection from '' to '/'. The request to the '/v2.1'
      # or '/2.0' without the ending '/' will get a response with status code
      # '302' returned.
      ('', '/'),

  to:

      # The following 3 lines replaces a redirect specification that caused additional network traffic and load. See bug #xxxxx.
      ('', {
          'GET': [version_controller, 'show']

  I've applied/tested a fix/workaround here: https://github.com/hep-
  gc/nova/commit/b9c27bf29f7042cf637b58c87d6a9b2f3a9b78b6

  To recreate:
  1. Install Openstack (ocata/pike) as per https://docs.openstack.org/pike/install/
  2. Monitor network traffic (tcpdump) on client.
  3. Login to the dashboard, and view compute->project->overview

  To see "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))" error:
  4. Install HAProxy.
  5. Serve the nova public endpoint via the SSL termination proxy server. Our HAProxy configuration for this is as follows:
    frontend nova_public
      bind controller_fqdn:18774 ssl crt /etc/letsencrypt/live/controller_fqdn/web_crt_key.pem
      reqadd X-Forwarded-Proto:\ https
      default_backend nova_internal

    backend nova_internal
      redirect scheme https code 301 if !{ ssl_fc }
      server controller controller:8774 check
  6. Redefine the nova public endpoint in the sql database:
    mysql -ukeystone -p
    connect keystone;
    update endpoint set url="https://otter.heprc.uvic.ca:18774/v2.1"; where id="xxxxxxx"
  7. Dashboard will display "Unable to retrieve usage data" red flag each time the project overview page is displayed, and the http error log will report the connection failure.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1728732/+subscriptions