yahoo-eng-team team mailing list archive

[Roman Podoliaka <rpodolyaka+bugs@xxxxxxxxxxxx>]

Public bug reported:

Description
===========

If a Keystone token issued for a admin user (e.g. ceilometer) is expired
or revoked right after it's been validated by
keystoneauthtoken_middleware in nova-api, but before it's validated by
the very same middleware in neutron-server, nova-api will respond with
400 Bad Request instead of expected 401 Unauthorized, so that the
original request can be properly retried after re-authentication.


Steps to reproduce
==================

The condition described above is easy to reproduce synthetically by
putting breakpoints into Nova code and revoking a token. One can
reproduce the very same problem in real life by running enough
ceilometer polling agents.

Make sure you use credentials of an admin user (e.g. admin or ceilometer
in Devstack) and have at least 1 instance running (so that `nova list`
triggers an HTTP request to neutron-server).

1. Put a breakpoint on entering get_client() nova/network/neutronv2/api.py
2. Do `nova list`
3. Revoke the the issued token with `openstack token revoke $token` (you may also need to restart memcached to make sure token validation result is not cached)
4. Continue execution of nova-api

Expected result
===============

As token is now invalid (expired or revoked), it's expected that nova-
api responds with 401 Unauthorized, so that a client can handle this,
re-authenticate and retry the original request.

Actual result
=============

nova-api responds with 400 Bad Request and outputs the following error
into logs

2017-01-19 15:02:09.952 595 ERROR nova.network.neutronv2.api [req-0c1558f5-9cc8-4411-9fb1-2fe7cb232725 admin admin] Neutron client was not able
 to generate a valid admin token, please verify Neutron admin credential located in nova.conf

Environment
===========

Devstack, master (Ocata), nova HEAD at
da54487edad28c87accbf6439471e7341b52ff48

** Affects: nova
     Importance: Undecided
     Assignee: Roman Podoliaka (rpodolyaka)
         Status: In Progress


** Tags: api neutron

** Changed in: nova
     Assignee: (unassigned) => Roman Podoliaka (rpodolyaka)

** Tags added: api neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1657774

Title:
  Nova does not re-raise 401 Unauthorized received from Neutron for
  admin users

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========

  If a Keystone token issued for a admin user (e.g. ceilometer) is
  expired or revoked right after it's been validated by
  keystoneauthtoken_middleware in nova-api, but before it's validated by
  the very same middleware in neutron-server, nova-api will respond with
  400 Bad Request instead of expected 401 Unauthorized, so that the
  original request can be properly retried after re-authentication.

  
  Steps to reproduce
  ==================

  The condition described above is easy to reproduce synthetically by
  putting breakpoints into Nova code and revoking a token. One can
  reproduce the very same problem in real life by running enough
  ceilometer polling agents.

  Make sure you use credentials of an admin user (e.g. admin or
  ceilometer in Devstack) and have at least 1 instance running (so that
  `nova list` triggers an HTTP request to neutron-server).

  1. Put a breakpoint on entering get_client() nova/network/neutronv2/api.py
  2. Do `nova list`
  3. Revoke the the issued token with `openstack token revoke $token` (you may also need to restart memcached to make sure token validation result is not cached)
  4. Continue execution of nova-api

  Expected result
  ===============

  As token is now invalid (expired or revoked), it's expected that nova-
  api responds with 401 Unauthorized, so that a client can handle this,
  re-authenticate and retry the original request.

  Actual result
  =============

  nova-api responds with 400 Bad Request and outputs the following error
  into logs

  2017-01-19 15:02:09.952 595 ERROR nova.network.neutronv2.api [req-0c1558f5-9cc8-4411-9fb1-2fe7cb232725 admin admin] Neutron client was not able
   to generate a valid admin token, please verify Neutron admin credential located in nova.conf

  Environment
  ===========

  Devstack, master (Ocata), nova HEAD at
  da54487edad28c87accbf6439471e7341b52ff48

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1657774/+subscriptions