← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #88787

[Bug 1971592] [NEW] System scoped token: System->system information->compute service list results in 403

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Hi all,
We have started to analyze Openstack Yoga a bit and there, one of the major new feature is the activation of scope based token for regular use in nova. While after some long lasting back and forth in configuring our role assignments and policies we could make it work on one of our test environments (Ubuntu) via Openstack SDK, however, we are still struggling with some system scoped API calls to nova from horizon.

We have an admin user for the domain 'Default' who has set the role ‚admin' for 'system all':
+-------+---------------+-------+---------+--------+--------+-----------+
| Role  | User          | Group | Project | Domain | System | Inherited |
+-------+---------------+-------+---------+--------+--------+-----------+
| admin | admin@Default |       |         |        | all    | False     |
+-------+---------------+-------+---------+--------+--------+-----------+ 
To make login in horizon successful, the user admin is also admin for the domain default and for the project 'admin' in the domain 'default'.

We have configured in local_settings.py:

SYSTEM_SCOPE_SERVICES = ['compute', 'image', 'volume', 'network‘]

(Note: this config line has been reverse engineered from horizon source
code as the syntax is nowhere possible to be found in the docs yet … -
so: not sure if it is correct)

Policy files are identical for horizon as for the services.

For the user admin, we then get an additional field in the
domain/project top line menu adding a ‚system scope‘ switch (this is
what we understand how it should look like) and - when switching to
system scope - also a system menu in the sidebar becomes visible (also
as expected).

If we then go to System->Systeminformation to see the nova service list,
we get an error ‚Unable to get nova services list‘, given reason is an
error: 'Policy doesn’t allow os_compute_api:os-services:list to be
performed (HTTP 403)‘. Informations for network and volume services are
shown normally (here scoped tokens are not activated yet).

apache2 error log in debug mode results for this call in:
[Wed May 04 13:40:45.314012 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:keystoneauth.identity.v3.base:Making authentication request to https://controller:5000/v3/auth/tokens
[Wed May 04 13:40:45.396814 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:https://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 12010
[Wed May 04 13:40:45.397786 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:keystoneauth.identity.v3.base:{"token": {"methods": ["password", "token"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "a837078533b64bac8be7f9fb0c57ead6", "name": "admin", "password_expires_at": null}, "audit_ids": ["EAXc26zRR4KUEwDSLwKe4w", "iUCwAVveTAay4FfkHwZ1sA"], "expires_at": "2022-05-04T12:40:30.000000Z", "issued_at": "2022-05-04T11:40:45.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6f79eb4041a142dfaffb5d1f8bff906c", "name": "admin"}, "is_domain": false, "roles": [{"id": "9109437956b544cebd1f5dfc87def5bd", "name": "admin"}, {"id": "c98267f0aa4144d8b7e89d9b40b46c0e", "name": "member"}, {"id": "eea735edc3e14222bae6703c2c672c02", "name": "reader"}], "catalog": [{"endpoints": [{"id": "7342dd7ff0ae438ba50ac669538feb4f", "interface": "
(...)

[Wed May 04 13:40:45.398554 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:8774
[Wed May 04 13:40:45.726949 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:http://controller:8774 "GET /v2.1/os-services HTTP/1.1" 403 112
[Wed May 04 13:40:45.727379 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] Recoverable error: Policy doesn't allow os_compute_api:os-services:list to be performed. (HTTP 403) (Request-ID: req-592af610-2daa-4318-8a2a-2f7fa89f83c6)


Logs indicate that horizon is using still a project-scoped token and not a system-scoped one for these requests although ’system scope’ is active. 

Putting the same request from Openstack SDK with the same user 'admin'
results in

$ openstack compute service list
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
+----+------------------+-------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host        | Zone     | Status  | State | Updated At                 |
+----+------------------+-------------+----------+---------+-------+----------------------------+
|  4 | nova-consoleauth | controller  | nova     | enabled | down  | 2019-10-31T14:59:33.000000 |
|  5 | nova-scheduler   | controller  | nova     | enabled | up    | 2022-05-04T08:52:48.000000 |
|  6 | nova-conductor   | controller  | nova     | enabled | up    | 2022-05-04T08:52:42.000000 |
| 12 | nova-compute     | compute3    | Crandale | enabled | up    | 2022-05-04T08:52:40.000000 |
| 13 | nova-conductor   | controller3 | nova     | enabled | down  | 2020-06-28T14:45:31.000000 |
| 14 | nova-scheduler   | controller3 | nova     | enabled | down  | 2020-06-28T14:45:24.000000 |
+----+------------------+-------------+----------+---------+-------+—————————————--------------—+

Which indicates that role assignments to user admin are correct. The
same command with -—debug also proves that a system scoped token is
generated.

clouds.yaml for this user look like:
clouds:
  mycloud-admin:
    region_name: RegionOne
    auth:
      # domain_name: 'Default'
      username: 'admin'
      password: 'secretsecret'
      # project_name: 'admin'
      auth_url: 'https://controller:5000/v3'
      #project_domain_name: 'default'
      user_domain_name: 'default'
      system_scope: all
    interface: 'public'
    identity_api_version: 3
    volume_api_version: 3.68
    compute_api_version: 2.90

observation is, that a system scoped toked is ONLY generated if neither
'domain_name' nor 'project_(domain_)name' are defined in clouds.yaml -
only then, the service list request suceeds; otherwise a domain resp.
project scoped token is generated.Therefore they have been commended out
in clouds.yaml.

Here, another issue comes in: if we use the downloaded open-rc.sh file
from horizon for the user 'admin',

#!/usr/bin/env bash
# To use an OpenStack cloud you need to authenticate against the Identity
# service named keystone, which returns a **Token** and **Service Catalog**.
# The catalog contains the endpoints for all services the user/tenant has
# access to - such as Compute, Image Service, Identity, Object Storage, Block
# Storage, and Networking (code-named nova, glance, keystone, swift,
# cinder, and neutron).
#
# *NOTE*: Using the 3 *Identity API* does not necessarily mean any other
# OpenStack API is version 3. For example, your cloud provider may implement
# Image API v1.1, Block Storage API v2, and Compute API v2.0. OS_AUTH_URL is
# only for the Identity API served through keystone.
export OS_AUTH_URL=https://controller:5000
# With the addition of Keystone we have standardized on the term **project**
# as the entity that owns the resources.
export OS_PROJECT_ID=None
export OS_PROJECT_NAME="None"
export OS_USER_DOMAIN_NAME="Default"
if [ -z "$OS_USER_DOMAIN_NAME" ]; then unset OS_USER_DOMAIN_NAME; fi
export OS_PROJECT_DOMAIN_ID="None"
if [ -z "$OS_PROJECT_DOMAIN_ID" ]; then unset OS_PROJECT_DOMAIN_ID; fi
# unset v2.0 items in case set
unset OS_TENANT_ID
unset OS_TENANT_NAME
# In addition to the owning entity (tenant), OpenStack stores the entity
# performing the action as the **user**.
export OS_USERNAME="admin"
# With Keystone you pass the keystone password.
echo "Please enter your OpenStack Password for project $OS_PROJECT_NAME as user $OS_USERNAME: "
read -sr OS_PASSWORD_INPUT
export OS_PASSWORD=$OS_PASSWORD_INPUT
# If your configuration has multiple regions, we set that information here.
# OS_REGION_NAME is optional and only valid in certain environments.
export OS_REGION_NAME="RegionOne"
# Don't leave a blank variable, unset it if it was empty
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3

it is obvious, that Project_ID and project_name are defined and set to
'None'. Applying this for openstack SDK,

$ openstack compute service list

also throws an error 403 as a project scoped token is generated.

Not sure whether this is a second issue but at least it looks
inconsistent to the required behavior and would explain why the request
in horizon fails.

** Affects: horizon
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1971592

Title:
  System scoped token: System->system information->compute service list
  results in 403

Status in OpenStack Dashboard (Horizon):
  New

Bug description:
  Hi all,
  We have started to analyze Openstack Yoga a bit and there, one of the major new feature is the activation of scope based token for regular use in nova. While after some long lasting back and forth in configuring our role assignments and policies we could make it work on one of our test environments (Ubuntu) via Openstack SDK, however, we are still struggling with some system scoped API calls to nova from horizon.

  We have an admin user for the domain 'Default' who has set the role ‚admin' for 'system all':
  +-------+---------------+-------+---------+--------+--------+-----------+
  | Role  | User          | Group | Project | Domain | System | Inherited |
  +-------+---------------+-------+---------+--------+--------+-----------+
  | admin | admin@Default |       |         |        | all    | False     |
  +-------+---------------+-------+---------+--------+--------+-----------+ 
  To make login in horizon successful, the user admin is also admin for the domain default and for the project 'admin' in the domain 'default'.

  We have configured in local_settings.py:

  SYSTEM_SCOPE_SERVICES = ['compute', 'image', 'volume', 'network‘]

  (Note: this config line has been reverse engineered from horizon
  source code as the syntax is nowhere possible to be found in the docs
  yet … - so: not sure if it is correct)

  Policy files are identical for horizon as for the services.

  For the user admin, we then get an additional field in the
  domain/project top line menu adding a ‚system scope‘ switch (this is
  what we understand how it should look like) and - when switching to
  system scope - also a system menu in the sidebar becomes visible (also
  as expected).

  If we then go to System->Systeminformation to see the nova service
  list, we get an error ‚Unable to get nova services list‘, given reason
  is an error: 'Policy doesn’t allow os_compute_api:os-services:list to
  be performed (HTTP 403)‘. Informations for network and volume services
  are shown normally (here scoped tokens are not activated yet).

  apache2 error log in debug mode results for this call in:
  [Wed May 04 13:40:45.314012 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:keystoneauth.identity.v3.base:Making authentication request to https://controller:5000/v3/auth/tokens
  [Wed May 04 13:40:45.396814 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:https://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 12010
  [Wed May 04 13:40:45.397786 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:keystoneauth.identity.v3.base:{"token": {"methods": ["password", "token"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "a837078533b64bac8be7f9fb0c57ead6", "name": "admin", "password_expires_at": null}, "audit_ids": ["EAXc26zRR4KUEwDSLwKe4w", "iUCwAVveTAay4FfkHwZ1sA"], "expires_at": "2022-05-04T12:40:30.000000Z", "issued_at": "2022-05-04T11:40:45.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6f79eb4041a142dfaffb5d1f8bff906c", "name": "admin"}, "is_domain": false, "roles": [{"id": "9109437956b544cebd1f5dfc87def5bd", "name": "admin"}, {"id": "c98267f0aa4144d8b7e89d9b40b46c0e", "name": "member"}, {"id": "eea735edc3e14222bae6703c2c672c02", "name": "reader"}], "catalog": [{"endpoints": [{"id": "7342dd7ff0ae438ba50ac669538feb4f", "interface": "
  (...)

  [Wed May 04 13:40:45.398554 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:8774
  [Wed May 04 13:40:45.726949 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:http://controller:8774 "GET /v2.1/os-services HTTP/1.1" 403 112
  [Wed May 04 13:40:45.727379 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] Recoverable error: Policy doesn't allow os_compute_api:os-services:list to be performed. (HTTP 403) (Request-ID: req-592af610-2daa-4318-8a2a-2f7fa89f83c6)

  
  Logs indicate that horizon is using still a project-scoped token and not a system-scoped one for these requests although ’system scope’ is active. 

  Putting the same request from Openstack SDK with the same user 'admin'
  results in

  $ openstack compute service list
  /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
    from cryptography.utils import int_from_bytes
  /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
    from cryptography.utils import int_from_bytes
  +----+------------------+-------------+----------+---------+-------+----------------------------+
  | ID | Binary           | Host        | Zone     | Status  | State | Updated At                 |
  +----+------------------+-------------+----------+---------+-------+----------------------------+
  |  4 | nova-consoleauth | controller  | nova     | enabled | down  | 2019-10-31T14:59:33.000000 |
  |  5 | nova-scheduler   | controller  | nova     | enabled | up    | 2022-05-04T08:52:48.000000 |
  |  6 | nova-conductor   | controller  | nova     | enabled | up    | 2022-05-04T08:52:42.000000 |
  | 12 | nova-compute     | compute3    | Crandale | enabled | up    | 2022-05-04T08:52:40.000000 |
  | 13 | nova-conductor   | controller3 | nova     | enabled | down  | 2020-06-28T14:45:31.000000 |
  | 14 | nova-scheduler   | controller3 | nova     | enabled | down  | 2020-06-28T14:45:24.000000 |
  +----+------------------+-------------+----------+---------+-------+—————————————--------------—+

  Which indicates that role assignments to user admin are correct. The
  same command with -—debug also proves that a system scoped token is
  generated.

  clouds.yaml for this user look like:
  clouds:
    mycloud-admin:
      region_name: RegionOne
      auth:
        # domain_name: 'Default'
        username: 'admin'
        password: 'secretsecret'
        # project_name: 'admin'
        auth_url: 'https://controller:5000/v3'
        #project_domain_name: 'default'
        user_domain_name: 'default'
        system_scope: all
      interface: 'public'
      identity_api_version: 3
      volume_api_version: 3.68
      compute_api_version: 2.90

  observation is, that a system scoped toked is ONLY generated if
  neither 'domain_name' nor 'project_(domain_)name' are defined in
  clouds.yaml - only then, the service list request suceeds; otherwise a
  domain resp. project scoped token is generated.Therefore they have
  been commended out in clouds.yaml.

  Here, another issue comes in: if we use the downloaded open-rc.sh file
  from horizon for the user 'admin',

  #!/usr/bin/env bash
  # To use an OpenStack cloud you need to authenticate against the Identity
  # service named keystone, which returns a **Token** and **Service Catalog**.
  # The catalog contains the endpoints for all services the user/tenant has
  # access to - such as Compute, Image Service, Identity, Object Storage, Block
  # Storage, and Networking (code-named nova, glance, keystone, swift,
  # cinder, and neutron).
  #
  # *NOTE*: Using the 3 *Identity API* does not necessarily mean any other
  # OpenStack API is version 3. For example, your cloud provider may implement
  # Image API v1.1, Block Storage API v2, and Compute API v2.0. OS_AUTH_URL is
  # only for the Identity API served through keystone.
  export OS_AUTH_URL=https://controller:5000
  # With the addition of Keystone we have standardized on the term **project**
  # as the entity that owns the resources.
  export OS_PROJECT_ID=None
  export OS_PROJECT_NAME="None"
  export OS_USER_DOMAIN_NAME="Default"
  if [ -z "$OS_USER_DOMAIN_NAME" ]; then unset OS_USER_DOMAIN_NAME; fi
  export OS_PROJECT_DOMAIN_ID="None"
  if [ -z "$OS_PROJECT_DOMAIN_ID" ]; then unset OS_PROJECT_DOMAIN_ID; fi
  # unset v2.0 items in case set
  unset OS_TENANT_ID
  unset OS_TENANT_NAME
  # In addition to the owning entity (tenant), OpenStack stores the entity
  # performing the action as the **user**.
  export OS_USERNAME="admin"
  # With Keystone you pass the keystone password.
  echo "Please enter your OpenStack Password for project $OS_PROJECT_NAME as user $OS_USERNAME: "
  read -sr OS_PASSWORD_INPUT
  export OS_PASSWORD=$OS_PASSWORD_INPUT
  # If your configuration has multiple regions, we set that information here.
  # OS_REGION_NAME is optional and only valid in certain environments.
  export OS_REGION_NAME="RegionOne"
  # Don't leave a blank variable, unset it if it was empty
  if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
  export OS_INTERFACE=public
  export OS_IDENTITY_API_VERSION=3

  it is obvious, that Project_ID and project_name are defined and set to
  'None'. Applying this for openstack SDK,

  $ openstack compute service list

  also throws an error 403 as a project scoped token is generated.

  Not sure whether this is a second issue but at least it looks
  inconsistent to the required behavior and would explain why the
  request in horizon fails.

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1971592/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next