yahoo-eng-team team mailing list archive

[Radomir Dopieralski <1971592@xxxxxxxxxxxxxxxxxx>]

The compute service does not support system scope tokens fully yet.

** Changed in: horizon
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1971592

Title:
  System scoped token: System->system information->compute service list
  results in 403

Status in OpenStack Dashboard (Horizon):
  Invalid

Bug description:
  Hi all,
  We have started to analyze Openstack Yoga a bit and there, one of the major new feature is the activation of scope based token for regular use in nova. While after some long lasting back and forth in configuring our role assignments and policies we could make it work on one of our test environments (Ubuntu) via Openstack SDK, however, we are still struggling with some system scoped API calls to nova from horizon.

  We have an admin user for the domain 'Default' who has set the role ‚admin' for 'system all':
  +-------+---------------+-------+---------+--------+--------+-----------+
  | Role  | User          | Group | Project | Domain | System | Inherited |
  +-------+---------------+-------+---------+--------+--------+-----------+
  | admin | admin@Default |       |         |        | all    | False     |
  +-------+---------------+-------+---------+--------+--------+-----------+
  To make login in horizon successful, the user admin is also admin for the domain 'default' and for the project 'admin' in the domain 'default'.

  We have configured in local_settings.py:

  SYSTEM_SCOPE_SERVICES = ['compute', 'image', 'volume', 'network‘]

  (Note: this config line has been reverse engineered from horizon
  source code as the syntax is nowhere possible to be found in the docs
  yet … - so: not sure if it is correct)

  Policy files are identical for horizon as for the services. Policy for this api request is
  "os_compute_api:os-services:list": "rule:context_is_admin"

  For the user admin, we then get an additional field in the
  domain/project top line menu adding a ‚system scope‘ switch (this is
  what we understand how it should look like) and - when switching to
  system scope - also a 'system' menu in the sidebar becomes visible
  (also as expected).

  If we then go to System->Systeminformation to see the nova service
  list, we get an error ‚Unable to get nova services list‘, given reason
  is an error: 'Policy doesn’t allow os_compute_api:os-services:list to
  be performed (HTTP 403)‘. Informations for network and volume services
  are shown normally (here scoped tokens are not activated yet).

  apache2 error log in debug mode results for this call in:
  [Wed May 04 13:40:45.314012 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:keystoneauth.identity.v3.base:Making authentication request to https://controller:5000/v3/auth/tokens
  [Wed May 04 13:40:45.396814 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:https://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 12010
  [Wed May 04 13:40:45.397786 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:keystoneauth.identity.v3.base:{"token": {"methods": ["password", "token"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "a837078533b64bac8be7f9fb0c57ead6", "name": "admin", "password_expires_at": null}, "audit_ids": ["EAXc26zRR4KUEwDSLwKe4w", "iUCwAVveTAay4FfkHwZ1sA"], "expires_at": "2022-05-04T12:40:30.000000Z", "issued_at": "2022-05-04T11:40:45.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6f79eb4041a142dfaffb5d1f8bff906c", "name": "admin"}, "is_domain": false, "roles": [{"id": "9109437956b544cebd1f5dfc87def5bd", "name": "admin"}, {"id": "c98267f0aa4144d8b7e89d9b40b46c0e", "name": "member"}, {"id": "eea735edc3e14222bae6703c2c672c02", "name": "reader"}], "catalog": [{"endpoints": [{"id": "7342dd7ff0ae438ba50ac669538feb4f", "interface": "
  (...)

  [Wed May 04 13:40:45.398554 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:8774
  [Wed May 04 13:40:45.726949 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] DEBUG:urllib3.connectionpool:http://controller:8774 "GET /v2.1/os-services HTTP/1.1" 403 112
  [Wed May 04 13:40:45.727379 2022] [wsgi:error] [pid 3007771:tid 140637774444288] [remote 192.168.1.70:59546] Recoverable error: Policy doesn't allow os_compute_api:os-services:list to be performed. (HTTP 403) (Request-ID: req-592af610-2daa-4318-8a2a-2f7fa89f83c6)

  Logs indicate that horizon is using still a project-scoped token and
  not a system-scoped one for these requests although ’system scope’ is
  active.

  Putting the same request from Openstack SDK with the same user 'admin'
  results in

  $ openstack compute service list
  /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
    from cryptography.utils import int_from_bytes
  /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
    from cryptography.utils import int_from_bytes
  +----+------------------+-------------+----------+---------+-------+----------------------------+
  | ID | Binary           | Host        | Zone     | Status  | State | Updated At                 |
  +----+------------------+-------------+----------+---------+-------+----------------------------+
  |  4 | nova-consoleauth | controller  | nova     | enabled | down  | 2019-10-31T14:59:33.000000 |
  |  5 | nova-scheduler   | controller  | nova     | enabled | up    | 2022-05-04T08:52:48.000000 |
  |  6 | nova-conductor   | controller  | nova     | enabled | up    | 2022-05-04T08:52:42.000000 |
  | 12 | nova-compute     | compute3    | Crandale | enabled | up    | 2022-05-04T08:52:40.000000 |
  | 13 | nova-conductor   | controller3 | nova     | enabled | down  | 2020-06-28T14:45:31.000000 |
  | 14 | nova-scheduler   | controller3 | nova     | enabled | down  | 2020-06-28T14:45:24.000000 |
  +----+------------------+-------------+----------+---------+-------+—————————————--------------—+

  Which indicates that role assignments to user admin are correct. The
  same command with -—debug also proves that a system scoped token is
  generated.

  clouds.yaml for this user look like:
  clouds:
    mycloud-admin:
      region_name: RegionOne
      auth:
        # domain_name: 'Default'
        username: 'admin'
        password: 'secretsecret'
        # project_name: 'admin'
        auth_url: 'https://controller:5000/v3'
        #project_domain_name: 'default'
        user_domain_name: 'default'
        system_scope: all
      interface: 'public'
      identity_api_version: 3
      volume_api_version: 3.68
      compute_api_version: 2.90

  observation is, that a system scoped token is ONLY generated if
  neither 'domain_name' nor 'project_(domain_)name' are DEFINED in
  clouds.yaml - only then, the service list request suceeds; otherwise a
  domain resp. project scoped token is generated. Therefore they have
  been commented out in clouds.yaml - in the old environment variable
  logic, OS_PROJECT_ID etc. must left UNSET.

  Here, another issue comes in: if we use the downloaded open-rc.sh file
  from horizon for the user 'admin',

  #!/usr/bin/env bash
  # To use an OpenStack cloud you need to authenticate against the Identity
  # service named keystone, which returns a **Token** and **Service Catalog**.
  # The catalog contains the endpoints for all services the user/tenant has
  # access to - such as Compute, Image Service, Identity, Object Storage, Block
  # Storage, and Networking (code-named nova, glance, keystone, swift,
  # cinder, and neutron).
  #
  # *NOTE*: Using the 3 *Identity API* does not necessarily mean any other
  # OpenStack API is version 3. For example, your cloud provider may implement
  # Image API v1.1, Block Storage API v2, and Compute API v2.0. OS_AUTH_URL is
  # only for the Identity API served through keystone.
  export OS_AUTH_URL=https://controller:5000
  # With the addition of Keystone we have standardized on the term **project**
  # as the entity that owns the resources.
  export OS_PROJECT_ID=None
  export OS_PROJECT_NAME="None"
  export OS_USER_DOMAIN_NAME="Default"
  if [ -z "$OS_USER_DOMAIN_NAME" ]; then unset OS_USER_DOMAIN_NAME; fi
  export OS_PROJECT_DOMAIN_ID="None"
  if [ -z "$OS_PROJECT_DOMAIN_ID" ]; then unset OS_PROJECT_DOMAIN_ID; fi
  # unset v2.0 items in case set
  unset OS_TENANT_ID
  unset OS_TENANT_NAME
  # In addition to the owning entity (tenant), OpenStack stores the entity
  # performing the action as the **user**.
  export OS_USERNAME="admin"
  # With Keystone you pass the keystone password.
  echo "Please enter your OpenStack Password for project $OS_PROJECT_NAME as user $OS_USERNAME: "
  read -sr OS_PASSWORD_INPUT
  export OS_PASSWORD=$OS_PASSWORD_INPUT
  # If your configuration has multiple regions, we set that information here.
  # OS_REGION_NAME is optional and only valid in certain environments.
  export OS_REGION_NAME="RegionOne"
  # Don't leave a blank variable, unset it if it was empty
  if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
  export OS_INTERFACE=public
  export OS_IDENTITY_API_VERSION=3

  it is obvious, that Project_ID and project_name are defined and set to
  'None'. When trying this user config for any nova-api call with the
  policy set to 'rule:context_is_admin' with a system scoped call via
  Openstack SDK, it throws an error 400 resp. 403 as a project scoped
  token is generated resp. project 'None' results in an error; which
  consequently manifests in the nova-api (here example os-quota-sets)
  log as

  2022-05-04 18:25:34.253 2662915 INFO nova.api.openstack.wsgi [req-6e6ee519-36f2-4937-a674-6493d8232ff6 a837078533b64bac8be7f9fb0c57ead6 - - default default] HTTP exception thrown: Project ID None is not a valid project.
  2022-05-04 18:25:34.254 2662915 INFO nova.osapi_compute.wsgi.server [req-6e6ee519-36f2-4937-a674-6493d8232ff6 a837078533b64bac8be7f9fb0c57ead6 - - default default] 10.0.88.11 "GET /v2.1/os-quota-sets/None/defaults HTTP/1.1" status: 400 len: 505 time: 6.8056376

  Not sure whether this is a second issue but at least it looks
  inconsistent to the required behavior and would explain why the
  request in horizon fails.

  To me meanwhile it looks like a rather structural challenge and not
  like a simple code bug ...

  br c

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1971592/+subscriptions