← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #84810

[Bug 1909233] [NEW] Requests auth issue when there are multiple threads or processes

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Observed in train:

Steps to reproduce:

1. Setup devstack(stable/train)

2. Make the below config changes and restart keystone.

cat /etc/keystone/policy.json

```
{
  "admin_required": "role:admin or is_admin:1",
  "identity:get_domain_config_default": "rule:admin_required"
}
```

cat /etc/keystone/keystone.conf

```
[identity]
domain_specific_drivers_enabled = true
password_hash_rounds = 4
driver = sql

[assignment]
driver = sql

[role]
driver = sql

[resource]
driver = sql

[cache]
memcache_servers = localhost:11211
backend = dogpile.cache.memcached
enabled = True

[oslo_messaging_notifications]
transport_url = rabbit://stackrabbit:password@localhost:5672/

[DEFAULT]
max_token_size = 16384
debug = True
logging_exception_prefix = ERROR %(name)s %(instance)s
logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}
admin_endpoint = http://localhost/identity
public_endpoint = http://localhost/identity

[token]
provider = fernet
cache_time = 3600
caching = true
expiration = 43200

[revoke]
cache_time = 3600
caching = true
[role]
cache_time = 3600
caching = true

[database]
connection = mysql+pymysql://root:password@127.0.0.1/keystone?charset=utf8

[fernet_tokens]
key_repository = /etc/keystone/fernet-keys/

[credential]
key_repository = /etc/keystone/credential-keys/

[security_compliance]
unique_last_password_count = 2
lockout_duration = 10
lockout_failure_attempts = 2
```

cat /etc/keystone/keystone-uwsgi-public.ini

```
[uwsgi]
chmod-socket = 666
socket = /var/run/uwsgi/keystone-wsgi-public.socket
lazy-apps = true
add-header = Connection: close
buffer-size = 65535
hook-master-start = unix_signal:15 gracefully_kill_them_all
thunder-lock = true
plugins = http,python
enable-threads = true
worker-reload-mercy = 90
exit-on-reload = false
die-on-term = true
master = true
processes = 2
threads = 2
wsgi-file = /usr/local/bin/keystone-wsgi-public
```
3. Create test creds.

openstack domain create --description "Test Domain" test-domain 
openstack project create --domain Test-Domain --description "Test Project" test-token
openstack user create --domain Test-Domain --password 123456 test-user

4. Test script, provide the admin, test creds and required ids in the
test script.

# test-script.py
http://paste.openstack.org/show/801281/

5. Run the test script, python3 test-script.py <no of test repeations>

Run: python3 test-script.py 10

Expected final outcome: For the above script test-user with 'admin' role
should be allowed to get the default domain config and when the role is
switched to 'member' role get request response should be '403'. SCRIPT
OUTPUT: ACTION EXECUTIONS: 20 PASS: 20 FAIL: 0

Actual final outcome: Random failures,
http://paste.openstack.org/show/801283/

NOTE: Please increase the processes and/or threads count in
/etc/keystone/keystone-uwsgi-public.ini if issue not reproduced with the
above mentioned values.

Analysis: The above provided script yields successful result when there
are one or few processes and/or threads configured in  /etc/keystone
/keystone-uwsgi-public.ini. But, If the processes and/or threads config
is higher, and a user role is switched, requests are failing for
authorised users or requests are passing to users who are not
authorised(even after providing sleep after role switch). After enabling
debug_cache_backend = true in keystone.conf, I have observed that
keystone is using old request id and processing a part of the request
where the token is getting set in cache twice by different processes of
keystone but with different roles. I have observed the same in all-in-
one openstack-helm setup.

Will try to fetch and provide keystone logs when possible.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1909233

Title:
  Requests auth issue when there are multiple threads or processes

Status in OpenStack Identity (keystone):
  New

Bug description:
  Observed in train:

  Steps to reproduce:

  1. Setup devstack(stable/train)

  2. Make the below config changes and restart keystone.

  cat /etc/keystone/policy.json

  ```
  {
    "admin_required": "role:admin or is_admin:1",
    "identity:get_domain_config_default": "rule:admin_required"
  }
  ```

  cat /etc/keystone/keystone.conf

  ```
  [identity]
  domain_specific_drivers_enabled = true
  password_hash_rounds = 4
  driver = sql

  [assignment]
  driver = sql

  [role]
  driver = sql

  [resource]
  driver = sql

  [cache]
  memcache_servers = localhost:11211
  backend = dogpile.cache.memcached
  enabled = True

  [oslo_messaging_notifications]
  transport_url = rabbit://stackrabbit:password@localhost:5672/

  [DEFAULT]
  max_token_size = 16384
  debug = True
  logging_exception_prefix = ERROR %(name)s %(instance)s
  logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
  logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s
  logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}
  admin_endpoint = http://localhost/identity
  public_endpoint = http://localhost/identity

  [token]
  provider = fernet
  cache_time = 3600
  caching = true
  expiration = 43200

  [revoke]
  cache_time = 3600
  caching = true
  [role]
  cache_time = 3600
  caching = true

  [database]
  connection = mysql+pymysql://root:password@127.0.0.1/keystone?charset=utf8

  [fernet_tokens]
  key_repository = /etc/keystone/fernet-keys/

  [credential]
  key_repository = /etc/keystone/credential-keys/

  [security_compliance]
  unique_last_password_count = 2
  lockout_duration = 10
  lockout_failure_attempts = 2
  ```

  cat /etc/keystone/keystone-uwsgi-public.ini

  ```
  [uwsgi]
  chmod-socket = 666
  socket = /var/run/uwsgi/keystone-wsgi-public.socket
  lazy-apps = true
  add-header = Connection: close
  buffer-size = 65535
  hook-master-start = unix_signal:15 gracefully_kill_them_all
  thunder-lock = true
  plugins = http,python
  enable-threads = true
  worker-reload-mercy = 90
  exit-on-reload = false
  die-on-term = true
  master = true
  processes = 2
  threads = 2
  wsgi-file = /usr/local/bin/keystone-wsgi-public
  ```
  3. Create test creds.

  openstack domain create --description "Test Domain" test-domain 
  openstack project create --domain Test-Domain --description "Test Project" test-token
  openstack user create --domain Test-Domain --password 123456 test-user

  4. Test script, provide the admin, test creds and required ids in the
  test script.

  # test-script.py
  http://paste.openstack.org/show/801281/

  5. Run the test script, python3 test-script.py <no of test repeations>

  Run: python3 test-script.py 10

  Expected final outcome: For the above script test-user with 'admin'
  role should be allowed to get the default domain config and when the
  role is switched to 'member' role get request response should be
  '403'. SCRIPT OUTPUT: ACTION EXECUTIONS: 20 PASS: 20 FAIL: 0

  Actual final outcome: Random failures,
  http://paste.openstack.org/show/801283/

  NOTE: Please increase the processes and/or threads count in
  /etc/keystone/keystone-uwsgi-public.ini if issue not reproduced with
  the above mentioned values.

  Analysis: The above provided script yields successful result when
  there are one or few processes and/or threads configured in
  /etc/keystone/keystone-uwsgi-public.ini. But, If the processes and/or
  threads config is higher, and a user role is switched, requests are
  failing for authorised users or requests are passing to users who are
  not authorised(even after providing sleep after role switch). After
  enabling debug_cache_backend = true in keystone.conf, I have observed
  that keystone is using old request id and processing a part of the
  request where the token is getting set in cache twice by different
  processes of keystone but with different roles. I have observed the
  same in all-in-one openstack-helm setup.

  Will try to fetch and provide keystone logs when possible.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1909233/+subscriptions
  Thread Previous • Date Previous • • Thread Next