yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1702230] Re: fernet token fails with keystone HA

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lance Bragstad <lbragstad@xxxxxxxxx>
Date: Tue, 18 Jul 2017 19:29:54 -0000
Reply-to: Bug 1702230 <1702230@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
For what it's worth - if the concept described in comment #5 was broken,
we'd see a bunch of issues and failures in the gate with projects that
deploy keystone in HA configurations by default (e.g. the openstack-
ansible community does this with some of their gate tests).

I'm going to mark this as invalid for now. Please feel free to continue
using this report for discussion or questions. You can also swing by
#openstack-keystone on Freenode and I'd be happy to help explain fernet
key rotation in greater detail.

** Changed in: keystone
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1702230

Title:
  fernet token fails with keystone HA

Status in OpenStack Identity (keystone):
  Invalid

Bug description:
  I have newton release in my environment with keystone provider fernet on Centos 7.
  When I am trying to upload image to glance it is failing with below message.
  glance-api.log:

  2017-07-04 02:03:28.771 8105 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "This is not a recognized Fernet token gAAAAABZWy-wVlXrPlfa6_6PsyXky45ejM06Yt04UTLN6I51-CDT-kio83aIM00Xd6XL0bRzdwY8-Ks1L8SJD-xsGKyf-XUtm5TzskxpmhPXi0vDBYnM7pH2MnopcHW3RYH7YEUnqLIHGUVoBS5MGxgmSsgv0w20onikCu7xD-kDtR1gDOdryPU=", "code": 404, "title": "Not Found"}}
  2017-07-04 02:03:28.772 8105 WARNING keystonemiddleware.auth_token [-] Authorization failed for token

  Below is the debug message I am getting. 
  # openstack image list --debug
  START with options: [u'image', u'list', u'--debug']
  options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', auth_type='', auth_url='http://192.168.27.23:35357/v3', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='3', os_image_api_version='2', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
  Auth plugin password selected
  auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': 'gaian', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
  defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
  cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', u'orchestration_api_version': u'1', u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.27.23:35357/v3', 'password': '***', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
  compute API version 2, cmd group openstack.compute.v2
  network API version 2, cmd group openstack.network.v2
  image API version 2, cmd group openstack.image.v2
  volume API version 2, cmd group openstack.volume.v2
  identity API version 3, cmd group openstack.identity.v3
  object_store API version 1, cmd group openstack.object_store.v1
  neutronclient API version 2, cmd group openstack.neutronclient.v2
  Auth plugin password selected
  auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
  Auth plugin password selected
  auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
  command: image list -> openstackclient.image.v2.image.ListImage
  Using auth plugin: password
  Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.27.23:35357/v3', 'password': '***', 'project_domain_name': 'Default'}
  Get auth_ref
  REQ: curl -g -i -X GET http://192.168.27.23:35357/v3 -H "Accept: application/json" -H "User-Agent: osc-lib keystoneauth1/2.12.2 python-requests/2.11.1 CPython/2.7.5"
  Starting new HTTP connection (1): 192.168.27.23
  "GET /v3 HTTP/1.1" 200 253
  RESP: [200] Date: Tue, 04 Jul 2017 06:03:27 GMT Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token x-openstack-request-id: req-14636b59-b27e-48c4-8bb3-609136821eab Content-Length: 253 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
  RESP BODY: {"version": {"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://192.168.27.23:35357/v3/";, "rel": "self"}]}}

  Making authentication request to http://192.168.27.23:35357/v3/auth/tokens
  "POST /v3/auth/tokens HTTP/1.1" 201 1629
  {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8cb83746f16f4c0c86d578aef08d2909", "name": "admin"}], "expires_at": "2017-07-04T07:03:28.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cde189882de44539afb4247aa656acf1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.27.23:35357/v3/";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fcd48c87a884180a118c35ab84e0671"}, {"url": "http://192.168.27.23:5000/v3/";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "76ab1355510f4d20a20e3987511223c8"}, {"url": "http://192.168.27.23:35357/v3/";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d905017f8a9f4917acc576da8f6b8717"}], "type": "identity", "id": "61d50fd50a7f4d68aa2f7c95e51f4b51", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.27.23:9292";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "021fba1895ba423aa2693b9033184b87"}, {"url": "http://192.168.27.23:9292";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "86a5a30b748c4506b3fa763a58d6199e"}, {"url": "http://192.168.27.23:9292";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "91f25b85569445d6b7c69cff4e6b6b55"}], "type": "image", "id": "e7b681e057134e398a581e01ad99841d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "5796fd4f78134259865d83effb65521e", "name": "admin"}, "audit_ids": ["zQ6hqEQMQ4mghdQkGeAHCQ"], "issued_at": "2017-07-04T06:03:28.000000Z"}}
  run(Namespace(columns=[], formatter='table', limit=None, long=False, marker=None, max_width=0, noindent=False, page_size=None, private=False, property=None, public=False, quote_mode='nonnumeric', shared=False, sort=None))
  Instantiating image client: <class 'glanceclient.v2.client.Client'>
  Making authentication request to http://192.168.27.23:35357/v3/auth/tokens
  "POST /v3/auth/tokens HTTP/1.1" 201 1629
  {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8cb83746f16f4c0c86d578aef08d2909", "name": "admin"}], "expires_at": "2017-07-04T07:03:28.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cde189882de44539afb4247aa656acf1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.27.23:35357/v3/";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fcd48c87a884180a118c35ab84e0671"}, {"url": "http://192.168.27.23:5000/v3/";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "76ab1355510f4d20a20e3987511223c8"}, {"url": "http://192.168.27.23:35357/v3/";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d905017f8a9f4917acc576da8f6b8717"}], "type": "identity", "id": "61d50fd50a7f4d68aa2f7c95e51f4b51", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.27.23:9292";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "021fba1895ba423aa2693b9033184b87"}, {"url": "http://192.168.27.23:9292";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "86a5a30b748c4506b3fa763a58d6199e"}, {"url": "http://192.168.27.23:9292";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "91f25b85569445d6b7c69cff4e6b6b55"}], "type": "image", "id": "e7b681e057134e398a581e01ad99841d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "5796fd4f78134259865d83effb65521e", "name": "admin"}, "audit_ids": ["zCiqSllWQwuU87EU1eUmsA"], "issued_at": "2017-07-04T06:03:28.000000Z"}}
  Instantiating image api: <class 'openstackclient.api.image_v2.APIv2'>
  REQ: curl -g -i -X GET http://192.168.27.23:9292/v2/images -H "User-Agent: osc-lib keystoneauth1/2.12.2 python-requests/2.11.1 CPython/2.7.5" -H "X-Auth-Token: {SHA1}bd691c601e36cb572f7dca23c370cac02cc3dbfa"
  Starting new HTTP connection (1): 192.168.27.23
  "GET /v2/images HTTP/1.1" 401 253
  RESP: [401] Content-Length: 253 Content-Type: text/plain; charset=UTF-8 Www-Authenticate: Keystone uri='http://192.168.27.23:5000' Date: Tue, 04 Jul 2017 06:03:28 GMT Connection: keep-alive
  RESP BODY: 401 Unauthorized

  This server could not verify that you are authorized to access the
  document you requested. Either you supplied the wrong credentials
  (e.g., bad password), or your browser does not understand how to
  supply the credentials required.


  Making authentication request to http://192.168.27.23:35357/v3/auth/tokens
  "POST /v3/auth/tokens HTTP/1.1" 201 1629
  {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8cb83746f16f4c0c86d578aef08d2909", "name": "admin"}], "expires_at": "2017-07-04T07:03:29.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cde189882de44539afb4247aa656acf1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.27.23:35357/v3/";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fcd48c87a884180a118c35ab84e0671"}, {"url": "http://192.168.27.23:5000/v3/";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "76ab1355510f4d20a20e3987511223c8"}, {"url": "http://192.168.27.23:35357/v3/";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d905017f8a9f4917acc576da8f6b8717"}], "type": "identity", "id": "61d50fd50a7f4d68aa2f7c95e51f4b51", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.27.23:9292";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "021fba1895ba423aa2693b9033184b87"}, {"url": "http://192.168.27.23:9292";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "86a5a30b748c4506b3fa763a58d6199e"}, {"url": "http://192.168.27.23:9292";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "91f25b85569445d6b7c69cff4e6b6b55"}], "type": "image", "id": "e7b681e057134e398a581e01ad99841d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "5796fd4f78134259865d83effb65521e", "name": "admin"}, "audit_ids": ["PRP42NBYQ7iU8RcSOc1ybQ"], "issued_at": "2017-07-04T06:03:29.000000Z"}}
  "GET /v2/images HTTP/1.1" 401 253
  RESP: [401] Content-Length: 253 Content-Type: text/plain; charset=UTF-8 Www-Authenticate: Keystone uri='http://192.168.27.23:5000' Date: Tue, 04 Jul 2017 06:03:29 GMT Connection: keep-alive
  RESP BODY: 401 Unauthorized

  This server could not verify that you are authorized to access the
  document you requested. Either you supplied the wrong credentials
  (e.g., bad password), or your browser does not understand how to
  supply the credentials required.


  Request returned failure status: 401
  Unauthorized (HTTP 401)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/usr/lib/python2.7/site-packages/cliff/display.py", line 100, in run
      column_names, data = self.take_action(parsed_args)
    File "/usr/lib/python2.7/site-packages/openstackclient/image/v2/image.py", line 518, in take_action
      data = image_client.api.image_list(**kwargs)
    File "/usr/lib/python2.7/site-packages/openstackclient/api/image_v2.py", line 74, in image_list
      return self.list(url, **filter)['images']
    File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 198, in list
      params=params,
    File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 84, in _request
      return session.request(url, method, **kwargs)
    File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request
      resp = super(TimingSession, self).request(url, method, **kwargs)
    File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
      return wrapped(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 570, in request
      raise exceptions.from_response(resp, method, url)
  Unauthorized: Unauthorized (HTTP 401)
  clean_up ListImage: Unauthorized (HTTP 401)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
      ret_val = super(OpenStackShell, self).run(argv)
    File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run
      result = self.run_subcommand(remainder)
    File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
      ret_value = super(OpenStackShell, self).run_subcommand(argv)
    File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/usr/lib/python2.7/site-packages/cliff/display.py", line 100, in run
      column_names, data = self.take_action(parsed_args)
    File "/usr/lib/python2.7/site-packages/openstackclient/image/v2/image.py", line 518, in take_action
      data = image_client.api.image_list(**kwargs)
    File "/usr/lib/python2.7/site-packages/openstackclient/api/image_v2.py", line 74, in image_list
      return self.list(url, **filter)['images']
    File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 198, in list
      params=params,
    File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 84, in _request
      return session.request(url, method, **kwargs)
    File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request
      resp = super(TimingSession, self).request(url, method, **kwargs)
    File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
      return wrapped(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 570, in request
      raise exceptions.from_response(resp, method, url)
  Unauthorized: Unauthorized (HTTP 401)

  END return value: 1

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1702230/+subscriptions
References

[Bug 1702230] [NEW] fernet token fails with keystone HA
From: narasimha18sv, 2017-07-04