yahoo-eng-team team mailing list archive

[Bjoern Teipel <bjoern.teipel@xxxxxxxxxxxxx>]

** Changed in: openstack-ansible
       Status: In Progress => Fix Committed

** Changed in: openstack-ansible
       Status: Fix Committed => Fix Released

** Changed in: glance
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1582279

Title:
  Glance (swift) download broken after Kilo to Liberty Upgrade

Status in Glance:
  Invalid
Status in openstack-ansible:
  Fix Released

Bug description:
  After upgrading a Kilo environment to Liberty I noticed the issue that
  I can not download all glance images backed by swift when they were
  using Keystone v2 API and getting a 401 wrapped into a 404 swift
  error:

  2016-05-13 17:45:51.253 4708 ERROR swiftclient [req-4a041b13-2081-45b4
  -a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
  7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure.
  Authorization failed: The resource could not be found. (HTTP 404)
  (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)

  2016-05-13 17:45:51.253 4708 ERROR swiftclient Traceback (most recent call last):
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1413, in _retry
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     self.url, self.token = self.get_auth()
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1367, in get_auth
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     timeout=self.timeout)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 490, in get_auth
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     auth_version=auth_version)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 418, in get_auth_keystone
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     raise ClientException('Authorization Failure. %s' % err)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient ClientException: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient
  2016-05-13 17:45:51.254 4708 WARNING glance.location [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Get image 95576f28-afed-4b63-93b4-1d07928930da data failed: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404).

  2016-05-13 17:45:51.254 4708 ERROR glance.location [req-
  4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
  7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Glance tried all active
  locations to get data for image 95576f28-afed-4b63-93b4-1d07928930da
  but all have failed.

  2016-05-13 17:45:51.256 4708 INFO eventlet.wsgi.server [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Traceback (most recent call last):
    File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/eventlet/wsgi.py", line

  On further debugging I noticed that the swift client tried to retrieve
  a token from a Keystone v3 URL on a v2 API endpoint:

  POST /v2.0/auth/tokens HTTP/1.1
  Host: 1.2.3.4:5000
  Content-Length: 254
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-keystoneclient
  Connection: keep-alive
  Content-Type: application/json

  {"auth": {"scope": {"project": {"domain": {"id": "default"}, "name": "service"}}, "identity": {"password": {"user": {"domain": {"id": "default"}, "password": "xxxxx"
  , "name": "glance"}}, "methods": ["password"]}}}

  HTTP/1.1 404 Not Found
  Date: Fri, 13 May 2016 22:54:23 GMT
  Server: Apache
  Vary: X-Auth-Token
  x-openstack-request-id: req-22a9f19a-e72c-4f22-87d3-c3f25fb78a9f
  Content-Length: 93
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: application/json

  {"error": {"message": "The resource could not be found.", "code": 404,
  "title": "Not Found"}}

  which seems to be related to the config change
  swift_store_auth_version from 2 in Kilo to 3 in Liberty.

  Current Glance config

  default_store = swift
  stores = swift,http,cinder
  swift_store_auth_version = 3
  swift_store_auth_address = http://1.2.3.4:5000/v3
  swift_store_auth_insecure = False
  swift_store_user = service:glance
  swift_store_key = xxxxx
  swift_store_region = RegionOne
  swift_store_container = glance_images
  swift_store_endpoint_type = internalURL

  To overcome this issue I updated all glance image_locations to point
  to a V3 keystone API endpoint:

  update image_locations set value=replace(value,
  '5000/v2.0/glance_images', '5000/v3/glance_images') where value like
  'swift%5000/v2.0/glance_images%' and status='active';

  After that all image downloads started working.

  As a additional information, Keystone was running in a stock
  configuration, Fernet with SQL identity back end.

  Interestingly I did another upgrade to Liberty trying to reproduce this issue but was not successful yet.
  I have yet to determine what influenced the swift client to cause this behavior.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1582279/+subscriptions