yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1582279] [NEW] Glance (swift) download broken after Kilo to Liberty Upgrade

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bjoern Teipel <bjoern.teipel@xxxxxxxxxxxxx>
Date: Mon, 16 May 2016 15:26:12 -0000
Reply-to: Bug 1582279 <1582279@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

After upgrading a Kilo environment to Liberty I noticed the issue that I
can not download all glance images backed by swift when they were using
Keystone v2 API and getting a 401 wrapped into a 404 swift error:

2016-05-13 17:45:51.253 4708 ERROR swiftclient [req-4a041b13-2081-45b4
-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure.
Authorization failed: The resource could not be found. (HTTP 404)
(Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)

2016-05-13 17:45:51.253 4708 ERROR swiftclient Traceback (most recent call last):
2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1413, in _retry
2016-05-13 17:45:51.253 4708 ERROR swiftclient     self.url, self.token = self.get_auth()
2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1367, in get_auth
2016-05-13 17:45:51.253 4708 ERROR swiftclient     timeout=self.timeout)
2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 490, in get_auth
2016-05-13 17:45:51.253 4708 ERROR swiftclient     auth_version=auth_version)
2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 418, in get_auth_keystone
2016-05-13 17:45:51.253 4708 ERROR swiftclient     raise ClientException('Authorization Failure. %s' % err)
2016-05-13 17:45:51.253 4708 ERROR swiftclient ClientException: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
2016-05-13 17:45:51.253 4708 ERROR swiftclient
2016-05-13 17:45:51.254 4708 WARNING glance.location [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Get image 95576f28-afed-4b63-93b4-1d07928930da data failed: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404).

2016-05-13 17:45:51.254 4708 ERROR glance.location [req-
4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Glance tried all active
locations to get data for image 95576f28-afed-4b63-93b4-1d07928930da but
all have failed.

2016-05-13 17:45:51.256 4708 INFO eventlet.wsgi.server [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Traceback (most recent call last):
  File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/eventlet/wsgi.py", line


On further debugging I noticed that either the swift client or glance
API tried to retrieve a token from a Keystone v3 URL on a v2 API
endpoint:

POST /v2.0/auth/tokens HTTP/1.1
Host: 1.2.3.4:5000
Content-Length: 254
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-keystoneclient
Connection: keep-alive
Content-Type: application/json

{"auth": {"scope": {"project": {"domain": {"id": "default"}, "name": "service"}}, "identity": {"password": {"user": {"domain": {"id": "default"}, "password": "xxxxx"
, "name": "glance"}}, "methods": ["password"]}}}

HTTP/1.1 404 Not Found
Date: Fri, 13 May 2016 22:54:23 GMT
Server: Apache
Vary: X-Auth-Token
x-openstack-request-id: req-22a9f19a-e72c-4f22-87d3-c3f25fb78a9f
Content-Length: 93
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/json

{"error": {"message": "The resource could not be found.", "code": 404,
"title": "Not Found"}}

which seems to be related to the config change swift_store_auth_version
from 2 in Kilo to 3 in Liberty.

Current Glance config

default_store = swift
stores = swift,http,cinder
swift_store_auth_version = 3
swift_store_auth_address = http://1.2.3.4:5000/v3
swift_store_auth_insecure = False
swift_store_user = service:glance
swift_store_key = xxxxx
swift_store_region = RegionOne
swift_store_container = glance_images
swift_store_endpoint_type = internalURL

To overcome this issue I updated all glance image_locations to point to
a V3 keystone API endpoint:

update image_locations set value=replace(value,
'5000/v2.0/glance_images', '5000/v3/glance_images') where value like
'swift%5000/v2.0/glance_images%' and status='active';

After that all image downloads started working.

As a additional information, Keystone was running in a stock
configuration, Fernet with SQL identity back end.

Interestingly I did another upgrade to Liberty trying to reproduce this issue but was not successful yet.
I have yet to determine what influenced the swift client to cause this behavior.

** Affects: glance
     Importance: Undecided
         Status: New

** Affects: openstack-ansible
     Importance: Undecided
     Assignee: Bjoern Teipel (bjoern-teipel)
         Status: New

** Changed in: openstack-ansible
     Assignee: (unassigned) => Bjoern Teipel (bjoern-teipel)

** Description changed:

  After upgrading a Kilo environment to Liberty I noticed the issue that I
  can not download all glance images backed by swift when they were using
  Keystone v2 API and getting a 401 wrapped into a 404 swift error:
  
- 2016-05-13 16:31:21.786 4359 ERROR swiftclient [req-8868d7aa-5885-43fe-9625-a28c94e8b59a a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure. Authorization failed:
-  The resource could not be found. (HTTP 404) (Request-ID: req-3e5dba07-7e5b-4e53-b896-df7148e801ca) (HTTP 404)
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient [req-4a041b13-2081-45b4
+ -a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
+ 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure.
+ Authorization failed: The resource could not be found. (HTTP 404)
+ (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
+ 
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient Traceback (most recent call last):
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1413, in _retry
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient     self.url, self.token = self.get_auth()
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1367, in get_auth
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient     timeout=self.timeout)
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 490, in get_auth
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient     auth_version=auth_version)
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 418, in get_auth_keystone
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient     raise ClientException('Authorization Failure. %s' % err)
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient ClientException: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
+ 2016-05-13 17:45:51.253 4708 ERROR swiftclient
+ 2016-05-13 17:45:51.254 4708 WARNING glance.location [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Get image 95576f28-afed-4b63-93b4-1d07928930da data failed: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404).
+ 
+ 2016-05-13 17:45:51.254 4708 ERROR glance.location [req-
+ 4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
+ 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Glance tried all active
+ locations to get data for image 95576f28-afed-4b63-93b4-1d07928930da but
+ all have failed.
+ 
+ 2016-05-13 17:45:51.256 4708 INFO eventlet.wsgi.server [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Traceback (most recent call last):
+   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/eventlet/wsgi.py", line
+ 
  
  On further debugging I noticed that either the swift client or glance
  API tried to retrieve a token from a Keystone v3 URL on a v2 API
  endpoint:
  
  POST /v2.0/auth/tokens HTTP/1.1
  Host: 1.2.3.4:5000
  Content-Length: 254
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-keystoneclient
  Connection: keep-alive
  Content-Type: application/json
  
  {"auth": {"scope": {"project": {"domain": {"id": "default"}, "name": "service"}}, "identity": {"password": {"user": {"domain": {"id": "default"}, "password": "xxxxx"
  , "name": "glance"}}, "methods": ["password"]}}}
  
- 
  HTTP/1.1 404 Not Found
  Date: Fri, 13 May 2016 22:54:23 GMT
  Server: Apache
  Vary: X-Auth-Token
  x-openstack-request-id: req-22a9f19a-e72c-4f22-87d3-c3f25fb78a9f
  Content-Length: 93
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: application/json
  
  {"error": {"message": "The resource could not be found.", "code": 404,
  "title": "Not Found"}}
  
- 
- which seems to be related to the config change swift_store_auth_version from 2 in Kilo to 3 in Liberty.
- 
+ which seems to be related to the config change swift_store_auth_version
+ from 2 in Kilo to 3 in Liberty.
  
  Current Glance config
  
  default_store = swift
  stores = swift,http,cinder
  swift_store_auth_version = 3
  swift_store_auth_address = http://1.2.3.4:5000/v3
  swift_store_auth_insecure = False
  swift_store_user = service:glance
  swift_store_key = xxxxx
  swift_store_region = RegionOne
  swift_store_container = glance_images
  swift_store_endpoint_type = internalURL
  
- 
- To overcome this issue I updated all glance image_locations to point to a V3 keystone API endpoint:
+ To overcome this issue I updated all glance image_locations to point to
+ a V3 keystone API endpoint:
  
  update image_locations set value=replace(value,
  '5000/v2.0/glance_images', '5000/v3/glance_images') where value like
  'swift%5000/v2.0/glance_images%' and status='active';
  
  After that all image downloads started working.
  
  As a additional information, Keystone was running in a stock
  configuration, Fernet with SQL identity back end.
  
- 
  Interestingly I did another upgrade to Liberty trying to reproduce this issue but was not successful yet.
  I have yet to determine what influenced the swift client to cause this behavior.

** Also affects: glance
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1582279

Title:
  Glance (swift) download broken after Kilo to Liberty Upgrade

Status in Glance:
  New
Status in openstack-ansible:
  New

Bug description:
  After upgrading a Kilo environment to Liberty I noticed the issue that
  I can not download all glance images backed by swift when they were
  using Keystone v2 API and getting a 401 wrapped into a 404 swift
  error:

  2016-05-13 17:45:51.253 4708 ERROR swiftclient [req-4a041b13-2081-45b4
  -a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
  7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure.
  Authorization failed: The resource could not be found. (HTTP 404)
  (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)

  2016-05-13 17:45:51.253 4708 ERROR swiftclient Traceback (most recent call last):
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1413, in _retry
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     self.url, self.token = self.get_auth()
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1367, in get_auth
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     timeout=self.timeout)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 490, in get_auth
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     auth_version=auth_version)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient   File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 418, in get_auth_keystone
  2016-05-13 17:45:51.253 4708 ERROR swiftclient     raise ClientException('Authorization Failure. %s' % err)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient ClientException: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
  2016-05-13 17:45:51.253 4708 ERROR swiftclient
  2016-05-13 17:45:51.254 4708 WARNING glance.location [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Get image 95576f28-afed-4b63-93b4-1d07928930da data failed: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404).

  2016-05-13 17:45:51.254 4708 ERROR glance.location [req-
  4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
  7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Glance tried all active
  locations to get data for image 95576f28-afed-4b63-93b4-1d07928930da
  but all have failed.

  2016-05-13 17:45:51.256 4708 INFO eventlet.wsgi.server [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Traceback (most recent call last):
    File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/eventlet/wsgi.py", line


  On further debugging I noticed that either the swift client or glance
  API tried to retrieve a token from a Keystone v3 URL on a v2 API
  endpoint:

  POST /v2.0/auth/tokens HTTP/1.1
  Host: 1.2.3.4:5000
  Content-Length: 254
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-keystoneclient
  Connection: keep-alive
  Content-Type: application/json

  {"auth": {"scope": {"project": {"domain": {"id": "default"}, "name": "service"}}, "identity": {"password": {"user": {"domain": {"id": "default"}, "password": "xxxxx"
  , "name": "glance"}}, "methods": ["password"]}}}

  HTTP/1.1 404 Not Found
  Date: Fri, 13 May 2016 22:54:23 GMT
  Server: Apache
  Vary: X-Auth-Token
  x-openstack-request-id: req-22a9f19a-e72c-4f22-87d3-c3f25fb78a9f
  Content-Length: 93
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: application/json

  {"error": {"message": "The resource could not be found.", "code": 404,
  "title": "Not Found"}}

  which seems to be related to the config change
  swift_store_auth_version from 2 in Kilo to 3 in Liberty.

  Current Glance config

  default_store = swift
  stores = swift,http,cinder
  swift_store_auth_version = 3
  swift_store_auth_address = http://1.2.3.4:5000/v3
  swift_store_auth_insecure = False
  swift_store_user = service:glance
  swift_store_key = xxxxx
  swift_store_region = RegionOne
  swift_store_container = glance_images
  swift_store_endpoint_type = internalURL

  To overcome this issue I updated all glance image_locations to point
  to a V3 keystone API endpoint:

  update image_locations set value=replace(value,
  '5000/v2.0/glance_images', '5000/v3/glance_images') where value like
  'swift%5000/v2.0/glance_images%' and status='active';

  After that all image downloads started working.

  As a additional information, Keystone was running in a stock
  configuration, Fernet with SQL identity back end.

  Interestingly I did another upgrade to Liberty trying to reproduce this issue but was not successful yet.
  I have yet to determine what influenced the swift client to cause this behavior.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1582279/+subscriptions
Follow ups

[Bug 1582279] Re: Glance (swift) download broken after Kilo to Liberty Upgrade
From: Bjoern Teipel, 2016-06-21