yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1733274] [NEW] Second stage call (re-staging) fails with 500 Internal server error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Abhishek Kekane <akekane@xxxxxxxxxx>
Date: Mon, 20 Nov 2017 06:27:09 -0000
Reply-to: Bug 1733274 <1733274@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When user calls stage command again then it fails with 500 internal server error.
This is because after stage call image status changes to 'uploading' and for second call it again tries to set status to 'uploading' and fails with 'InvalidImageStatusTransition: Image status transition from uploading to uploading is not allowed' exception which is not caught at the controller side.

Steps to reproduce:
1. Create image
   $ glance image-create --container-format ami --disk-format ami --name cirros_image
2. Add image to staging area using stage call
   $ glance image-stage <IMAGE_ID>
3. Again run the same call
   $ glance image-stage <IMAGE_ID>

Output:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 699, in main
    OpenStackImagesShell().main(argv)
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 603, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 410, in do_image_stage
    gc.images.stage(args.id, image_data, args.size)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 545, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 252, in stage
    u_url=url)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 545, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 232, in upload
    resp, body = self.http_client.put(url, headers=hdrs, data=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 313, in put
    return self.request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 349, in request
    return self._handle_response(resp)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 98, in _handle_response
    raise exc.from_response(resp, resp.content)
HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)


g-api logs:
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi [None req-c6c78013-c211-492a-80df-694119e4b76f admin admin] Caught error: Image status transition from uploading to uploading is not allowed: InvalidImageStatusTransition: Image status transition from uploading to uploading is not allowed
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi Traceback (most recent call last):
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/wsgi.py", line 1222, in __call__
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     request, **action_args)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/wsgi.py", line 1261, in dispatch
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return method(*args, **kwargs)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/utils.py", line 363, in wrapped
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/v2/image_data.py", line 343, in stage
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     self._restore(image_repo, image)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     self.force_reraise()
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/v2/image_data.py", line 299, in stage
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     image.status = 'uploading'
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/__init__.py", line 151, in status
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     raise e
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi InvalidImageStatusTransition: Image status transition from uploading to uploading is not allowed
Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi

** Affects: glance
     Importance: Undecided
     Assignee: Abhishek Kekane (abhishek-kekane)
         Status: New

** Changed in: glance
     Assignee: (unassigned) => Abhishek Kekane (abhishek-kekane)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1733274

Title:
  Second stage call (re-staging) fails with 500 Internal server error

Status in Glance:
  New

Bug description:
  When user calls stage command again then it fails with 500 internal server error.
  This is because after stage call image status changes to 'uploading' and for second call it again tries to set status to 'uploading' and fails with 'InvalidImageStatusTransition: Image status transition from uploading to uploading is not allowed' exception which is not caught at the controller side.

  Steps to reproduce:
  1. Create image
     $ glance image-create --container-format ami --disk-format ami --name cirros_image
  2. Add image to staging area using stage call
     $ glance image-stage <IMAGE_ID>
  3. Again run the same call
     $ glance image-stage <IMAGE_ID>

  Output:
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 699, in main
      OpenStackImagesShell().main(argv)
    File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 603, in main
      args.func(client, args)
    File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 410, in do_image_stage
      gc.images.stage(args.id, image_data, args.size)
    File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 545, in inner
      return RequestIdProxy(wrapped(*args, **kwargs))
    File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 252, in stage
      u_url=url)
    File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 545, in inner
      return RequestIdProxy(wrapped(*args, **kwargs))
    File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 232, in upload
      resp, body = self.http_client.put(url, headers=hdrs, data=body)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 313, in put
      return self.request(url, 'PUT', **kwargs)
    File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 349, in request
      return self._handle_response(resp)
    File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 98, in _handle_response
      raise exc.from_response(resp, resp.content)
  HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

  
  g-api logs:
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi [None req-c6c78013-c211-492a-80df-694119e4b76f admin admin] Caught error: Image status transition from uploading to uploading is not allowed: InvalidImageStatusTransition: Image status transition from uploading to uploading is not allowed
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi Traceback (most recent call last):
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/wsgi.py", line 1222, in __call__
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     request, **action_args)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/wsgi.py", line 1261, in dispatch
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return method(*args, **kwargs)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/utils.py", line 363, in wrapped
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/v2/image_data.py", line 343, in stage
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     self._restore(image_repo, image)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     self.force_reraise()
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/v2/image_data.py", line 299, in stage
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     image.status = 'uploading'
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 23, in set_attr
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     return setattr(getattr(self, target), attr, value)
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/__init__.py", line 151, in status
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi     raise e
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi InvalidImageStatusTransition: Image status transition from uploading to uploading is not allowed
  Nov 20 06:16:54 devstack devstack@g-api.service[15827]: ERROR glance.common.wsgi

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1733274/+subscriptions
Follow ups

[Bug 1733274] Re: Second stage call (re-staging) fails with 500 Internal server error
From: OpenStack Infra, 2017-12-11