yahoo-eng-team team mailing list archive

[Raymond Pekowski <raymond_pekowski@xxxxxxxx>]

Sorry for the false alarm on this bug.  I didn't realize that my code
base had diverged from the OpenStack trunk.  Apparently, we have some
code changes to support image tagging from Mirantis.  I found the
problem, but I don't think it is applicable to how image tagging has
been officially implemented in OpenStack.

As for future bug reports, I will try to following the guidelines more
carefully.  Thanks for you patience.

** Changed in: glance
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1180836

Title:
  Glance GET /images/detail DetachedIntanceError on 'tags' attribute

Status in OpenStack Image Registry and Delivery Service (Glance):
  Invalid

Bug description:
  ---------- Forwarded message ----------
  From: Ray Pekowski <pekowski@xxxxxxxxx>
  Date: Wed, May 15, 2013 at 3:51 PM
  Subject: Glance GET /images/detail DetachedIntanceError on 'tags' attribute
  To: OpenStack Development Mailing List <openstack-dev@xxxxxxxxxxxxxxxxxxx>

  
  In my testing of Folsom in an HA environment including the use of haproxy fronting three "contoller" nodes and mysql using wsrep on each of those nodes I have been seeing an occasional HTTP 500 error on Glance image list operations.  I don't know if this is due to my HA environment.  I have only done a little troubleshooting and web searching on the problem.  I'll be a little lazy and post what I am seeing here in case it is a well known problem.  Here is a wireshark dump of the failing API flying across the wire:

  GET /images/detail?limit=20 HTTP/1.1
  Host: 10.49.56.250:9191
  Accept-Encoding: identity
  x-openstack-request-id: req-b1d626b9-fb91-45d6-8e7b-029caf143fd6
  x-auth-token: 3600d26b1a6348859141ac5cf4e80b9f
   
  HTTP/1.1 500 Internal Server Error
  Content-Type: text/plain
  Content-Length: 2839
  Date: Wed, 15 May 2013 17:39:57 GMT
  Connection: close
   
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
      result = self.application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 279, in __call__
      return self.app(env, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
      return self.func(req, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 326, in __call__
      response = req.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
      return self.func(req, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 533, in __call__
      request, **action_args)
    File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 550, in dispatch
      return method(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/glance/registry/api/v1/images.py", line 117, in detail
      image_dicts = [make_image_dict(i) for i in images]
    File "/usr/lib/python2.7/dist-packages/glance/registry/api/v1/images.py", line 447, in make_image_dict
      image_tags = image['tags']
    File "/usr/lib/python2.7/dist-packages/glance/db/sqlalchemy/models.py", line 77, in __getitem__
      return getattr(self, key)
    File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 168, in __get__
      return self.impl.get(instance_state(instance),dict_)
    File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 453, in get
      value = self.callable_(state, passive)
    File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 481, in _load_for_state
      (mapperutil.state_str(state), self.key)
  DetachedInstanceError: Parent instance <Image at 0x352f810> is not bound to a Session; lazy load operation of attribute 'tags' cannot proceed

  Any clue?

  Ray

  ---------- Forwarded message ----------
  From: Flavio Percoco <flavio@xxxxxxxxxx>
  Date: Thu, May 16, 2013 at 3:27 AM
  Subject: Re: [openstack-dev] Glance GET /images/detail DetachedIntanceError on 'tags' attribute
  To: OpenStack Development Mailing List <openstack-dev@xxxxxxxxxxxxxxxxxxx>

  
  On 15/05/13 15:51 -0500, Ray Pekowski wrote:

        In my testing of Folsom in an HA environment including the use of
        haproxy fronting three "contoller" nodes and mysql using wsrep on each
        of those nodes I have been seeing an occasional HTTP 500 error on
        Glance image list operations.  I don't know if this is due to my HA
        environment.  I have only done a little troubleshooting and web
        searching on the problem.  I'll be a little lazy and post what I am
        seeing here in case it is a well known problem.  Here is a wireshark
        dump of the failing API flying across the wire: 

        Any clue?


  Yep, that looks like a bug.
  TL;DR:

  The session attribute is removed from images dictionaries in the
  to_dict method[0] and it looks like that's happening before tags are
  loaded, which means that it won't be possible to load them since
  there's no session and sqlalchemy gets really mad. :D

  Please, file a bug for this.

  Thanks a lot.
  FF

  [0]
  https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L606

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1180836/+subscriptions