yahoo-eng-team team mailing list archive

[Jordan Pittier <jordan.pittier@xxxxxxxxxxxxx>]

Public bug reported:

Hi,
Sometime I get the following stack trace in Glance-API : 

GET /v1/images/42646b2b-cf0b-4b15-b011-19d0a6880ffb HTTP/1.1" 200 4970175 2.403391
    for chunk in image_iter:
  File "/opt/stack/new/glance/glance/api/middleware/cache.py", line 281, in get_from_cache
    yield chunk
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 373, in open_for_read
    with self.get_db() as db:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 391, in get_db
    conn.execute('PRAGMA synchronous = NORMAL')
  File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 77, in execute
    return self._timeout(lambda: sqlite3.Connection.execute(
  File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 74, in _timeout
    sleep(0.05)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 31, in sleep
    hub.switch()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
Timeout: 2 seconds

It happens also from time to time in the Gate. See the following
logstash request :

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwicmV0dXJuIHNlbGYuZ3JlZW5sZXQuc3dpdGNoKClcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1nLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwOTEyNjQ1NjU3NywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==


This caused the gate failure of : http://logs.openstack.org/22/116622/2/check/check-tempest-dsvm-postgres-full/f079ef9/logs/screen-g-api.txt.gz?  (wait for a full load of this page then grep "Timeout: 2 seconds")

Sorry for not being able to investigate more.

Jordan

** Affects: glance
     Importance: Undecided
         Status: New

** Affects: tempest
     Importance: Undecided
         Status: New

** Also affects: tempest
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1362048

Title:
  SQLite timeout in glance image_cache

Status in OpenStack Image Registry and Delivery Service (Glance):
  New
Status in Tempest:
  New

Bug description:
  Hi,
  Sometime I get the following stack trace in Glance-API : 

  GET /v1/images/42646b2b-cf0b-4b15-b011-19d0a6880ffb HTTP/1.1" 200 4970175 2.403391
      for chunk in image_iter:
    File "/opt/stack/new/glance/glance/api/middleware/cache.py", line 281, in get_from_cache
      yield chunk
    File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
      self.gen.next()
    File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 373, in open_for_read
      with self.get_db() as db:
    File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
      return self.gen.next()
    File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 391, in get_db
      conn.execute('PRAGMA synchronous = NORMAL')
    File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 77, in execute
      return self._timeout(lambda: sqlite3.Connection.execute(
    File "/opt/stack/new/glance/glance/image_cache/drivers/sqlite.py", line 74, in _timeout
      sleep(0.05)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 31, in sleep
      hub.switch()
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
      return self.greenlet.switch()
  Timeout: 2 seconds

  It happens also from time to time in the Gate. See the following
  logstash request :

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwicmV0dXJuIHNlbGYuZ3JlZW5sZXQuc3dpdGNoKClcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1nLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwOTEyNjQ1NjU3NywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

  
  This caused the gate failure of : http://logs.openstack.org/22/116622/2/check/check-tempest-dsvm-postgres-full/f079ef9/logs/screen-g-api.txt.gz?  (wait for a full load of this page then grep "Timeout: 2 seconds")

  Sorry for not being able to investigate more.

  Jordan

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1362048/+subscriptions