yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1584109] [NEW] Swift UI failures when deleting large numbers of objects

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Travis Tripp <1584109@xxxxxxxxxxxxxxxxxx>
Date: Fri, 20 May 2016 15:22:49 -0000
Reply-to: Bug 1584109 <1584109@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Failed to establish a new connection: [Errno 24] Too many open files',))

Basically I first create a bunch of containers objects and folders with this script:
 - https://github.com/openstack/searchlight/blob/master/test-scripts/generate-swift-data.py
 -  ./generate-swift-data.py 1000 5 10 
Then I add an empty nested sub-folder foo/bar/hello

Then I try to delete all.

I got an error 500

The browser screen freezes and I cannot do anything.

I manually refresh.

I then re-select all and click delete.

It works.

But I am able to make it do the above again.

Screen shot: http://imgur.com/a/4t4AV

ConnectionError: HTTPConnectionPool(host='192.168.200.200', port=8080): Max retries exceeded with url: /v1/AUTH_4bade81378e6428db0e896db77d68e02/scale_3/BL/object_669 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x108781990>: Failed to establish a new connection: [Errno 24] Too many open files',))
[20/May/2016 15:11:20] "DELETE /api/swift/containers/scale_3/object/BL/object_669 HTTP/1.1" 500 332
HTTP exception with no status/code
Traceback (most recent call last):
  File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/rest/utils.py", line 126, in _wrapped
    data = function(self, request, *args, **kw)
  File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/rest/swift.py", line 211, in delete
    api.swift.swift_delete_object(request, container, object_name)
  File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/swift.py", line 314, in swift_delete_object
    swift_api(request).delete_object(container_name, object_name)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1721, in delete_object
    response_dict=response_dict)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1369, in delete_object
    conn.request('DELETE', path, '', headers)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 401, in request
    files=files, **self.requests_args)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 384, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='192.168.200.200', port=8080): Max retries exceeded with url: /v1/AUTH_4bade81378e6428db0e896db77d68e02/scale_3/BL/object_667 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x1080f0cd0>: Failed to establish a new connection: [Errno 24] Too many open files',))
[20/May/2016 15:11:20] "DELETE /api/swift/containers/scale_3/object/BL/object_667 HTTP/1.1" 500 332

** Affects: horizon
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1584109

Title:
  Swift UI failures when deleting large numbers of objects

Status in OpenStack Dashboard (Horizon):
  New

Bug description:
  Failed to establish a new connection: [Errno 24] Too many open
  files',))

  Basically I first create a bunch of containers objects and folders with this script:
   - https://github.com/openstack/searchlight/blob/master/test-scripts/generate-swift-data.py
   -  ./generate-swift-data.py 1000 5 10 
  Then I add an empty nested sub-folder foo/bar/hello

  Then I try to delete all.

  I got an error 500

  The browser screen freezes and I cannot do anything.

  I manually refresh.

  I then re-select all and click delete.

  It works.

  But I am able to make it do the above again.

  Screen shot: http://imgur.com/a/4t4AV

  ConnectionError: HTTPConnectionPool(host='192.168.200.200', port=8080): Max retries exceeded with url: /v1/AUTH_4bade81378e6428db0e896db77d68e02/scale_3/BL/object_669 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x108781990>: Failed to establish a new connection: [Errno 24] Too many open files',))
  [20/May/2016 15:11:20] "DELETE /api/swift/containers/scale_3/object/BL/object_669 HTTP/1.1" 500 332
  HTTP exception with no status/code
  Traceback (most recent call last):
    File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/rest/utils.py", line 126, in _wrapped
      data = function(self, request, *args, **kw)
    File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/rest/swift.py", line 211, in delete
      api.swift.swift_delete_object(request, container, object_name)
    File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/swift.py", line 314, in swift_delete_object
      swift_api(request).delete_object(container_name, object_name)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1721, in delete_object
      response_dict=response_dict)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry
      service_token=self.service_token, **kwargs)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1369, in delete_object
      conn.request('DELETE', path, '', headers)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 401, in request
      files=files, **self.requests_args)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 384, in _request
      return self.request_session.request(*arg, **kwarg)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
      resp = self.send(prep, **send_kwargs)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
      r = adapter.send(request, **kwargs)
    File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
      raise ConnectionError(e, request=request)
  ConnectionError: HTTPConnectionPool(host='192.168.200.200', port=8080): Max retries exceeded with url: /v1/AUTH_4bade81378e6428db0e896db77d68e02/scale_3/BL/object_667 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x1080f0cd0>: Failed to establish a new connection: [Errno 24] Too many open files',))
  [20/May/2016 15:11:20] "DELETE /api/swift/containers/scale_3/object/BL/object_667 HTTP/1.1" 500 332

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1584109/+subscriptions
Follow ups

[Bug 1584109] Re: Swift UI failures when deleting large numbers of objects
From: OpenStack Infra, 2016-06-02