yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1669805] [NEW] rally job failing in gate due to "Quota for tenant X could not be found" Error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Daniel Alvarez <1669805@xxxxxxxxxxxxxxxxxx>
Date: Fri, 03 Mar 2017 14:37:38 -0000
Reply-to: Bug 1669805 <1669805@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Rally job is failing in the gate due to the following error during
cleanup [0]:

2017-03-03 13:14:56.897549 | 2017-03-03 13:14:56.897 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     rutils.retry(resource._max_attempts, resource.delete)
2017-03-03 13:14:56.899015 | 2017-03-03 13:14:56.898 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/opt/stack/new/rally/rally/common/utils.py", line 223, in retry
2017-03-03 13:14:56.900375 | 2017-03-03 13:14:56.900 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     return func(*args, **kwargs)
2017-03-03 13:14:56.901935 | 2017-03-03 13:14:56.901 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/opt/stack/new/rally/rally/plugins/openstack/cleanup/resources.py", line 472, in delete
2017-03-03 13:14:56.903254 | 2017-03-03 13:14:56.902 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     self._manager().delete_quota(self.tenant_uuid)
2017-03-03 13:14:56.904746 | 2017-03-03 13:14:56.904 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator
2017-03-03 13:14:56.906156 | 2017-03-03 13:14:56.905 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     return wrapped(*args, **kwargs)
2017-03-03 13:14:56.907779 | 2017-03-03 13:14:56.907 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 742, in delete_quota
2017-03-03 13:14:56.909166 | 2017-03-03 13:14:56.908 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     return self.delete(self.quota_path % (project_id))
2017-03-03 13:14:56.910455 | 2017-03-03 13:14:56.910 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in delete
2017-03-03 13:14:56.911722 | 2017-03-03 13:14:56.911 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     headers=headers, params=params)
2017-03-03 13:14:56.913068 | 2017-03-03 13:14:56.912 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request
2017-03-03 13:14:56.914760 | 2017-03-03 13:14:56.914 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     headers=headers, params=params)
2017-03-03 13:14:56.916094 | 2017-03-03 13:14:56.915 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 301, in do_request
2017-03-03 13:14:56.917732 | 2017-03-03 13:14:56.917 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     self._handle_fault_response(status_code, replybody, resp)
2017-03-03 13:14:56.919266 | 2017-03-03 13:14:56.918 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 276, in _handle_fault_response
2017-03-03 13:14:56.920699 | 2017-03-03 13:14:56.920 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     exception_handler_v20(status_code, error_body)
2017-03-03 13:14:56.922342 | 2017-03-03 13:14:56.922 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
2017-03-03 13:14:56.923773 | 2017-03-03 13:14:56.923 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     request_ids=request_ids)
2017-03-03 13:14:56.925720 | 2017-03-03 13:14:56.925 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager NotFound: Quota for tenant 82ee0ba1b6534f958d1acd2f717b5c3d could not be found.

Seems like we're hitting this errors from 1AM today (03/03/2017) [1]

[0] http://logs.openstack.org/91/431691/30/check/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/ab3471c/console.html#_2017-03-03_13_14_56_925720
[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20%5C%22gate-rally-dsvm-neutron-neutron-ubuntu-xenial%5C%22%20AND%20message%3A%20%5C%22Quota%20for%20tenant%5C%22

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1669805

Title:
  rally job failing in gate due to "Quota for tenant X could not be
  found" Error

Status in neutron:
  New

Bug description:
  Rally job is failing in the gate due to the following error during
  cleanup [0]:

  2017-03-03 13:14:56.897549 | 2017-03-03 13:14:56.897 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     rutils.retry(resource._max_attempts, resource.delete)
  2017-03-03 13:14:56.899015 | 2017-03-03 13:14:56.898 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/opt/stack/new/rally/rally/common/utils.py", line 223, in retry
  2017-03-03 13:14:56.900375 | 2017-03-03 13:14:56.900 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     return func(*args, **kwargs)
  2017-03-03 13:14:56.901935 | 2017-03-03 13:14:56.901 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/opt/stack/new/rally/rally/plugins/openstack/cleanup/resources.py", line 472, in delete
  2017-03-03 13:14:56.903254 | 2017-03-03 13:14:56.902 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     self._manager().delete_quota(self.tenant_uuid)
  2017-03-03 13:14:56.904746 | 2017-03-03 13:14:56.904 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator
  2017-03-03 13:14:56.906156 | 2017-03-03 13:14:56.905 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     return wrapped(*args, **kwargs)
  2017-03-03 13:14:56.907779 | 2017-03-03 13:14:56.907 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 742, in delete_quota
  2017-03-03 13:14:56.909166 | 2017-03-03 13:14:56.908 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     return self.delete(self.quota_path % (project_id))
  2017-03-03 13:14:56.910455 | 2017-03-03 13:14:56.910 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in delete
  2017-03-03 13:14:56.911722 | 2017-03-03 13:14:56.911 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     headers=headers, params=params)
  2017-03-03 13:14:56.913068 | 2017-03-03 13:14:56.912 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request
  2017-03-03 13:14:56.914760 | 2017-03-03 13:14:56.914 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     headers=headers, params=params)
  2017-03-03 13:14:56.916094 | 2017-03-03 13:14:56.915 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 301, in do_request
  2017-03-03 13:14:56.917732 | 2017-03-03 13:14:56.917 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     self._handle_fault_response(status_code, replybody, resp)
  2017-03-03 13:14:56.919266 | 2017-03-03 13:14:56.918 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 276, in _handle_fault_response
  2017-03-03 13:14:56.920699 | 2017-03-03 13:14:56.920 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     exception_handler_v20(status_code, error_body)
  2017-03-03 13:14:56.922342 | 2017-03-03 13:14:56.922 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
  2017-03-03 13:14:56.923773 | 2017-03-03 13:14:56.923 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager     request_ids=request_ids)
  2017-03-03 13:14:56.925720 | 2017-03-03 13:14:56.925 | 2017-03-03 13:14:56.886 6099 ERROR rally.plugins.openstack.cleanup.manager NotFound: Quota for tenant 82ee0ba1b6534f958d1acd2f717b5c3d could not be found.

  Seems like we're hitting this errors from 1AM today (03/03/2017) [1]

  [0] http://logs.openstack.org/91/431691/30/check/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/ab3471c/console.html#_2017-03-03_13_14_56_925720
  [1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20%5C%22gate-rally-dsvm-neutron-neutron-ubuntu-xenial%5C%22%20AND%20message%3A%20%5C%22Quota%20for%20tenant%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1669805/+subscriptions
Follow ups

[Bug 1669805] Re: rally job failing in gate due to "Quota for tenant X could not be found" Error
From: Ihar Hrachyshka, 2017-08-22