yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1914665] Re: Cinder Multistore job hits Cinder Quota error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Abhishek Kekane <1914665@xxxxxxxxxxxxxxxxxx>
Date: Thu, 17 Jun 2021 07:29:46 -0000
Reply-to: Bug 1914665 <1914665@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: glance
       Status: In Progress => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1914665

Title:
  Cinder Multistore job hits Cinder Quota error

Status in Glance:
  Won't Fix

Bug description:
  Noticed during a cinder multistore test run, we hit a quota not found
  error. It looks like we don't handle this well, which causes nova to
  see a 503: Proxy Error. I dunno if there's anything better can do than
  raise a 5xx, but we should probably explain in the error what happened
  when we know, as we clearly do here.

  From this:

  https://cbff25b854b00bc0ff99-8ce5690b0835baabd00baac02d43f418.ssl.cf5.rackcdn.com/770629/5/check
  /glance-multistore-cinder-
  import/7c71502/controller/logs/screen-g-api.txt

  this log text (see the end):

  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi Traceback (most recent call last):
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/wsgi.py", line 1347, in __call__
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     action_result = self.dispatch(self.controller, action,
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/wsgi.py", line 1391, in dispatch
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return method(*args, **kwargs)
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/common/utils.py", line 416, in wrapped
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/v2/image_data.py", line 299, in upload
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self._restore(image_repo, image)
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self.force_reraise()
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     raise value
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/v2/image_data.py", line 164, in upload
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/domain/proxy.py", line 208, in set_data
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self.base.set_data(data, size, backend=backend, set_active=set_active)
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/notifier.py", line 501, in set_data
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self.force_reraise()
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     raise value
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/notifier.py", line 447, in set_data
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self.repo.set_data(data, size, backend=backend,
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/api/policy.py", line 206, in set_data
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/quota/__init__.py", line 322, in set_data
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self.image.set_data(data, size=size, backend=backend,
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/location.py", line 567, in set_data
  Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     self._upload_to_store(data, verifier, backend, size)
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/opt/stack/glance/glance/location.py", line 458, in _upload_to_store
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     multihash, loc_meta) = self.store_api.add_with_multihash(
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/multi_backend.py", line 398, in add_with_multihash
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return store_add_to_backend_with_multihash(
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     (location, size, checksum, multihash, metadata) = store.add(
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/driver.py", line 279, in add_adapter
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     metadata_dict) = store_add_fun(*args, **kwargs)
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/capabilities.py", line 176, in op_checker
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/cinder.py", line 852, in add
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     volume = client.volumes.create(size_gb, name=name, metadata=metadata,
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/cinderclient/v3/volumes.py", line 125, in create
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return self._create('/volumes', body, 'volume')
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/cinderclient/base.py", line 302, in _create
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     resp, body = self.api.client.post(url, body=body)
  Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 485, in post
  Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     return self._cs_request(url, 'POST', **kwargs)
  Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 438, in _cs_request
  Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     resp, body = self.request(url, method, **kwargs)
  Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 420, in request
  Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi     raise exceptions.from_response(resp, body)
  Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 devstack@g-api.service[93292]: ERROR glance.common.wsgi cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5a3e84d0-70fb-425c-b8b2-fdddc14b7bf1)

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1914665/+subscriptions
References

[Bug 1914665] [NEW] Cinder Multistore job hits Cinder Quota error
From: Dan Smith, 2021-02-04