← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #66091

[Bug 1706405] [NEW] ceph jobs failing to upload images in pike due to "AttributeError: 'NoneType' object has no attribute 'Rados'" on py35

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm-
py35-full-devstack-plugin-ceph-ubuntu-xenial-
nv/6066e44/logs/screen-g-api.txt.gz?level=TRACE#_Jul_25_14_37_19_667434

Jul 25 14:37:19.667434 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.api.v2.image_data [None req-19231148-54d7-4337-be6b-a9a73c34b617 admin admin] Failed to upload image data due to internal error
Jul 25 14:37:19.704732 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi [None req-19231148-54d7-4337-be6b-a9a73c34b617 admin admin] Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'
Jul 25 14:37:19.704898 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi Traceback (most recent call last):
Jul 25 14:37:19.705014 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/wsgi.py", line 1155, in __call__
Jul 25 14:37:19.705130 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     request, **action_args)
Jul 25 14:37:19.705242 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/wsgi.py", line 1194, in dispatch
Jul 25 14:37:19.705345 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return method(*args, **kwargs)
Jul 25 14:37:19.705445 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/utils.py", line 363, in wrapped
Jul 25 14:37:19.705546 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
Jul 25 14:37:19.705646 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 269, in upload
Jul 25 14:37:19.705751 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self._restore(image_repo, image)
Jul 25 14:37:19.705850 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 25 14:37:19.705960 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.force_reraise()
Jul 25 14:37:19.706062 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 25 14:37:19.706174 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
Jul 25 14:37:19.706274 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/six.py", line 686, in reraise
Jul 25 14:37:19.706373 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     raise value
Jul 25 14:37:19.706472 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 134, in upload
Jul 25 14:37:19.706572 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     image.set_data(data, size)
Jul 25 14:37:19.706670 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/domain/proxy.py", line 195, in set_data
Jul 25 14:37:19.706787 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.base.set_data(data, size)
Jul 25 14:37:19.706914 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/notifier.py", line 480, in set_data
Jul 25 14:37:19.707029 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
Jul 25 14:37:19.707133 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 25 14:37:19.707233 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.force_reraise()
Jul 25 14:37:19.707345 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 25 14:37:19.707459 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
Jul 25 14:37:19.707554 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/six.py", line 686, in reraise
Jul 25 14:37:19.707649 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     raise value
Jul 25 14:37:19.707752 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/notifier.py", line 427, in set_data
Jul 25 14:37:19.707848 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.repo.set_data(data, size)
Jul 25 14:37:19.707942 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/policy.py", line 192, in set_data
Jul 25 14:37:19.708036 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
Jul 25 14:37:19.708138 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/quota/__init__.py", line 304, in set_data
Jul 25 14:37:19.708238 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.image.set_data(data, size=size)
Jul 25 14:37:19.708333 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/location.py", line 439, in set_data
Jul 25 14:37:19.708432 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     verifier=verifier)
Jul 25 14:37:19.708536 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/backend.py", line 453, in add_to_backend
Jul 25 14:37:19.708635 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     verifier)
Jul 25 14:37:19.708734 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/backend.py", line 426, in store_add_to_backend
Jul 25 14:37:19.708833 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     verifier=verifier)
Jul 25 14:37:19.708938 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/capabilities.py", line 225, in op_checker
Jul 25 14:37:19.709037 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
Jul 25 14:37:19.709142 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/_drivers/rbd.py", line 454, in add
Jul 25 14:37:19.709237 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     rados_id=self.user) as conn:
Jul 25 14:37:19.709337 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
Jul 25 14:37:19.709436 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return next(self.gen)
Jul 25 14:37:19.709531 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/_drivers/rbd.py", line 268, in get_connection
Jul 25 14:37:19.709625 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     client = rados.Rados(conffile=conffile, rados_id=rados_id)
Jul 25 14:37:19.709719 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi AttributeError: 'NoneType' object has no attribute 'Rados'
Jul 25 14:37:19.709869 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi

** Affects: glance-store
     Importance: Undecided
         Status: Confirmed


** Tags: ceph rbd

** Also affects: glance-store
   Importance: Undecided
       Status: New

** Tags added: ceph rbd

** No longer affects: glance

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1706405

Title:
  ceph jobs failing to upload images in pike due to "AttributeError:
  'NoneType' object has no attribute 'Rados'" on py35

Status in glance_store:
  Confirmed

Bug description:
  http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm-
  py35-full-devstack-plugin-ceph-ubuntu-xenial-
  nv/6066e44/logs/screen-g-api.txt.gz?level=TRACE#_Jul_25_14_37_19_667434

  Jul 25 14:37:19.667434 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.api.v2.image_data [None req-19231148-54d7-4337-be6b-a9a73c34b617 admin admin] Failed to upload image data due to internal error
  Jul 25 14:37:19.704732 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi [None req-19231148-54d7-4337-be6b-a9a73c34b617 admin admin] Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'
  Jul 25 14:37:19.704898 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi Traceback (most recent call last):
  Jul 25 14:37:19.705014 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/wsgi.py", line 1155, in __call__
  Jul 25 14:37:19.705130 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     request, **action_args)
  Jul 25 14:37:19.705242 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/wsgi.py", line 1194, in dispatch
  Jul 25 14:37:19.705345 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return method(*args, **kwargs)
  Jul 25 14:37:19.705445 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/utils.py", line 363, in wrapped
  Jul 25 14:37:19.705546 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
  Jul 25 14:37:19.705646 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 269, in upload
  Jul 25 14:37:19.705751 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self._restore(image_repo, image)
  Jul 25 14:37:19.705850 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 25 14:37:19.705960 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.force_reraise()
  Jul 25 14:37:19.706062 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 25 14:37:19.706174 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Jul 25 14:37:19.706274 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/six.py", line 686, in reraise
  Jul 25 14:37:19.706373 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     raise value
  Jul 25 14:37:19.706472 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 134, in upload
  Jul 25 14:37:19.706572 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     image.set_data(data, size)
  Jul 25 14:37:19.706670 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/domain/proxy.py", line 195, in set_data
  Jul 25 14:37:19.706787 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.base.set_data(data, size)
  Jul 25 14:37:19.706914 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/notifier.py", line 480, in set_data
  Jul 25 14:37:19.707029 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
  Jul 25 14:37:19.707133 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jul 25 14:37:19.707233 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.force_reraise()
  Jul 25 14:37:19.707345 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jul 25 14:37:19.707459 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Jul 25 14:37:19.707554 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/six.py", line 686, in reraise
  Jul 25 14:37:19.707649 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     raise value
  Jul 25 14:37:19.707752 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/notifier.py", line 427, in set_data
  Jul 25 14:37:19.707848 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.repo.set_data(data, size)
  Jul 25 14:37:19.707942 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/policy.py", line 192, in set_data
  Jul 25 14:37:19.708036 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
  Jul 25 14:37:19.708138 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/quota/__init__.py", line 304, in set_data
  Jul 25 14:37:19.708238 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     self.image.set_data(data, size=size)
  Jul 25 14:37:19.708333 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/location.py", line 439, in set_data
  Jul 25 14:37:19.708432 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     verifier=verifier)
  Jul 25 14:37:19.708536 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/backend.py", line 453, in add_to_backend
  Jul 25 14:37:19.708635 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     verifier)
  Jul 25 14:37:19.708734 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/backend.py", line 426, in store_add_to_backend
  Jul 25 14:37:19.708833 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     verifier=verifier)
  Jul 25 14:37:19.708938 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/capabilities.py", line 225, in op_checker
  Jul 25 14:37:19.709037 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
  Jul 25 14:37:19.709142 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/_drivers/rbd.py", line 454, in add
  Jul 25 14:37:19.709237 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     rados_id=self.user) as conn:
  Jul 25 14:37:19.709337 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
  Jul 25 14:37:19.709436 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     return next(self.gen)
  Jul 25 14:37:19.709531 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.5/dist-packages/glance_store/_drivers/rbd.py", line 268, in get_connection
  Jul 25 14:37:19.709625 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi     client = rados.Rados(conffile=conffile, rados_id=rados_id)
  Jul 25 14:37:19.709719 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi AttributeError: 'NoneType' object has no attribute 'Rados'
  Jul 25 14:37:19.709869 ubuntu-xenial-ovh-bhs1-10067578 devstack@g-api.service[21781]: ERROR glance.common.wsgi

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance-store/+bug/1706405/+subscriptions
  Thread Previous • Date Previous • • Thread Next