yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1733642] [NEW] AttributeError: 'NoneType' object has no attribute 'get_token'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 21 Nov 2017 16:44:22 -0000
Reply-to: Bug 1733642 <1733642@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Someone reported this in nova IRC:

http://paste.openstack.org/show/626966/

They were doing a snapshot of an instance with an NFS-backed volume and
using the nova service user configuration in the [service_user] section
of nova.conf, however, it looks like their configuration was incomplete
which resulted in this:

Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver [req-3278cb0b-5c22-4874-905c-7f19ef735188 req-9e15496a-ae04-49ad-adbd-dd65b1bc9aaf admin admin] Failed to send updated snapshot status to volume service.: AttributeError: 'NoneType' object has no attribute 'get_token'
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1952, in _volume_snapshot_update_status
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     status)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/volume/cinder.py", line 241, in wrapper
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     res = method(self, ctx, *args, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/volume/cinder.py", line 291, in wrapper
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     res = method(self, ctx, snapshot_id, *args, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/volume/cinder.py", line 540, in update_snapshot_status
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     'progress': '90%'}
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/volume_snapshots.py", line 166, in update_snapshot_status
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     base.getid(snapshot), update_dict)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/volume_snapshots.py", line 161, in _action
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     resp, body = self.api.client.post(url, body=body)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 202, in post
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return self._cs_request(url, 'POST', **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 190, in _cs_request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return self.request(url, method, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 173, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return self.session.request(url, method, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     auth_headers = self.get_auth_headers(auth)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return auth.get_headers(self, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     token = self.service_auth.get_token(session, **kwargs)
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver AttributeError: 'NoneType' object has no attribute 'get_token'
Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver

Looking at the KSA code, this can happen if auth_type isn't configured:

https://github.com/openstack/keystoneauth/blob/3.2.0/keystoneauth1/loading/conf.py#L124

But nova doesn't handle this and it results in a NoneType error later.

** Affects: nova
     Importance: Medium
     Assignee: Matt Riedemann (mriedem)
         Status: Triaged

** Affects: nova/ocata
     Importance: Medium
         Status: New

** Affects: nova/pike
     Importance: Medium
         Status: Triaged

** Changed in: nova
       Status: New => Triaged

** Changed in: nova
   Importance: Undecided => Medium

** Also affects: nova/pike
   Importance: Undecided
       Status: New

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1733642

Title:
  AttributeError: 'NoneType' object has no attribute 'get_token'

Status in OpenStack Compute (nova):
  Triaged
Status in OpenStack Compute (nova) ocata series:
  New
Status in OpenStack Compute (nova) pike series:
  Triaged

Bug description:
  Someone reported this in nova IRC:

  http://paste.openstack.org/show/626966/

  They were doing a snapshot of an instance with an NFS-backed volume
  and using the nova service user configuration in the [service_user]
  section of nova.conf, however, it looks like their configuration was
  incomplete which resulted in this:

  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver [req-3278cb0b-5c22-4874-905c-7f19ef735188 req-9e15496a-ae04-49ad-adbd-dd65b1bc9aaf admin admin] Failed to send updated snapshot status to volume service.: AttributeError: 'NoneType' object has no attribute 'get_token'
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1952, in _volume_snapshot_update_status
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     status)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/volume/cinder.py", line 241, in wrapper
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     res = method(self, ctx, *args, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/volume/cinder.py", line 291, in wrapper
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     res = method(self, ctx, snapshot_id, *args, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/volume/cinder.py", line 540, in update_snapshot_status
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     'progress': '90%'}
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/volume_snapshots.py", line 166, in update_snapshot_status
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     base.getid(snapshot), update_dict)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/volume_snapshots.py", line 161, in _action
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     resp, body = self.api.client.post(url, body=body)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 202, in post
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return self._cs_request(url, 'POST', **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 190, in _cs_request
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return self.request(url, method, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 173, in request
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return self.session.request(url, method, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     auth_headers = self.get_auth_headers(auth)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     return auth.get_headers(self, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver     token = self.service_auth.get_token(session, **kwargs)
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver AttributeError: 'NoneType' object has no attribute 'get_token'
  Nov 21 19:06:50 openstack-VirtualBox nova-compute[10751]: ERROR nova.virt.libvirt.driver

  Looking at the KSA code, this can happen if auth_type isn't
  configured:

  https://github.com/openstack/keystoneauth/blob/3.2.0/keystoneauth1/loading/conf.py#L124

  But nova doesn't handle this and it results in a NoneType error later.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1733642/+subscriptions
Follow ups

[Bug 1733642] Re: AttributeError: 'NoneType' object has no attribute 'get_token'
From: OpenStack Infra, 2017-12-10