← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #45245

[Bug 1534066] Re: Resize failed when instance get large

  Thread Previous • Date Previous • • Thread Next 
@leehom:

Bug skimming result
===================
Juno has reached end of life (EOL) [1] and won't get any updates
anymore. That's why I close this bug as "Invalid". 

If you observe the issue(s) in one or more of the supported releases 
(kilo, liberty, mitaka=master), then reopen the bug report and provide
more information with template [2] to allow an analysis of the issue.

If you have questions, I'm available in the IRC channel #openstack-nova
under the name "markus_z".

References
==========
[1] http://docs.openstack.org/releases/
[2] https://wiki.openstack.org/wiki/Nova/BugsTeam/BugReportTemplate

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1534066

Title:
  Resize failed when instance get large

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  I'm using OpenStack Juno
  $ rpm -qa |grep nova
  python-nova-2014.2.2-1.el7.noarch
  openstack-nova-scheduler-2014.2.2-1.el7.noarch
  python-novaclient-2.20.0-1.el7.centos.noarch
  openstack-nova-common-2014.2.2-1.el7.noarch
  openstack-nova-api-2014.2.2-1.el7.noarch
  openstack-nova-cert-2014.2.2-1.el7.noarch
  openstack-nova-console-2014.2.2-1.el7.noarch
  openstack-nova-conductor-2014.2.2-1.el7.noarch
  openstack-nova-novncproxy-2014.2.2-1.el7.noarch

  There has two problems
  Problem1:
  Token not expired, but somehow deleted which cause /usr/lib/python2.7/site-packages/neutronclient/client.py:do_request() failed with exceptions.Unauthorized

      def do_request(self, url, method, **kwargs):
          self.authenticate_and_fetch_endpoint_url()
          # Perform the request once. If we get a 401 back then it
          # might be because the auth token expired, so try to
          # re-authenticate and try again. If it still fails, bail.
          try:
              kwargs.setdefault('headers', {})
              if self.auth_token is None:
                  self.auth_token = ""
              kwargs['headers']['X-Auth-Token'] = self.auth_token
              resp, body = self._cs_request(self.endpoint_url + url, method,
                                            **kwargs)
              return resp, body
          except exceptions.Unauthorized:
              self.authenticate()
              kwargs.setdefault('headers', {})
              kwargs['headers']['X-Auth-Token'] = self.auth_token
              resp, body = self._cs_request(
                  self.endpoint_url + url, method, **kwargs)
              return resp, body

  Problem2:
  In except branch, self.authenticate() failed with exceptions.Unauthorized
  which is a duplicate of https://bugs.launchpad.net/nova/+bug/1418529
  But there has no fix for juno.

  Here is log found from nova-compute
  ########### Log for problem1
  2016-01-07 16:08:42.396 1334 DEBUG neutronclient.client [req-7d95c82d-b26a-4584-851b-95de5c99f17c ]
  REQ: curl -i https://hf1-neutron.qa.webex.com:443/v2.0/extensions.json -X GET -H "X-Auth-Token: 119b7628442f4e19b4bd1041d05c2afa" -H "User-Agent: python-neutronclient"
   http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:140
  2016-01-07 16:08:42.473 1334 DEBUG neutronclient.client [req-7d95c82d-b26a-4584-851b-95de5c99f17c ] RESP:401 {'content-length': '23', 'www-authenticate': "Keystone uri='https://hf1-keystone-srv.qa.webex.com:443/v2.0'", 'connection': 'keep-alive', 'date': 'Thu, 07 Jan 2016 16:08:42 GMT', 'content-type': 'text/plain', 'x-openstack-request-id': 'req-36035958-9e5d-44db-a12a-0ed0764a7d0e'} Authentication required
   http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:149

  ########### Log for problem2
  2016-01-07 16:08:42.474 1334 ERROR nova.compute.manager [req-7d95c82d-b26a-4584-851b-95de5c99f17c None] [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127] Setting instance vm_state to ERROR
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127] Traceback (most recent call last):
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3904, in finish_resize
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     disk_info, image)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3843, in _finish_resize
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     migration_p)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1241, in migrate_instance_finish
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     if not self._has_port_binding_extension(context, refresh_cache=True):
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 471, in _has_port_binding_extension
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     self._refresh_neutron_extensions_cache(context)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 463, in _refresh_neutron_extensions_cache
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     extensions_list = neutron.list_extensions()['extensions']
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     ret = self.function(instance, *args, **kwargs)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in list_extensions
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     return self.get(self.extensions_path, params=_params)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1320, in get
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     headers=headers, params=params)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     headers=headers, params=params)
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     content_type=self.content_type())
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 183, in do_request
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     self.authenticate()
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 247, in authenticate
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     self._authenticate_keystone()
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 220, in _authenticate_keystone
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]     raise exceptions.NoAuthURLProvided()
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127] NoAuthURLProvided: auth_url was not provided to the Neutron client
  2016-01-07 16:08:42.474 1334 TRACE nova.compute.manager [instance: fc8d35d5-874d-4c09-9f23-9bf7587b2127]

  Here is the log found from keystone
  ########## Log for problem1
  2016-01-07 05:16:33.697 21392 DEBUG keystone.common.wsgi [-] arg_dict: {'token_id': u'119b7628442f4e19b4bd1041d05c2afa'} __call__ /usr/lib/python2.7/site-packages/keystone/common/wsgi.py:191
  2016-01-07 05:16:33.698 21392 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:validate_token(token_id=119b7628442f4e19b4bd1041d05c2afa) _build_policy_check_credentials /usr/lib/python2.7/site-packages/keystone/common/controller.py:55
  2016-01-07 05:16:33.721 21392 INFO eventlet.wsgi.server [-] 10.224.241.140,10.224.236.93 - - [07/Jan/2016 05:16:33] "GET /v2.0/tokens/119b7628442f4e19b4bd1041d05c2afa HTTP/1.1" 200 2157 0.040029
  2016-01-07 05:22:00.632 21393 DEBUG keystone.common.wsgi [-] arg_dict: {'token_id': u'119b7628442f4e19b4bd1041d05c2afa'} __call__ /usr/lib/python2.7/site-packages/keystone/common/wsgi.py:191
  2016-01-07 05:22:00.633 21393 DEBUG keystone.common.controller [-] RBAC: Authorizing identity:validate_token(token_id=119b7628442f4e19b4bd1041d05c2afa) _build_policy_check_credentials /usr/lib/python2.7/site-packages/keystone/common/controller.py:55
  2016-01-07 05:22:00.648 21393 INFO eventlet.wsgi.server [-] 10.224.241.139,10.224.236.93 - - [07/Jan/2016 05:22:00] "GET /v2.0/tokens/119b7628442f4e19b4bd1041d05c2afa HTTP/1.1" 200 2157 0.034250
  2016-01-07 06:19:05.034 21410 DEBUG keystone.common.wsgi [-] arg_dict: {'token_id': u'119b7628442f4e19b4bd1041d05c2afa'} __call__ /usr/lib/python2.7/site-packages/keystone/common/wsgi.py:191

  ########## I think this part is the key, but I do not understand where and why need to delete this token before it expire
  2016-01-07 06:19:05.069 21410 INFO eventlet.wsgi.server [-] 10.224.241.139,10.224.236.93 - - [07/Jan/2016 06:19:05] "DELETE /v2.0/tokens/119b7628442f4e19b4bd1041d05c2afa HTTP/1.1" 204 127 0.049150

  And here is found token from database
  ########## Log for problem1
  | id                                                                         | expires                            | extra  | valid | trust_id | user_id                          |
  | 119b7628442f4e19b4bd1041d05c2afa | 2016-01-08 04:47:23 | {"bind": null, "token_data": {"access": {"token": {"issued_at": "2016-01-07T04:48:10.638860", "expires": "2016-01-08T04:47:23Z", ...  |     0 | NULL     | e35c77abe1384bf2a1f50363d870b44f |

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1534066/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next