yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1553149] [NEW] Instance in ERROR state due to ConnectFailure with keystone

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Prashant Shetty <prashantshetty@xxxxxxxxxx>
Date: Fri, 04 Mar 2016 11:22:13 -0000
Reply-to: Bug 1553149 <1553149@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When tried to run below rally scenario with concurrency 50, seeing issue with keystone. Can someone take a look?
NOTE: Things will work fine with concurrency 10.

1. Create tenant, create network. 
2. Create T1 router and set external network as gateway
3. Add network created in step 1 to T1 router
4. Launch instance(on kvm) in the private network and assign FIP. Ping FIP


Setup:

Single controller(32vCPU, 48GB RAM)
3 Network Nodes
100 ESX computes and 100 KVM computes

Rally reports and logs attached to  bug.

Logs:

2016-03-01 01:26:34.699 DEBUG oslo_concurrency.lockutils [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] Releasing semaphore "refresh_cache-8c324106-c6dd-4b90-876d-e3cc33adfebf" from (pid=26585) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2016-03-01 01:26:34.704 ERROR nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Instance failed to spawn
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Traceback (most recent call last):
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 2190, in _build_resources
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     yield resources
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 2036, in _build_and_run_instance
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     block_device_info=block_device_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2758, in spawn
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     admin_pass=admin_password)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3251, in _create_image
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     content=files, extra_md=extra_md, network_info=network_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/api/metadata/base.py", line 160, in __init__
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.network_metadata = netutils.get_network_metadata(network_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/netutils.py", line 185, in get_network_metadata
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     if not network_info:
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 526, in __len__
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._sync_wrapper(fn, *args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 513, in _sync_wrapper
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.wait()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 545, in wait
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self[:] = self._gt.wait()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._exit_event.wait()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return hubs.get_hub().switch()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.greenlet.switch()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     result = function(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/utils.py", line 1160, in context_wrapper
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 1583, in _allocate_network_async
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     six.reraise(*exc_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 1566, in _allocate_network_async
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     bind_host_id=bind_host_id)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 668, in allocate_for_instance
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     update_cells=True)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/base_api.py", line 246, in get_instance_nw_info
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     result = self._get_instance_nw_info(context, instance, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 911, in _get_instance_nw_info
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     preexisting_port_ids)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1713, in _build_network_info_model
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     current_neutron_port)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1565, in _nw_info_get_ips
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     client, fixed_ip['ip_address'], port['id'])
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1496, in _get_floating_ips_by_fixed_and_port
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     port_id=port)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1467, in _safe_get_floating_ips
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return client.list_floatingips(**kwargs)['floatingips']
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 100, in with_params
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     ret = self.function(instance, *args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 699, in list_floatingips
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     **_params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, in list
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     for r in self._pagination(collection, path, **params):
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in _pagination
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     res = self.get(path, params=params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 266, in get
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     headers=headers, params=params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 243, in retry_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     headers=headers, params=params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 195, in do_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp, replybody = self.httpclient.do_request(action, method, body=body)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in do_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.request(url, method, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp = super(SessionClient, self).request(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.session.request(url, method, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     auth_headers = self.get_auth_headers(auth)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return auth.get_headers(self, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     token = self.get_token(session)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.get_access(session).auth_token
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.auth_ref = self.get_auth_ref(session)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._plugin.get_auth_ref(session, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     authenticated=False, log=False, **rkwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.request(url, 'POST', **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp = send(**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 496, in _send_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     raise exceptions.ConnectFailure(msg)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ConnectFailure: Unable to establish connection to http://172.17.4.231:35357/v3/auth/tokens
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] 
2016-03-01 01:26:34.707 INFO nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Terminating instance

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1553149

Title:
  Instance in ERROR state due to ConnectFailure with keystone

Status in OpenStack Identity (keystone):
  New

Bug description:
  When tried to run below rally scenario with concurrency 50, seeing issue with keystone. Can someone take a look?
  NOTE: Things will work fine with concurrency 10.

  1. Create tenant, create network. 
  2. Create T1 router and set external network as gateway
  3. Add network created in step 1 to T1 router
  4. Launch instance(on kvm) in the private network and assign FIP. Ping FIP

  
  Setup:

  Single controller(32vCPU, 48GB RAM)
  3 Network Nodes
  100 ESX computes and 100 KVM computes

  Rally reports and logs attached to  bug.

  Logs:

  2016-03-01 01:26:34.699 DEBUG oslo_concurrency.lockutils [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] Releasing semaphore "refresh_cache-8c324106-c6dd-4b90-876d-e3cc33adfebf" from (pid=26585) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
  2016-03-01 01:26:34.704 ERROR nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Instance failed to spawn
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Traceback (most recent call last):
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 2190, in _build_resources
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     yield resources
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 2036, in _build_and_run_instance
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     block_device_info=block_device_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2758, in spawn
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     admin_pass=admin_password)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3251, in _create_image
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     content=files, extra_md=extra_md, network_info=network_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/api/metadata/base.py", line 160, in __init__
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.network_metadata = netutils.get_network_metadata(network_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/netutils.py", line 185, in get_network_metadata
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     if not network_info:
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 526, in __len__
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._sync_wrapper(fn, *args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 513, in _sync_wrapper
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.wait()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 545, in wait
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self[:] = self._gt.wait()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._exit_event.wait()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return hubs.get_hub().switch()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.greenlet.switch()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     result = function(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/utils.py", line 1160, in context_wrapper
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 1583, in _allocate_network_async
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     six.reraise(*exc_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 1566, in _allocate_network_async
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     bind_host_id=bind_host_id)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 668, in allocate_for_instance
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     update_cells=True)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/base_api.py", line 246, in get_instance_nw_info
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     result = self._get_instance_nw_info(context, instance, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 911, in _get_instance_nw_info
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     preexisting_port_ids)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1713, in _build_network_info_model
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     current_neutron_port)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1565, in _nw_info_get_ips
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     client, fixed_ip['ip_address'], port['id'])
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1496, in _get_floating_ips_by_fixed_and_port
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     port_id=port)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1467, in _safe_get_floating_ips
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return client.list_floatingips(**kwargs)['floatingips']
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 100, in with_params
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     ret = self.function(instance, *args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 699, in list_floatingips
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     **_params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, in list
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     for r in self._pagination(collection, path, **params):
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in _pagination
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     res = self.get(path, params=params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 266, in get
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     headers=headers, params=params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 243, in retry_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     headers=headers, params=params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 195, in do_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp, replybody = self.httpclient.do_request(action, method, body=body)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in do_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.request(url, method, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp = super(SessionClient, self).request(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.session.request(url, method, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     auth_headers = self.get_auth_headers(auth)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return auth.get_headers(self, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     token = self.get_token(session)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.get_access(session).auth_token
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.auth_ref = self.get_auth_ref(session)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._plugin.get_auth_ref(session, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     authenticated=False, log=False, **rkwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.request(url, 'POST', **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp = send(**kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 496, in _send_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     raise exceptions.ConnectFailure(msg)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ConnectFailure: Unable to establish connection to http://172.17.4.231:35357/v3/auth/tokens
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] 
  2016-03-01 01:26:34.707 INFO nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Terminating instance

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1553149/+subscriptions
Follow ups

[Bug 1553149] Re: Instance in ERROR state due to ConnectFailure with keystone
From: Dolph Mathews, 2016-03-08