yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1553149] Re: Instance in ERROR state due to ConnectFailure with keystone

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dolph Mathews <1553149@xxxxxxxxxxxxxxxxxx>
Date: Tue, 08 Mar 2016 14:45:16 -0000
Reply-to: Bug 1553149 <1553149@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Apache will refuse connections that it cannot assign to threads once
MaxClients is exhausted, and if you're only running 10 threads, then I'm
also guessing that your MaxClients is set to be less than the number of
concurrent connections you're throwing at it.

I'm closing this because this is just an Apache tuning issue.

** Changed in: keystone
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1553149

Title:
  Instance in ERROR state due to ConnectFailure with keystone

Status in OpenStack Identity (keystone):
  Invalid

Bug description:
  When tried to run below rally scenario with concurrency 50, seeing issue with keystone. Can someone take a look?
  NOTE: Things will work fine with concurrency 10.

  1. Create tenant, create network. 
  2. Create T1 router and set external network as gateway
  3. Add network created in step 1 to T1 router
  4. Launch instance(on kvm) in the private network and assign FIP. Ping FIP

  
  Setup:

  Single controller(32vCPU, 48GB RAM)
  3 Network Nodes
  100 ESX computes and 100 KVM computes

  Rally reports and logs attached to  bug.

  Logs:

  2016-03-01 01:26:34.699 DEBUG oslo_concurrency.lockutils [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] Releasing semaphore "refresh_cache-8c324106-c6dd-4b90-876d-e3cc33adfebf" from (pid=26585) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
  2016-03-01 01:26:34.704 ERROR nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Instance failed to spawn
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Traceback (most recent call last):
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 2190, in _build_resources
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     yield resources
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 2036, in _build_and_run_instance
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     block_device_info=block_device_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2758, in spawn
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     admin_pass=admin_password)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3251, in _create_image
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     content=files, extra_md=extra_md, network_info=network_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/api/metadata/base.py", line 160, in __init__
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.network_metadata = netutils.get_network_metadata(network_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/virt/netutils.py", line 185, in get_network_metadata
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     if not network_info:
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 526, in __len__
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._sync_wrapper(fn, *args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 513, in _sync_wrapper
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.wait()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/model.py", line 545, in wait
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self[:] = self._gt.wait()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._exit_event.wait()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return hubs.get_hub().switch()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.greenlet.switch()
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     result = function(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/utils.py", line 1160, in context_wrapper
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 1583, in _allocate_network_async
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     six.reraise(*exc_info)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/compute/manager.py", line 1566, in _allocate_network_async
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     bind_host_id=bind_host_id)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 668, in allocate_for_instance
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     update_cells=True)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/base_api.py", line 246, in get_instance_nw_info
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     result = self._get_instance_nw_info(context, instance, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 911, in _get_instance_nw_info
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     preexisting_port_ids)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1713, in _build_network_info_model
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     current_neutron_port)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1565, in _nw_info_get_ips
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     client, fixed_ip['ip_address'], port['id'])
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1496, in _get_floating_ips_by_fixed_and_port
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     port_id=port)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1467, in _safe_get_floating_ips
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return client.list_floatingips(**kwargs)['floatingips']
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 100, in with_params
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     ret = self.function(instance, *args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 699, in list_floatingips
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     **_params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, in list
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     for r in self._pagination(collection, path, **params):
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in _pagination
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     res = self.get(path, params=params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 266, in get
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     headers=headers, params=params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 243, in retry_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     headers=headers, params=params)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 195, in do_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp, replybody = self.httpclient.do_request(action, method, body=body)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in do_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.request(url, method, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp = super(SessionClient, self).request(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.session.request(url, method, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     auth_headers = self.get_auth_headers(auth)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return auth.get_headers(self, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     token = self.get_token(session)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.get_access(session).auth_token
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     self.auth_ref = self.get_auth_ref(session)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self._plugin.get_auth_ref(session, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     authenticated=False, log=False, **rkwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return self.request(url, 'POST', **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     return func(*args, **kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     resp = send(**kwargs)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 496, in _send_request
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf]     raise exceptions.ConnectFailure(msg)
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ConnectFailure: Unable to establish connection to http://172.17.4.231:35357/v3/auth/tokens
  2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] 
  2016-03-01 01:26:34.707 INFO nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Terminating instance

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1553149/+subscriptions
References

[Bug 1553149] [NEW] Instance in ERROR state due to ConnectFailure with keystone
From: Prashant Shetty, 2016-03-04