yahoo-eng-team team mailing list archive

[Joshua Houle <joshua.houle@xxxxxxxxxxx>]

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1606995

Title:
  Nova fails to provision machine but can pull existing machines

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  After switching from Keystone V2.0 to Keystone V3 we can no longer
  provision machines, we can still see existing machines in Horizon and
  log in Horizon.

  Nova config for Keystone:

  [keystone_authtoken]

  #
  # From keystonemiddleware.auth_token
  #

  # Complete public Identity API endpoint. (string value)
  #auth_uri = <None>
  auth_uri = http://192.168.0.2:5000/

  # API version of the admin Identity API endpoint. (string value)
  #auth_version = <None>

  # Do not handle authorization requests within the middleware, but delegate the
  # authorization decision to downstream WSGI components. (boolean value)
  #delay_auth_decision = false

  # Request timeout value for communicating with Identity API server. (integer
  # value)
  #http_connect_timeout = <None>

  # How many times are we trying to reconnect when communicating with Identity
  # API Server. (integer value)
  #http_request_max_retries = 3

  # Env key for the swift cache. (string value)
  #cache = <None>

  # Required if identity server requires client certificate (string value)
  #certfile = <None>

  # Required if identity server requires client certificate (string value)
  #keyfile = <None>

  # A PEM encoded Certificate Authority to use when verifying HTTPs connections.
  # Defaults to system CAs. (string value)
  #cafile = <None>

  # Verify HTTPS connections. (boolean value)
  #insecure = false

  # The region in which the identity server can be found. (string value)
  #region_name = <None>

  # Directory used to cache files related to PKI tokens. (string value)
  #signing_dir = <None>
  signing_dir = /tmp/keystone-signing-nova

  # Optionally specify a list of memcached server(s) to use for caching. If left
  # undefined, tokens will instead be cached in-process. (list value)
  # Deprecated group/name - [DEFAULT]/memcache_servers
  #memcached_servers = <None>

  # In order to prevent excessive effort spent validating tokens, the middleware
  # caches previously-seen tokens for a configurable duration (in seconds). Set
  # to -1 to disable caching completely. (integer value)
  #token_cache_time = 300

  # Determines the frequency at which the list of revoked tokens is retrieved
  # from the Identity service (in seconds). A high number of revocation events
  # combined with a low cache duration may significantly reduce performance.
  # (integer value)
  #revocation_cache_time = 10

  # (Optional) If defined, indicate whether token data should be authenticated or
  # authenticated and encrypted. Acceptable values are MAC or ENCRYPT.  If MAC,
  # token data is authenticated (with HMAC) in the cache. If ENCRYPT, token data
  # is encrypted and authenticated in the cache. If the value is not one of these
  # options or empty, auth_token will raise an exception on initialization.
  # (string value)
  #memcache_security_strategy = <None>

  # (Optional, mandatory if memcache_security_strategy is defined) This string is
  # used for key derivation. (string value)
  #memcache_secret_key = <None>

  # (Optional) Number of seconds memcached server is considered dead before it is
  # tried again. (integer value)
  #memcache_pool_dead_retry = 300

  # (Optional) Maximum total number of open connections to every memcached
  # server. (integer value)
  #memcache_pool_maxsize = 10

  # (Optional) Socket timeout in seconds for communicating with a memcached
  # server. (integer value)
  #memcache_pool_socket_timeout = 3

  # (Optional) Number of seconds a connection to memcached is held unused in the
  # pool before it is closed. (integer value)
  #memcache_pool_unused_timeout = 60

  # (Optional) Number of seconds that an operation will wait to get a memcached
  # client connection from the pool. (integer value)
  #memcache_pool_conn_get_timeout = 10

  # (Optional) Use the advanced (eventlet safe) memcached client pool. The
  # advanced pool will only work under python 2.x. (boolean value)
  #memcache_use_advanced_pool = false

  # (Optional) Indicate whether to set the X-Service-Catalog header. If False,
  # middleware will not ask for service catalog on token validation and will not
  # set the X-Service-Catalog header. (boolean value)
  #include_service_catalog = true

  # Used to control the use and type of token binding. Can be set to: "disabled"
  # to not check token binding. "permissive" (default) to validate binding
  # information if the bind type is of a form known to the server and ignore it
  # if not. "strict" like "permissive" but if the bind type is unknown the token
  # will be rejected. "required" any form of token binding is needed to be
  # allowed. Finally the name of a binding method that must be present in tokens.
  # (string value)
  #enforce_token_bind = permissive

  # If true, the revocation list will be checked for cached tokens. This requires
  # that PKI tokens are configured on the identity server. (boolean value)
  #check_revocations_for_cached = false

  # Hash algorithms to use for hashing PKI tokens. This may be a single algorithm
  # or multiple. The algorithms are those supported by Python standard
  # hashlib.new(). The hashes will be tried in the order given, so put the
  # preferred one first for performance. The result of the first hash will be
  # stored in the cache. This will typically be set to multiple values only while
  # migrating from a less secure algorithm to a more secure one. Once all the old
  # tokens are expired this option should be set to a single value for better
  # performance. (list value)
  #hash_algorithms = md5

  # Prefix to prepend at the beginning of the path. Deprecated, use identity_uri.
  # (string value)
  #auth_admin_prefix =

  # Host providing the admin Identity API endpoint. Deprecated, use identity_uri.
  # (string value)

  # Port of the admin Identity API endpoint. Deprecated, use identity_uri.
  # (integer value)
  #auth_port = 35357

  # Protocol of the admin Identity API endpoint (http or https). Deprecated, use
  # identity_uri. (string value)
  #auth_protocol = https

  # Complete admin Identity API endpoint. This should specify the unversioned
  # root endpoint e.g. https://localhost:35357/ (string value)
  #identity_uri = <None>
  identity_uri = http://192.168.0.2:35357/

  # This option is deprecated and may be removed in a future release. Single
  # shared secret with the Keystone configuration used for bootstrapping a
  # Keystone installation, or otherwise bypassing the normal authentication
  # process. This option should not be used, use `admin_user` and
  # `admin_password` instead. (string value)
  #admin_token = <None>

  # Service username. (string value)
  admin_user = nova

  # Service user password. (string value)
  admin_password = **************************

  # Service tenant name. (string value)
  admin_tenant_name = services
  signing_dirname=/tmp/keystone-signing-nova

  
  Nova API log:

  2016-07-27 11:48:00.056 27383 WARNING oslo_config.cfg [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-07-27 11:48:00.056 27383 WARNING oslo_config.cfg [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-07-27 11:48:00.057 27383 WARNING oslo_config.cfg [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] Option "admin_username" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-07-27 11:48:00.057 27383 WARNING oslo_config.cfg [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] Option "admin_password" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-07-27 11:48:00.057 27383 WARNING oslo_config.cfg [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] Unexpected exception in API method
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1581, in create
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1181, in _create_instance
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1094, in create_pci_requests_for_sriov_ports
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 237, in get_client
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-eb91ed23-8c4f-46ef-bc9e-a163b9e78b0e)
  2016-07-27 11:48:00.063 27383 ERROR nova.api.openstack.extensions
  2016-07-27 11:48:00.183 27383 INFO nova.api.openstack.wsgi [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'keystoneclient.exceptions.NotFound'>
  2016-07-27 11:48:00.184 27383 INFO nova.osapi_compute.wsgi.server [req-47f073db-becc-4613-b9de-b4315e332e7e da86a3e1c898410daf7dfd4eb5b495d0 870e97f74a1442a589fbb70a5ead3da6 - - -] 192.168.0.2 "POST /v2/870e97f74a1442a589fbb70a5ead3da6/servers HTTP/1.1" status: 500 len: 434 time: 1.4834981

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1606995/+subscriptions