yahoo-eng-team team mailing list archive

[OpenStack Infra <1398128@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/139770
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=0354640587cde740aa0299c722f019ae1c01e05d
Submitter: Jenkins
Branch:    master

commit 0354640587cde740aa0299c722f019ae1c01e05d
Author: Adam Gandelman <adamg@xxxxxxxxxx>
Date:   Fri Dec 5 16:49:12 2014 -0800

    Move ironic ssh key creation early in preparation
    
    SSH creds should be in place before nodes are enrolled.  If not,
    ironic cannot sync power state causing nova to skip nodes in
    its resource tracker.
    
    Change-Id: I6b98ae57ce33783f69e2cf9ba357807d384b3012
    Closes-bug: #1398128


** Changed in: devstack
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1398128

Title:
  ironic tempest tests periodically failing: No valid host was found

Status in devstack - openstack dev environments:
  Fix Released
Status in OpenStack Bare Metal Provisioning Service (Ironic):
  Invalid
Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  This was noticed on the stable/juno ironic sideways grenade jobs, but
  is also confirmed to be happening on the check-tempest-dsvm-ironic-
  parallel-nv job, which runs a similarly configured tempest run against
  Ironic:

  http://logs.openstack.org/84/137684/1/check/check-grenade-dsvm-ironic-
  sideways/6d118bc/

  A number of the early compute tests will fail to spawn an instance,
  getting a scheduling error on the client side:

  BuildErrorException: Server %(server_id)s failed to build and is in ERROR status
  Details: Server eb81ee40-ceba-484d-b665-92ec3bf4fedd failed to build and is in ERROR status
  Details: {u'message': u'No valid host was found. ', u'created': u'2014-11-27T17:44:05Z', u'code': 500}

  Looking through the nova logs, the request never even makes to the
  nova-scheduler.  The last error is reported in conductor:

  2014-11-27 17:44:01.005 WARNING nova.scheduler.driver [req-a3c046e5
  -66db-4bca-a6f8-2263763e49a6 SecurityGroupsTestJSON-2119055496
  SecurityGroupsTestJSON-1381566740] [instance: 9008811a-f400-42ae-
  98d5-caf828fa34dc] NoValidHost exception with message: 'No valid host
  was found.'

  Looking at the time stamps of the requests, the first instance is
  requested at 17:44:00

  2014-11-27 17:44:00.944 24730 DEBUG tempest.common.rest_client [req-
  a3c046e5-66db-4bca-a6f8-2263763e49a6 None] Request
  (SecurityGroupsTestJSON:test_server_security_groups): 202 POST
  http://127.0.0.1:8774/v2/adf4838f0d15462da4601a5d853eafbf/servers
  0.515s

  However, on the nova-compute side, the resource tracker has not been
  updated to include the enlisted Ironic nodes until much later.  This
  first time the tracker contains any of the ironic resources is at
  17:44:06:

  2014-11-27 17:44:06.224 21645 AUDIT nova.compute.resource_tracker [-]
  Total physical ram (MB): 512, total allocated virtual ram (MB): 0

  So there's a race between the resource tracker's initial inclusion of
  available resources and Tempest running the first set of tests that
  require an instance.   This can be worked around in a couple of ways:

  * Adjust the periodic task interval on nova-compute to update much more frequently, tho this will just narrow the window.  
  * Have tempest run an admin 'nova hypervisor-stats' call on the client side and wait for resources before running any instances (in the case of baremetal only)
  * Adjust devstack's nova cpu deployment to spin until hypervisor-stats reflect the ironic node parameters

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1398128/+subscriptions