yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

On teardown this method is called:

https://github.com/openstack/tempest/blob/master/tempest/api/compute/base.py#L152

That waits for the shared server to be active, which it should be after
the unshelve runs:

    def server_check_teardown(cls):
        """Checks is the shared server clean enough for subsequent test.
           Method will delete the server when it's dirty.
           The setUp method is responsible for creating a new server.
           Exceptions raised in tearDown class are fails the test case,
           This method supposed to use only by tearDown methods, when
           the shared server_id is stored in the server_id of the class.
        """
        if getattr(cls, 'server_id', None) is not None:
            try:
                waiters.wait_for_server_status(cls.servers_client,
                                               cls.server_id, 'ACTIVE')
            except Exception as exc:
                LOG.exception(exc)
                cls.servers_client.delete_server(cls.server_id)
                waiters.wait_for_server_termination(cls.servers_client,
                                                    cls.server_id)
                cls.server_id = None
                raise

But then right before the suspend fails, the server is ACTIVE with no
task_state but it doesn't have a host assigned:

2016-08-06 23:42:20.558 18269 DEBUG tempest.lib.common.rest_client [req-eeea93df-9dc5-4b51-8cea-15a5f8ae9a3f ] Request - Headers: {'Content-Type': 'application/json', 'X-Auth-Token': '<omitted>', 'Accept': 'application/json'}
        Body: None
    Response - Headers: {'x-compute-request-id': 'req-eeea93df-9dc5-4b51-8cea-15a5f8ae9a3f', 'date': 'Sat, 06 Aug 2016 23:42:20 GMT', 'vary': 'X-OpenStack-Nova-API-Version', 'content-location': 'http://127.0.0.1:8774/v2.1/servers/480c8b6f-ab2d-4a49-8344-a3a679b01472', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'status': '200', 'x-openstack-nova-api-version': '2.1', 'connection': 'close', 'content-length': '1372'}
        Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e4:3e:93", "version": 4, "addr": "10.1.9.1", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://127.0.0.1:8774/v2.1/servers/480c8b6f-ab2d-4a49-8344-a3a679b01472";, "rel": "self"}, {"href": "http://127.0.0.1:8774/servers/480c8b6f-ab2d-4a49-8344-a3a679b01472";, "rel": "bookmark"}], "image": {"id": "58ef0441-c37e-4691-9ce4-9ea30b071aba", "links": [{"href": "http://127.0.0.1:8774/images/58ef0441-c37e-4691-9ce4-9ea30b071aba";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-08-06T23:42:19.000000", "flavor": {"id": "42", "links": [{"href": "http://127.0.0.1:8774/flavors/42";, "rel": "bookmark"}]}, "id": "480c8b6f-ab2d-4a49-8344-a3a679b01472", "security_groups": [{"name": "default"}], "user_id": "707a3c38bf0e4a36a286ccfeb4a2b13a", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "", "metadata": {}, "status": "ACTIVE", "updated": "2016-08-06T23:42:19Z", "hostId": "", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-1034238505", "created": "2016-08-06T23:41:23Z", "tenant_id": "79ff191035e240be962ea4992a46a11e", "os-extended-volumes:volumes_attached": [], "config_drive": "True"}} _log_request_full tempest/lib/common/rest_client.py:433

We shouldn't have an ACTIVE server with no host...so this is probably a
race bug in nova with the unshelve code.

** Changed in: nova
       Status: Invalid => Confirmed

** Changed in: tempest
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1611008

Title:
  ServersNegativeTestJSON.test_suspend_server_invalid_state fails with
  "NovaException: Unable to find host for Instance"

Status in OpenStack Compute (nova):
  Confirmed
Status in tempest:
  Invalid

Bug description:
  Seen here:

  http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-
  ubuntu-
  xenial/6a11005/logs/screen-n-api.txt.gz?level=TRACE#_2016-08-06_23_42_20_711

  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions [req-86f8ef99-a9e4-49f0-8258-a0f40dea4a51 tempest-ServersNegativeTestJSON-16764714 tempest-ServersNegativeTestJSON-16764714] Unexpected exception in API method
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/suspend_server.py", line 41, in _suspend
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     self.compute_api.suspend(context, server)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 164, in inner
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 171, in _wrapped
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     return fn(self, context, instance, *args, **kwargs)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 145, in inner
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 2782, in suspend
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     self.compute_rpcapi.suspend_instance(context, instance)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 953, in suspend_instance
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     server=_compute_host(None, instance), version=version)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 53, in _compute_host
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions     'Instance %s') % instance.uuid)
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions NovaException: Unable to find host for Instance 480c8b6f-ab2d-4a49-8344-a3a679b01472
  2016-08-06 23:42:20.711 10592 ERROR nova.api.openstack.extensions 

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22NovaException%3A%20Unable%20to%20find%20host%20for%20Instance%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

  There are 4 hits in 7 days, check and gate, all failures, all on the
  master branch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1611008/+subscriptions