yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1674809] [NEW] test_create_server_with_pci_dev_and_numa functional test randomly fails with "OpenStackApiNotFoundException: Item not found"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 21 Mar 2017 19:58:34 -0000
Reply-to: Bug 1674809 <1674809@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

http://logs.openstack.org/03/448203/2/check/gate-nova-tox-functional-
py35-ubuntu-xenial/5f60ae1/console.html#_2017-03-21_18_29_51_237243

2017-03-21 18:29:51.237243 | nova.tests.functional.libvirt.test_pci_sriov_servers.SRIOVServersTest.test_create_server_with_pci_dev_and_numa
2017-03-21 18:29:51.237307 | --------------------------------------------------------------------------------------------------------------
2017-03-21 18:29:51.237322 | 
2017-03-21 18:29:51.237340 | Captured traceback:
2017-03-21 18:29:51.237358 | ~~~~~~~~~~~~~~~~~~~
2017-03-21 18:29:51.237381 |     b'Traceback (most recent call last):'
2017-03-21 18:29:51.237440 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/.tox/functional-py35/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched'
2017-03-21 18:29:51.237466 |     b'    return func(*args, **keywargs)'
2017-03-21 18:29:51.237530 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/libvirt/test_pci_sriov_servers.py", line 272, in test_create_server_with_pci_dev_and_numa'
2017-03-21 18:29:51.237572 |     b'    pf_server = self._run_build_test(flavor_id, filter_mock)'
2017-03-21 18:29:51.237632 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/libvirt/test_pci_sriov_servers.py", line 124, in _run_build_test'
2017-03-21 18:29:51.237666 |     b"    found_server = self._wait_for_state_change(found_server, 'BUILD')"
2017-03-21 18:29:51.237731 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/test_servers.py", line 64, in _wait_for_state_change'
2017-03-21 18:29:51.237782 |     b"    server = self.api.get_server(server['id'])"
2017-03-21 18:29:51.237836 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 250, in get_server'
2017-03-21 18:29:51.237870 |     b"    return self.api_get('/servers/%s' % server_id).body['server']"
2017-03-21 18:29:51.237919 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 207, in api_get'
2017-03-21 18:29:51.237951 |     b'    return APIResponse(self.api_request(relative_uri, **kwargs))'
2017-03-21 18:29:51.238002 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 189, in api_request'
2017-03-21 18:29:51.238032 |     b'    raise OpenStackApiNotFoundException(response=response)'
2017-03-21 18:29:51.238067 |     b'nova.tests.functional.api.client.OpenStackApiNotFoundException: Item not found'
2017-03-21 18:29:51.238085 |     b''

Looks like it fails while waiting for a server to be active.

The test is polling every .1 second, so I'm wondering if we're hitting a
window between the time that the instance is created and mapped to a
cell and the build_request is deleted, but I thought we hit something
like that in Tempest in Ocata and fixed it (but Tempest polls every 1
second, not every .1 second).

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: functional testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1674809

Title:
  test_create_server_with_pci_dev_and_numa functional test randomly
  fails with "OpenStackApiNotFoundException: Item not found"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/03/448203/2/check/gate-nova-tox-functional-
  py35-ubuntu-xenial/5f60ae1/console.html#_2017-03-21_18_29_51_237243

  2017-03-21 18:29:51.237243 | nova.tests.functional.libvirt.test_pci_sriov_servers.SRIOVServersTest.test_create_server_with_pci_dev_and_numa
  2017-03-21 18:29:51.237307 | --------------------------------------------------------------------------------------------------------------
  2017-03-21 18:29:51.237322 | 
  2017-03-21 18:29:51.237340 | Captured traceback:
  2017-03-21 18:29:51.237358 | ~~~~~~~~~~~~~~~~~~~
  2017-03-21 18:29:51.237381 |     b'Traceback (most recent call last):'
  2017-03-21 18:29:51.237440 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/.tox/functional-py35/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched'
  2017-03-21 18:29:51.237466 |     b'    return func(*args, **keywargs)'
  2017-03-21 18:29:51.237530 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/libvirt/test_pci_sriov_servers.py", line 272, in test_create_server_with_pci_dev_and_numa'
  2017-03-21 18:29:51.237572 |     b'    pf_server = self._run_build_test(flavor_id, filter_mock)'
  2017-03-21 18:29:51.237632 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/libvirt/test_pci_sriov_servers.py", line 124, in _run_build_test'
  2017-03-21 18:29:51.237666 |     b"    found_server = self._wait_for_state_change(found_server, 'BUILD')"
  2017-03-21 18:29:51.237731 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/test_servers.py", line 64, in _wait_for_state_change'
  2017-03-21 18:29:51.237782 |     b"    server = self.api.get_server(server['id'])"
  2017-03-21 18:29:51.237836 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 250, in get_server'
  2017-03-21 18:29:51.237870 |     b"    return self.api_get('/servers/%s' % server_id).body['server']"
  2017-03-21 18:29:51.237919 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 207, in api_get'
  2017-03-21 18:29:51.237951 |     b'    return APIResponse(self.api_request(relative_uri, **kwargs))'
  2017-03-21 18:29:51.238002 |     b'  File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 189, in api_request'
  2017-03-21 18:29:51.238032 |     b'    raise OpenStackApiNotFoundException(response=response)'
  2017-03-21 18:29:51.238067 |     b'nova.tests.functional.api.client.OpenStackApiNotFoundException: Item not found'
  2017-03-21 18:29:51.238085 |     b''

  Looks like it fails while waiting for a server to be active.

  The test is polling every .1 second, so I'm wondering if we're hitting
  a window between the time that the instance is created and mapped to a
  cell and the build_request is deleted, but I thought we hit something
  like that in Tempest in Ocata and fixed it (but Tempest polls every 1
  second, not every .1 second).

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1674809/+subscriptions