← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #61058

[Bug 1660878] [NEW] test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-
full-ubuntu-xenial/f218227/console.html#_2017-02-01_02_06_33_592237

2017-02-01 02:06:33.592237 | tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_deleted_server[id-581a397d-5eab-486f-9cf9-1014bbd4c984,negative]
2017-02-01 02:06:33.592305 | ------------------------------------------------------------------------------------------------------------------------------------------------------
2017-02-01 02:06:33.592321 | 
2017-02-01 02:06:33.592340 | Captured traceback:
2017-02-01 02:06:33.592367 | ~~~~~~~~~~~~~~~~~~~
2017-02-01 02:06:33.592398 |     Traceback (most recent call last):
2017-02-01 02:06:33.592453 |       File "tempest/api/compute/servers/test_servers_negative.py", line 190, in test_reboot_deleted_server
2017-02-01 02:06:33.593010 |         server['id'], type='SOFT')
2017-02-01 02:06:33.593072 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 485, in assertRaises
2017-02-01 02:06:33.593110 |         self.assertThat(our_callable, matcher)
2017-02-01 02:06:33.593162 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 496, in assertThat
2017-02-01 02:06:33.593205 |         mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
2017-02-01 02:06:33.593266 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 547, in _matchHelper
2017-02-01 02:06:33.593294 |         mismatch = matcher.match(matchee)
2017-02-01 02:06:33.593345 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
2017-02-01 02:06:33.593388 |         mismatch = self.exception_matcher.match(exc_info)
2017-02-01 02:06:33.593443 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
2017-02-01 02:06:33.593468 |         mismatch = matcher.match(matchee)
2017-02-01 02:06:33.593515 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 475, in match
2017-02-01 02:06:33.593544 |         reraise(*matchee)
2017-02-01 02:06:33.593597 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
2017-02-01 02:06:33.593618 |         result = matchee()
2017-02-01 02:06:33.593667 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 1049, in __call__
2017-02-01 02:06:33.593699 |         return self._callable_object(*self._args, **self._kwargs)
2017-02-01 02:06:33.593736 |       File "tempest/lib/services/compute/servers_client.py", line 236, in reboot_server
2017-02-01 02:06:33.593777 |         return self.action(server_id, 'reboot', **kwargs)
2017-02-01 02:06:33.593814 |       File "tempest/lib/services/compute/servers_client.py", line 186, in action
2017-02-01 02:06:33.594170 |         post_body)
2017-02-01 02:06:33.594219 |       File "tempest/lib/common/rest_client.py", line 275, in post
2017-02-01 02:06:33.594261 |         return self.request('POST', url, extra_headers, headers, body, chunked)
2017-02-01 02:06:33.594298 |       File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
2017-02-01 02:06:33.594328 |         method, url, extra_headers, headers, body, chunked)
2017-02-01 02:06:33.594360 |       File "tempest/lib/common/rest_client.py", line 663, in request
2017-02-01 02:06:33.594392 |         self._error_checker(resp, resp_body)
2017-02-01 02:06:33.594449 |       File "tempest/lib/common/rest_client.py", line 775, in _error_checker
2017-02-01 02:06:33.594496 |         raise exceptions.Conflict(resp_body, resp=resp)
2017-02-01 02:06:33.594555 |     tempest.lib.exceptions.Conflict: An object with that identifier already exists
2017-02-01 02:06:33.594641 |     Details: {u'code': 409, u'message': u"Cannot 'reboot' instance d9978a80-0349-4cb8-84fa-46aa0b8eaa16 while it is in vm_state building"}

http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/f218227/logs/screen-n-api.txt.gz#_2017-02-01_01_25_02_367

The test creates a server, then deletes the server and waits for it to
be gone and then tries to reboot it and expects a 404 but gets a 409
because the instance is building.

We're probably hitting a window of time between when we have a build
request and a real instance due to the recent change of moving instance
creation from nova-api to nova-conductor.

Looks like this started on 1/27:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Cannot%20'reboot'%20instance%5C%22%20AND%20message%3A%5C%22while%20it%20is%20in%20vm_state%20building%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

Which is when this merged: https://review.openstack.org/#/c/319379/

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: cells ocata-rc-potential

** Tags added: ocata-rc-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1660878

Title:
  test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance
  while it is in vm_state building"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-
  full-ubuntu-xenial/f218227/console.html#_2017-02-01_02_06_33_592237

  2017-02-01 02:06:33.592237 | tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_deleted_server[id-581a397d-5eab-486f-9cf9-1014bbd4c984,negative]
  2017-02-01 02:06:33.592305 | ------------------------------------------------------------------------------------------------------------------------------------------------------
  2017-02-01 02:06:33.592321 | 
  2017-02-01 02:06:33.592340 | Captured traceback:
  2017-02-01 02:06:33.592367 | ~~~~~~~~~~~~~~~~~~~
  2017-02-01 02:06:33.592398 |     Traceback (most recent call last):
  2017-02-01 02:06:33.592453 |       File "tempest/api/compute/servers/test_servers_negative.py", line 190, in test_reboot_deleted_server
  2017-02-01 02:06:33.593010 |         server['id'], type='SOFT')
  2017-02-01 02:06:33.593072 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 485, in assertRaises
  2017-02-01 02:06:33.593110 |         self.assertThat(our_callable, matcher)
  2017-02-01 02:06:33.593162 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 496, in assertThat
  2017-02-01 02:06:33.593205 |         mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
  2017-02-01 02:06:33.593266 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 547, in _matchHelper
  2017-02-01 02:06:33.593294 |         mismatch = matcher.match(matchee)
  2017-02-01 02:06:33.593345 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
  2017-02-01 02:06:33.593388 |         mismatch = self.exception_matcher.match(exc_info)
  2017-02-01 02:06:33.593443 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
  2017-02-01 02:06:33.593468 |         mismatch = matcher.match(matchee)
  2017-02-01 02:06:33.593515 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 475, in match
  2017-02-01 02:06:33.593544 |         reraise(*matchee)
  2017-02-01 02:06:33.593597 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
  2017-02-01 02:06:33.593618 |         result = matchee()
  2017-02-01 02:06:33.593667 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 1049, in __call__
  2017-02-01 02:06:33.593699 |         return self._callable_object(*self._args, **self._kwargs)
  2017-02-01 02:06:33.593736 |       File "tempest/lib/services/compute/servers_client.py", line 236, in reboot_server
  2017-02-01 02:06:33.593777 |         return self.action(server_id, 'reboot', **kwargs)
  2017-02-01 02:06:33.593814 |       File "tempest/lib/services/compute/servers_client.py", line 186, in action
  2017-02-01 02:06:33.594170 |         post_body)
  2017-02-01 02:06:33.594219 |       File "tempest/lib/common/rest_client.py", line 275, in post
  2017-02-01 02:06:33.594261 |         return self.request('POST', url, extra_headers, headers, body, chunked)
  2017-02-01 02:06:33.594298 |       File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
  2017-02-01 02:06:33.594328 |         method, url, extra_headers, headers, body, chunked)
  2017-02-01 02:06:33.594360 |       File "tempest/lib/common/rest_client.py", line 663, in request
  2017-02-01 02:06:33.594392 |         self._error_checker(resp, resp_body)
  2017-02-01 02:06:33.594449 |       File "tempest/lib/common/rest_client.py", line 775, in _error_checker
  2017-02-01 02:06:33.594496 |         raise exceptions.Conflict(resp_body, resp=resp)
  2017-02-01 02:06:33.594555 |     tempest.lib.exceptions.Conflict: An object with that identifier already exists
  2017-02-01 02:06:33.594641 |     Details: {u'code': 409, u'message': u"Cannot 'reboot' instance d9978a80-0349-4cb8-84fa-46aa0b8eaa16 while it is in vm_state building"}

  http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/f218227/logs/screen-n-api.txt.gz#_2017-02-01_01_25_02_367

  The test creates a server, then deletes the server and waits for it to
  be gone and then tries to reboot it and expects a 404 but gets a 409
  because the instance is building.

  We're probably hitting a window of time between when we have a build
  request and a real instance due to the recent change of moving
  instance creation from nova-api to nova-conductor.

  Looks like this started on 1/27:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Cannot%20'reboot'%20instance%5C%22%20AND%20message%3A%5C%22while%20it%20is%20in%20vm_state%20building%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

  Which is when this merged: https://review.openstack.org/#/c/319379/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1660878/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next