← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76138

[Bug 1807407] [NEW] snapshot related tests intermittently fail in cells v1 job

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

There are two snapshot test failures in this job run under class
ImagesOneServerTestJSON:

http://logs.openstack.org/47/623247/2/check/nova-cells-v1/18338f0/job-
output.txt.gz

2018-12-06 23:40:45.318619 | primary | {1} tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name [0.238073s] ... FAILED
2018-12-06 23:40:45.318645 | primary |
2018-12-06 23:40:45.318688 | primary | Captured pythonlogging:
2018-12-06 23:40:45.318730 | primary | ~~~~~~~~~~~~~~~~~~~~~~~
2018-12-06 23:40:45.318952 | primary |     2018-12-06 23:40:45,023 5498 INFO     [tempest.lib.common.rest_client] Request (ImagesOneServerTestJSON:test_create_image_specify_multibyte_character_image_name): 409 POST http://158.69.74.95/compute/v2.1/servers/b76c5c2c-290f-4734-b751-57585b6ee86c/action 0.122s
2018-12-06 23:40:45.319576 | primary |     2018-12-06 23:40:45,023 5498 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2018-12-06 23:40:45.319635 | primary |             Body: {"createImage": {"name": "tempest-\u20a1-25178005"}}
2018-12-06 23:40:45.319981 | primary |         Response - Headers: {u'x-compute-request-id': 'req-07883b0e-53dd-48c9-ae73-28f232ad2547', u'x-openstack-request-id': 'req-07883b0e-53dd-48c9-ae73-28f232ad2547', u'openstack-api-version': 'compute 2.1', 'status': '409', u'x-openstack-nova-api-version': '2.1', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-location': 'http://158.69.74.95/compute/v2.1/servers/b76c5c2c-290f-4734-b751-57585b6ee86c/action', u'content-length': '151', u'server': 'Apache/2.4.18 (Ubuntu)', u'date': 'Thu, 06 Dec 2018 23:40:44 GMT', u'content-type': 'application/json; charset=UTF-8', u'connection': 'close'}
2018-12-06 23:40:45.320102 | primary |             Body: {"conflictingRequest": {"message": "Cannot 'createImage' instance b76c5c2c-290f-4734-b751-57585b6ee86c while it is in vm_state building", "code": 409}}
2018-12-06 23:40:45.321130 | primary |
2018-12-06 23:40:45.321154 | primary |
2018-12-06 23:40:45.321174 | primary | Captured traceback:
2018-12-06 23:40:45.321194 | primary | ~~~~~~~~~~~~~~~~~~~
2018-12-06 23:40:45.321223 | primary |     Traceback (most recent call last):
2018-12-06 23:40:45.321302 | primary |       File "tempest/api/compute/images/test_images_oneserver.py", line 105, in test_create_image_specify_multibyte_character_image_name
2018-12-06 23:40:45.321333 | primary |         self.server_id, name=utf8_name)
2018-12-06 23:40:45.321385 | primary |       File "tempest/lib/services/compute/images_client.py", line 43, in create_image
2018-12-06 23:40:45.321404 | primary |         post_body)
2018-12-06 23:40:45.321447 | primary |       File "tempest/lib/common/rest_client.py", line 280, in post
2018-12-06 23:40:45.321514 | primary |         return self.request('POST', url, extra_headers, headers, body, chunked)
2018-12-06 23:40:45.321662 | primary |       File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
2018-12-06 23:40:45.321744 | primary |         method, url, extra_headers, headers, body, chunked)
2018-12-06 23:40:45.321818 | primary |       File "tempest/lib/common/rest_client.py", line 676, in request
2018-12-06 23:40:45.321876 | primary |         self._error_checker(resp, resp_body)
2018-12-06 23:40:45.321931 | primary |       File "tempest/lib/common/rest_client.py", line 797, in _error_checker
2018-12-06 23:40:45.321969 | primary |         raise exceptions.Conflict(resp_body, resp=resp)
2018-12-06 23:40:45.322016 | primary |     tempest.lib.exceptions.Conflict: Conflict with state of target resource
2018-12-06 23:40:45.322099 | primary |     Details: {u'message': u"Cannot 'createImage' instance b76c5c2c-290f-4734-b751-57585b6ee86c while it is in vm_state building", u'code': 409}

and

2018-12-06 23:40:45.006764 | primary | {1}
tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image
[210.094519s] ... FAILED

2018-12-06 23:40:45.315570 | primary | Captured traceback:
2018-12-06 23:40:45.315590 | primary | ~~~~~~~~~~~~~~~~~~~
2018-12-06 23:40:45.316307 | primary |     Traceback (most recent call last):
2018-12-06 23:40:45.316397 | primary |       File "tempest/api/compute/images/test_images_oneserver.py", line 70, in test_create_delete_image
2018-12-06 23:40:45.316423 | primary |         wait_until='ACTIVE')
2018-12-06 23:40:45.316473 | primary |       File "tempest/api/compute/base.py", line 373, in create_image_from_server
2018-12-06 23:40:45.317497 | primary |         server_id, 'ACTIVE')
2018-12-06 23:40:45.317580 | primary |       File "tempest/common/waiters.py", line 96, in wait_for_server_status
2018-12-06 23:40:45.317619 | primary |         raise lib_exc.TimeoutException(message)
2018-12-06 23:40:45.318138 | primary |     tempest.lib.exceptions.TimeoutException: Request timed out
2018-12-06 23:40:45.318320 | primary |     Details: (ImagesOneServerTestJSON:test_create_delete_image) Server b76c5c2c-290f-4734-b751-57585b6ee86c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: BUILD. Current task state: None.

We've seen similar race bugs for the cells v1 job and snapshot tests in
the past, see bug 1620761.

Given the deprecated nature of cells v1 I suggest we simply disable
snapshot testing in the cells v1 job.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: cells gate-failure snapshot v1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1807407

Title:
  snapshot related tests intermittently fail in cells v1 job

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  There are two snapshot test failures in this job run under class
  ImagesOneServerTestJSON:

  http://logs.openstack.org/47/623247/2/check/nova-cells-v1/18338f0/job-
  output.txt.gz

  2018-12-06 23:40:45.318619 | primary | {1} tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name [0.238073s] ... FAILED
  2018-12-06 23:40:45.318645 | primary |
  2018-12-06 23:40:45.318688 | primary | Captured pythonlogging:
  2018-12-06 23:40:45.318730 | primary | ~~~~~~~~~~~~~~~~~~~~~~~
  2018-12-06 23:40:45.318952 | primary |     2018-12-06 23:40:45,023 5498 INFO     [tempest.lib.common.rest_client] Request (ImagesOneServerTestJSON:test_create_image_specify_multibyte_character_image_name): 409 POST http://158.69.74.95/compute/v2.1/servers/b76c5c2c-290f-4734-b751-57585b6ee86c/action 0.122s
  2018-12-06 23:40:45.319576 | primary |     2018-12-06 23:40:45,023 5498 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  2018-12-06 23:40:45.319635 | primary |             Body: {"createImage": {"name": "tempest-\u20a1-25178005"}}
  2018-12-06 23:40:45.319981 | primary |         Response - Headers: {u'x-compute-request-id': 'req-07883b0e-53dd-48c9-ae73-28f232ad2547', u'x-openstack-request-id': 'req-07883b0e-53dd-48c9-ae73-28f232ad2547', u'openstack-api-version': 'compute 2.1', 'status': '409', u'x-openstack-nova-api-version': '2.1', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-location': 'http://158.69.74.95/compute/v2.1/servers/b76c5c2c-290f-4734-b751-57585b6ee86c/action', u'content-length': '151', u'server': 'Apache/2.4.18 (Ubuntu)', u'date': 'Thu, 06 Dec 2018 23:40:44 GMT', u'content-type': 'application/json; charset=UTF-8', u'connection': 'close'}
  2018-12-06 23:40:45.320102 | primary |             Body: {"conflictingRequest": {"message": "Cannot 'createImage' instance b76c5c2c-290f-4734-b751-57585b6ee86c while it is in vm_state building", "code": 409}}
  2018-12-06 23:40:45.321130 | primary |
  2018-12-06 23:40:45.321154 | primary |
  2018-12-06 23:40:45.321174 | primary | Captured traceback:
  2018-12-06 23:40:45.321194 | primary | ~~~~~~~~~~~~~~~~~~~
  2018-12-06 23:40:45.321223 | primary |     Traceback (most recent call last):
  2018-12-06 23:40:45.321302 | primary |       File "tempest/api/compute/images/test_images_oneserver.py", line 105, in test_create_image_specify_multibyte_character_image_name
  2018-12-06 23:40:45.321333 | primary |         self.server_id, name=utf8_name)
  2018-12-06 23:40:45.321385 | primary |       File "tempest/lib/services/compute/images_client.py", line 43, in create_image
  2018-12-06 23:40:45.321404 | primary |         post_body)
  2018-12-06 23:40:45.321447 | primary |       File "tempest/lib/common/rest_client.py", line 280, in post
  2018-12-06 23:40:45.321514 | primary |         return self.request('POST', url, extra_headers, headers, body, chunked)
  2018-12-06 23:40:45.321662 | primary |       File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
  2018-12-06 23:40:45.321744 | primary |         method, url, extra_headers, headers, body, chunked)
  2018-12-06 23:40:45.321818 | primary |       File "tempest/lib/common/rest_client.py", line 676, in request
  2018-12-06 23:40:45.321876 | primary |         self._error_checker(resp, resp_body)
  2018-12-06 23:40:45.321931 | primary |       File "tempest/lib/common/rest_client.py", line 797, in _error_checker
  2018-12-06 23:40:45.321969 | primary |         raise exceptions.Conflict(resp_body, resp=resp)
  2018-12-06 23:40:45.322016 | primary |     tempest.lib.exceptions.Conflict: Conflict with state of target resource
  2018-12-06 23:40:45.322099 | primary |     Details: {u'message': u"Cannot 'createImage' instance b76c5c2c-290f-4734-b751-57585b6ee86c while it is in vm_state building", u'code': 409}

  and

  2018-12-06 23:40:45.006764 | primary | {1}
  tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image
  [210.094519s] ... FAILED

  2018-12-06 23:40:45.315570 | primary | Captured traceback:
  2018-12-06 23:40:45.315590 | primary | ~~~~~~~~~~~~~~~~~~~
  2018-12-06 23:40:45.316307 | primary |     Traceback (most recent call last):
  2018-12-06 23:40:45.316397 | primary |       File "tempest/api/compute/images/test_images_oneserver.py", line 70, in test_create_delete_image
  2018-12-06 23:40:45.316423 | primary |         wait_until='ACTIVE')
  2018-12-06 23:40:45.316473 | primary |       File "tempest/api/compute/base.py", line 373, in create_image_from_server
  2018-12-06 23:40:45.317497 | primary |         server_id, 'ACTIVE')
  2018-12-06 23:40:45.317580 | primary |       File "tempest/common/waiters.py", line 96, in wait_for_server_status
  2018-12-06 23:40:45.317619 | primary |         raise lib_exc.TimeoutException(message)
  2018-12-06 23:40:45.318138 | primary |     tempest.lib.exceptions.TimeoutException: Request timed out
  2018-12-06 23:40:45.318320 | primary |     Details: (ImagesOneServerTestJSON:test_create_delete_image) Server b76c5c2c-290f-4734-b751-57585b6ee86c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: BUILD. Current task state: None.

  We've seen similar race bugs for the cells v1 job and snapshot tests
  in the past, see bug 1620761.

  Given the deprecated nature of cells v1 I suggest we simply disable
  snapshot testing in the cells v1 job.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1807407/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next