yahoo-eng-team team mailing list archive

[OpenStack Infra <1807407@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/623538
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e02fbb53d5fe093469d2aa188007ed2c5c67c98b
Submitter: Zuul
Branch:    master

commit e02fbb53d5fe093469d2aa188007ed2c5c67c98b
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Fri Dec 7 10:59:37 2018 -0500

    Move nova-cells-v1 to experimental queue
    
    Cells v1 has been deprecated since Pike. CERN
    has been running with cells v2 since Queens.
    The cells v1 job used to be the only thing that
    ran with nova-network, but we switched the job
    to use neutron in Rocky:
    
      I9de6b710baffdffcd1d7ab19897d5776ef27ae7e
    
    The cells v1 job also suffers from intermittent
    test failures, like with snapshot tests.
    
    Given the deprecated nature of cells v1 we should
    just move it to the experimental queue so that it
    can be run on-demand if desired but does not gate
    on all nova changes, thus further moving along its
    eventual removal.
    
    This change also updates the cells v1 status doc
    and adds some documentation about the different
    job queues that nova uses for integration testing.
    
    Change-Id: I74985f1946fffd0ae4d38604696d0d4656b6bf4e
    Closes-Bug: #1807407


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1807407

Title:
  snapshot related tests intermittently fail in cells v1 job

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  There are two snapshot test failures in this job run under class
  ImagesOneServerTestJSON:

  http://logs.openstack.org/47/623247/2/check/nova-cells-v1/18338f0/job-
  output.txt.gz

  2018-12-06 23:40:45.318619 | primary | {1} tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name [0.238073s] ... FAILED
  2018-12-06 23:40:45.318645 | primary |
  2018-12-06 23:40:45.318688 | primary | Captured pythonlogging:
  2018-12-06 23:40:45.318730 | primary | ~~~~~~~~~~~~~~~~~~~~~~~
  2018-12-06 23:40:45.318952 | primary |     2018-12-06 23:40:45,023 5498 INFO     [tempest.lib.common.rest_client] Request (ImagesOneServerTestJSON:test_create_image_specify_multibyte_character_image_name): 409 POST http://158.69.74.95/compute/v2.1/servers/b76c5c2c-290f-4734-b751-57585b6ee86c/action 0.122s
  2018-12-06 23:40:45.319576 | primary |     2018-12-06 23:40:45,023 5498 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  2018-12-06 23:40:45.319635 | primary |             Body: {"createImage": {"name": "tempest-\u20a1-25178005"}}
  2018-12-06 23:40:45.319981 | primary |         Response - Headers: {u'x-compute-request-id': 'req-07883b0e-53dd-48c9-ae73-28f232ad2547', u'x-openstack-request-id': 'req-07883b0e-53dd-48c9-ae73-28f232ad2547', u'openstack-api-version': 'compute 2.1', 'status': '409', u'x-openstack-nova-api-version': '2.1', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-location': 'http://158.69.74.95/compute/v2.1/servers/b76c5c2c-290f-4734-b751-57585b6ee86c/action', u'content-length': '151', u'server': 'Apache/2.4.18 (Ubuntu)', u'date': 'Thu, 06 Dec 2018 23:40:44 GMT', u'content-type': 'application/json; charset=UTF-8', u'connection': 'close'}
  2018-12-06 23:40:45.320102 | primary |             Body: {"conflictingRequest": {"message": "Cannot 'createImage' instance b76c5c2c-290f-4734-b751-57585b6ee86c while it is in vm_state building", "code": 409}}
  2018-12-06 23:40:45.321130 | primary |
  2018-12-06 23:40:45.321154 | primary |
  2018-12-06 23:40:45.321174 | primary | Captured traceback:
  2018-12-06 23:40:45.321194 | primary | ~~~~~~~~~~~~~~~~~~~
  2018-12-06 23:40:45.321223 | primary |     Traceback (most recent call last):
  2018-12-06 23:40:45.321302 | primary |       File "tempest/api/compute/images/test_images_oneserver.py", line 105, in test_create_image_specify_multibyte_character_image_name
  2018-12-06 23:40:45.321333 | primary |         self.server_id, name=utf8_name)
  2018-12-06 23:40:45.321385 | primary |       File "tempest/lib/services/compute/images_client.py", line 43, in create_image
  2018-12-06 23:40:45.321404 | primary |         post_body)
  2018-12-06 23:40:45.321447 | primary |       File "tempest/lib/common/rest_client.py", line 280, in post
  2018-12-06 23:40:45.321514 | primary |         return self.request('POST', url, extra_headers, headers, body, chunked)
  2018-12-06 23:40:45.321662 | primary |       File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
  2018-12-06 23:40:45.321744 | primary |         method, url, extra_headers, headers, body, chunked)
  2018-12-06 23:40:45.321818 | primary |       File "tempest/lib/common/rest_client.py", line 676, in request
  2018-12-06 23:40:45.321876 | primary |         self._error_checker(resp, resp_body)
  2018-12-06 23:40:45.321931 | primary |       File "tempest/lib/common/rest_client.py", line 797, in _error_checker
  2018-12-06 23:40:45.321969 | primary |         raise exceptions.Conflict(resp_body, resp=resp)
  2018-12-06 23:40:45.322016 | primary |     tempest.lib.exceptions.Conflict: Conflict with state of target resource
  2018-12-06 23:40:45.322099 | primary |     Details: {u'message': u"Cannot 'createImage' instance b76c5c2c-290f-4734-b751-57585b6ee86c while it is in vm_state building", u'code': 409}

  and

  2018-12-06 23:40:45.006764 | primary | {1}
  tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image
  [210.094519s] ... FAILED

  2018-12-06 23:40:45.315570 | primary | Captured traceback:
  2018-12-06 23:40:45.315590 | primary | ~~~~~~~~~~~~~~~~~~~
  2018-12-06 23:40:45.316307 | primary |     Traceback (most recent call last):
  2018-12-06 23:40:45.316397 | primary |       File "tempest/api/compute/images/test_images_oneserver.py", line 70, in test_create_delete_image
  2018-12-06 23:40:45.316423 | primary |         wait_until='ACTIVE')
  2018-12-06 23:40:45.316473 | primary |       File "tempest/api/compute/base.py", line 373, in create_image_from_server
  2018-12-06 23:40:45.317497 | primary |         server_id, 'ACTIVE')
  2018-12-06 23:40:45.317580 | primary |       File "tempest/common/waiters.py", line 96, in wait_for_server_status
  2018-12-06 23:40:45.317619 | primary |         raise lib_exc.TimeoutException(message)
  2018-12-06 23:40:45.318138 | primary |     tempest.lib.exceptions.TimeoutException: Request timed out
  2018-12-06 23:40:45.318320 | primary |     Details: (ImagesOneServerTestJSON:test_create_delete_image) Server b76c5c2c-290f-4734-b751-57585b6ee86c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: BUILD. Current task state: None.

  We've seen similar race bugs for the cells v1 job and snapshot tests
  in the past, see bug 1620761.

  Given the deprecated nature of cells v1 I suggest we simply disable
  snapshot testing in the cells v1 job.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1807407/+subscriptions