yahoo-eng-team team mailing list archive

[Lee Yarwood <lyarwood@xxxxxxxxxx>]

Public bug reported:

tempest.api.compute.servers.test_servers.ServerShowV263Test aims to test
the optional trusted_image_certificates parameter introduced into n-api
in microversion 2.63:

Add trusted_image_certificates to REST API
https://review.opendev.org/#/c/486204/

Add new schema for Nova microversion 2.63
https://review.opendev.org/#/c/526485/

However this test currently relies on other tests having cached the
image used to create the test instance before it is run. Otherwise the
underlying logic within Nova [1] to validate the image using the
provided certificates will be executed and fail as the image is missing
image properties such as img_signature_hash_method etc [2].

[1] https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L402-L460
[2] https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2019-05-31_07_06_11_779

This behaviour can be easily reproduced in devstack by cleaning out the
local image cache and running
tempest.api.compute.servers.test_servers.ServerShowV263Test in
isolation:

$ rm -rf ~/data/nova/instances/_base/*
$ tempest run --regex tempest.api.compute.servers.test_servers.ServerShowV263Test
[..]
==============================
Failed 1 tests - output below:
==============================

tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server[id-71b8e3d5-11d2-494f-b917-b094a4afed3c]
-------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/servers/test_servers.py", line 194, in test_show_update_rebuild_list_server
        wait_until='ACTIVE')
      File "tempest/api/compute/base.py", line 246, in create_test_server
        **kwargs)
      File "tempest/common/compute.py", line 265, in create_test_server
        server['id'])
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "tempest/common/compute.py", line 236, in create_test_server
        clients.servers_client, server['id'], wait_until)
      File "tempest/common/waiters.py", line 76, in wait_for_server_status
        server_id=server_id)
    tempest.exceptions.BuildErrorException: Server 254fb4a1-74a8-4681-ad67-a287bb3cde66 failed to build and is in ERROR status
    Details: {u'message': u'Build of instance 254fb4a1-74a8-4681-ad67-a287bb3cde66 aborted: Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_hash_method.', u'code': 500, u'created': u'2019-06-06T08:38:11Z'}

Running all of the test_servers tests together however typically allows
this to pass:

$ rm -rf ~/data/nova/instances/_base/*
$ tempest run --regex tempest.api.compute.servers.test_servers
[..]
======
Totals
======
Ran: 59 tests in 150.0000 sec.
 - Passed: 58
 - Skipped: 1
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 522.7800 sec.

I'm going to tag nova into this bug report as well as I'm not sure that
the current behaviour of n-api is correct. At present we attempt to
verify the image when the optional trusted_image_certificates parameter
is provided, regardless of CONF.glance.verify_glance_signatures being
enabled within nova.conf:

https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L414

While documented that behaviour doesn't seem right and I believe led to
some incorrect assumptions being made by the initial writer of this
test.

https://docs.openstack.org/nova/latest/user/certificate-validation.html

"Certificate validation will only be performed if image signature
validation is enabled. However, the presence of trusted certificate IDs
overrides the enable_certificate_validation and verify_glance_signatures
settings. In other words, if a list of trusted certificate IDs is
provided to the instance create or rebuild commands, signature
verification and certificate validation will be performed, regardless of
their settings in the Nova configurations. See Using Signature
Verification for details."

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: tempest
     Importance: Undecided
         Status: New

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1831866

Title:
  tempest.api.compute.servers.test_servers.ServerShowV263Test only
  passing when image has already been cached

Status in OpenStack Compute (nova):
  New
Status in tempest:
  New

Bug description:
  tempest.api.compute.servers.test_servers.ServerShowV263Test aims to
  test the optional trusted_image_certificates parameter introduced into
  n-api in microversion 2.63:

  Add trusted_image_certificates to REST API
  https://review.opendev.org/#/c/486204/

  Add new schema for Nova microversion 2.63
  https://review.opendev.org/#/c/526485/

  However this test currently relies on other tests having cached the
  image used to create the test instance before it is run. Otherwise the
  underlying logic within Nova [1] to validate the image using the
  provided certificates will be executed and fail as the image is
  missing image properties such as img_signature_hash_method etc [2].

  [1] https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L402-L460
  [2] https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2019-05-31_07_06_11_779

  This behaviour can be easily reproduced in devstack by cleaning out
  the local image cache and running
  tempest.api.compute.servers.test_servers.ServerShowV263Test in
  isolation:

  $ rm -rf ~/data/nova/instances/_base/*
  $ tempest run --regex tempest.api.compute.servers.test_servers.ServerShowV263Test
  [..]
  ==============================
  Failed 1 tests - output below:
  ==============================

  tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server[id-71b8e3d5-11d2-494f-b917-b094a4afed3c]
  -------------------------------------------------------------------------------------------------

  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "tempest/api/compute/servers/test_servers.py", line 194, in test_show_update_rebuild_list_server
          wait_until='ACTIVE')
        File "tempest/api/compute/base.py", line 246, in create_test_server
          **kwargs)
        File "tempest/common/compute.py", line 265, in create_test_server
          server['id'])
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
          self.force_reraise()
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
          six.reraise(self.type_, self.value, self.tb)
        File "tempest/common/compute.py", line 236, in create_test_server
          clients.servers_client, server['id'], wait_until)
        File "tempest/common/waiters.py", line 76, in wait_for_server_status
          server_id=server_id)
      tempest.exceptions.BuildErrorException: Server 254fb4a1-74a8-4681-ad67-a287bb3cde66 failed to build and is in ERROR status
      Details: {u'message': u'Build of instance 254fb4a1-74a8-4681-ad67-a287bb3cde66 aborted: Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_hash_method.', u'code': 500, u'created': u'2019-06-06T08:38:11Z'}

  Running all of the test_servers tests together however typically
  allows this to pass:

  $ rm -rf ~/data/nova/instances/_base/*
  $ tempest run --regex tempest.api.compute.servers.test_servers
  [..]
  ======
  Totals
  ======
  Ran: 59 tests in 150.0000 sec.
   - Passed: 58
   - Skipped: 1
   - Expected Fail: 0
   - Unexpected Success: 0
   - Failed: 0
  Sum of execute time for each test: 522.7800 sec.

  I'm going to tag nova into this bug report as well as I'm not sure
  that the current behaviour of n-api is correct. At present we attempt
  to verify the image when the optional trusted_image_certificates
  parameter is provided, regardless of
  CONF.glance.verify_glance_signatures being enabled within nova.conf:

  https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L414

  While documented that behaviour doesn't seem right and I believe led
  to some incorrect assumptions being made by the initial writer of this
  test.

  https://docs.openstack.org/nova/latest/user/certificate-
  validation.html

  "Certificate validation will only be performed if image signature
  validation is enabled. However, the presence of trusted certificate
  IDs overrides the enable_certificate_validation and
  verify_glance_signatures settings. In other words, if a list of
  trusted certificate IDs is provided to the instance create or rebuild
  commands, signature verification and certificate validation will be
  performed, regardless of their settings in the Nova configurations.
  See Using Signature Verification for details."

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1831866/+subscriptions