yahoo-eng-team team mailing list archive

[OpenStack Infra <1746032@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/538961
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4a2c9a4887a219a6d4dfe83c430b040713fc4109
Submitter: Zuul
Branch:    master

commit 4a2c9a4887a219a6d4dfe83c430b040713fc4109
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Mon Jan 29 10:50:36 2018 -0500

    Rollback instance.image_ref on failed rebuild
    
    When rebuilding and changing the image, we run the new image
    through the scheduler to see if it's valid for the instance
    on its current compute host. The API saves off the new image
    ref on the instance before casting to conductor to run through
    the scheduler. If the scheduler fails, the instance.image_ref was
    not being rolled back, which meant a user could attempt the rebuild
    with the same invalid image a second time and the API, seeing the
    instance.image_ref hasn't changed (even though it's not the actual
    backing image for the server), will bypass the scheduler and rebuild
    the instance with that invalid image.
    
    This fixes the issue by using the original image ref, passed from
    API to conductor during rebuild, to reset the instance.image_ref
    in the case of a failure.
    
    Note that there are other things changed on the instance in the API
    which this patch does not attempt to recover as that's a bigger
    work item which likely involves substantial refactoring of the code.
    
    Closes-Bug: #1746032
    
    Change-Id: I3399a66fe9b1297cd6b0dca440145393ceaef41f


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1746032

Title:
  By rebuilding twice with the same "forbidden" image one can circumvent
  scheduler rebuild restrictions

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Won't Fix
Status in OpenStack Compute (nova) ocata series:
  In Progress
Status in OpenStack Compute (nova) pike series:
  In Progress

Bug description:
  Description
  ===========

  Since CVE-2017-16239, we call to the scheduler when doing a rebuild
  with a new image. If the scheduler refuses a rebuild because a filter
  forbids the new image on the instance's host (for example,
  IsolatedHostsFilter), at first there was no indication of this in the
  API (bug 1744325). Currently, with the fix for bug 1744325 merged [1],
  the instance goes to ERROR to indicate the refused rebuild. However,
  by rebuilding again with the same "forbidden" image it is possible to
  circumvent scheduler restrictions.

  Steps to reproduce
  ==================

  1. Configure IsolatedHostsFilter:

     [filter_scheduler]
     enabled_filters = [...],IsolatedHostsFilter
     isolated_images = 41d3e5ca-14cf-436c-9413-4826b5c8bdb1
     isolated_hosts = ubuntu
     restrict_isolated_hosts_to_isolated_images = true

  2. Have two images, one isolated and one not:

     $ openstack image list

       8d0581a5-ed9d-4b98-a766-a41efbc99929 | centos | active
       41d3e5ca-14cf-436c-9413-4826b5c8bdb1 | cirros-0.3.5-x86_64-disk | active

       cirros is the isolated one

  3. Have only one hypervisor (the isolated one):

     $ openstack hypervisor list

       ubuntu | QEMU | 192.168.100.194 | up

  5. Boot a cirros (isolated) image:

     $ openstack server create \
       --image 41d3e5ca-14cf-436c-9413-4826b5c8bdb1 \
       --flavor m1.nano \
       cirros-test-expect-success

     $ openstack server list

       cirros-test-expect-success | ACTIVE | [...] |
  cirros-0.3.5-x86_64-disk | m1.nano

  6. Rebuild the cirros instance with centos (this should be refused by
  the scheduler):

     $ nova --debug rebuild cirros-test-expect-success centos

       DEBUG (session:722) POST call to compute for
       http://192.168.100.194/compute/v2.1/servers/d9d98bf7-623e-4587-b82c-06f36abf59cb/action
       used request id req-c234346a-6e05-47cf-a0cd-45f89d11e15d

  8. Observe the instance going to ERROR,
     but still showing the new centos image :

     $ nova show cirros-test-expect-success

       [...]
       status | ERROR
       image  | centos (8d0581a5-ed9d-4b98-a766-a41efbc99929)
       [...]

  9. Rebuild again with the same centos image:

     $ nova rebuild cirros-test-expect-success centos

  10. The rebuild goes through.

  
  Expected result
  ===============

  At step 10, the rebuild should still be refused.

  Actual result
  =============

  The rebuild is allowed.

  Environment
  ===========

  1. Exact version of OpenStack you are running. See the following

     Was reported in Red Hat OpenStack 12, affects newton through
  master.

  2. Which hypervisor did you use?

     libvirt+kvm

  [1] https://review.openstack.org/#/c/536268/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1746032/+subscriptions