yahoo-eng-team team mailing list archive

[OpenStack Infra <1744325@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/536268
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d03a890a34f632adc9a19a33c8a5aebbccec50e4
Submitter: Zuul
Branch:    master

commit d03a890a34f632adc9a19a33c8a5aebbccec50e4
Author: int32bit <krystism@xxxxxxxxx>
Date:   Mon Jan 22 17:05:53 2018 +0800

    Set server status to ERROR if rebuild failed
    
    Currently there is no indication that the rebuild was refused,
    and worse, we may have a wrong imageref for the instance.
    
    This patch set the instance to ERROR status if rebuild failed in the
    scheduling stage. The user can rebuild the instance with valid image
    to get it out of ERROR state and reset with right instance metadata and
    properties.
    
    Closes-Bug: 1744325
    
    Change-Id: Ibb7bee15a3d4ee6f0ef53ba12e8b41f65a1fe999


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1744325

Title:
  If a rebuild is refused by the scheduler, the instance's imageref is
  not rolled back

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Won't Fix
Status in OpenStack Compute (nova) ocata series:
  In Progress
Status in OpenStack Compute (nova) pike series:
  In Progress

Bug description:
  Description
  ===========

  Since CVE-2017-16239, we now go through the scheduler for rebuilds. If
  the scheduler refuses a rebuild with a new image because of filter
  constraints (for example IsolatedHostsFilter), the instance's imageref
  is set to the new image and never rolled back.

  Steps to reproduce
  ==================

  1. Configure IsolatedHostsFilter:
   
     [filter_scheduler]
     enabled_filters = [...],IsolatedHostsFilter
     isolated_images = 41d3e5ca-14cf-436c-9413-4826b5c8bdb1
     isolated_hosts = ubuntu
     restrict_isolated_hosts_to_isolated_images = true

  2. Have two images, one isolated and one not:

     $ openstack image list

       8d0581a5-ed9d-4b98-a766-a41efbc99929 | centos                   | active
       41d3e5ca-14cf-436c-9413-4826b5c8bdb1 | cirros-0.3.5-x86_64-disk | active

       cirros is the isolated one

  3. Have only one hypervisor (the isolated one):

     $ openstack hypervisor list

       ubuntu              | QEMU            | 192.168.100.194 | up

  4. To confirm, boot a centos (non-isolated) image, expecting it to be
  refused by the scheduler:

     $ openstack server create \
       --image 8d0581a5-ed9d-4b98-a766-a41efbc99929 \
       --flavor \
       m1.nano centos-test-expect-fail

     $ openstack server list

       centos-test-expect-fail | ERROR  |          | centos | m1.nano

  5. Boot a cirros (isolated) image:

     $ openstack server create \
       --image 41d3e5ca-14cf-436c-9413-4826b5c8bdb1 \
       --flavor m1.nano \
       cirros-test-expect-success

     $ openstack server list

       cirros-test-expect-success | ACTIVE | [...] |
  cirros-0.3.5-x86_64-disk | m1.nano

  6. Rebuild the cirros instance with centos:

     $ nova --debug rebuild cirros-test-expect-success centos
     
       DEBUG (session:722) POST call to compute for 
       http://192.168.100.194/compute/v2.1/servers/d9d98bf7-623e-4587-b82c-06f36abf59cb/action
       used request id req-c234346a-6e05-47cf-a0cd-45f89d11e15d

  7. Observer the rebuild being refused in the conductor:

     WARNING nova.conductor.manager
     [None req-c234346a-6e05-47cf-a0cd-45f89d11e15d demo admin]
     [instance: d9d98bf7-623e-4587-b82c-06f36abf59cb]
     No valid host found for rebuild: NoValidHost_Remote:
     No valid host was found. There are not enough hosts available.

  8. Observe the API is showing the new centos image for the instance:

     $ nova show cirros-test-expect-success

       [...]
       image | centos (8d0581a5-ed9d-4b98-a766-a41efbc99929)
       [...]

  Expected result
  ===============

  Some indication that the rebuild was refused, or at least rolling back
  the instance's imageref.

  Actual result
  =============

  No indication that the rebuild was refused, and worse, we now have a
  wrong imageref for the instance.

  Environment
  ===========

  1. Exact version of OpenStack you are running. See the following
     
     This was picked up by QE for stable/pike, and is still present in master,
     and presumably in all versions affected by the CVE fix, including newton,
     which is now EOL.

  2. Which hypervisor did you use?
     libvirt+kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1744325/+subscriptions