yahoo-eng-team team mailing list archive

[Stephen Finucane <1888414@xxxxxxxxxxxxxxxxxx>]

I also see that when this fails, there are left over base files in
'/opt/stack/data/nova/instances/_base'.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1888414

Title:
  Snapshot of stopped, suspended instance fails

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Attempting to create a snapshot of a shutdown instance fails. It seems
  nova assumes the instance exists and is running when attempting to
  create the snapshot.

  # Steps to reproduce

    $ openstack server create \
        --os-compute-api-version=2.latest --flavor m1.tiny --image cirros-0.5.1-x86_64-disk \
        --nic none --wait test.server
    $ openstack server stop test.server
    $ openstack server image create test.server

  # Expected result

  A snapshot of the instance root disk should be created.

  # Actual result

  The snapshot is not created. Attempts to resume the instance fail:

    $ openstack server start test.server
    Cannot 'start' instance aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda while it is in task_state image_pending_upload (HTTP 409) (Request-ID: req-39d4bd58-366b-4b93-8d7d-72a487183088)

  # Additional details

  I see the following in the logs:

    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance instance-0000000c disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid 'aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda' (instance-0000000c)
    nova-compute[20898]: DEBUG nova.compute.manager [None req-0b7dfe74-d465-4c2b-90e3-54ed26ea4244 demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Instance disappeared during snapshot {{(pid=20898) _snapshot_instance /opt/stack/nova/nova/compute/manager.py:3874}}

  Compare with logs from snapshot of a running guest:

    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Skipping quiescing instance: QEMU guest agent is not enabled.
    nova-compute[20898]: DEBUG nova.privsep.utils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Path '/opt/stack/data/nova/instances' supports direct I/O {{(pid=20898) supports_direct_io /opt/stack/nova/nova/privsep/utils.py:64}}
    nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] Running cmd (subprocess): qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581 {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:371}}
    nova-compute[20898]: DEBUG oslo_concurrency.processutils [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] CMD "qemu-img convert -t none -O qcow2 -f qcow2 /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581.delta /opt/stack/data/nova/instances/snapshots/tmpal0gmbcx/d64bd94655da448495d69b274ca14581" returned: 0 in 0.403s {{(pid=20898) execute /usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py:408}}
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot extracted, beginning image upload
    nova-compute[20898]: INFO nova.virt.libvirt.driver [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Snapshot image upload complete
    nova-compute[20898]: INFO nova.compute.manager [None req-79e59ed6-4558-42be-a016-09ff2c4d60cb demo admin] [instance: aa82c7a9-dbc1-4c7b-b3f1-8dc6b83e1bda] Took 2.44 seconds to snapshot the instance on the hypervisor.

  We see the same issue if the issues is suspended ('openstack server
  suspend'). There are no issues if the instance is paused, however
  ('openstack server pause') but

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1888414/+subscriptions