yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Public bug reported:

We see this warning from the compute manager in CI runs:

https://github.com/openstack/nova/blob/7b96206699ac28f807676bd08c6dee7a89bcb77c/nova/compute/manager.py#L3340

http://logs.openstack.org/20/541420/2/check/tempest-
full/a416830/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Mar_22_16_45_28_061143

Mar 22 16:45:28.061143 ubuntu-xenial-ovh-bhs1-0003133028 nova-
compute[13877]: WARNING nova.compute.manager [None req-ee41217e-
bf1a-4622-aed5-f872fc772d5f tempest-ImagesTestJSON-641968735 tempest-
ImagesTestJSON-641968735] [instance:
6a3e1cb2-63ff-4514-aa32-c5c4c73f84d8] trying to snapshot a non-running
instance: (state: 3 expected: 1)

state=3 is PAUSED and 1=RUNNING.

And this:

Mar 22 16:45:51.800064 ubuntu-xenial-ovh-bhs1-0003133028 nova-
compute[13877]: WARNING nova.compute.manager [None req-d9c5694c-899c-
4a57-8318-043df137d564 tempest-ImagesTestJSON-641968735 tempest-
ImagesTestJSON-641968735] [instance: 3b7589a5-ce3b-4f82-a43a-
a48497de9382] trying to snapshot a non-running instance: (state: 4
expected: 1)

state=4 is SHUTDOWN.

Maybe this is related to bug 1741667 which for older versions of
libvirt, trying to snapshot a PAUSED instance would hang:

https://review.openstack.org/#/c/532214/

If you look at that patch, it's specifically about doing *live*
snapshots with the libvirt driver on a SHUTDOWN or PAUSED instance. Live
snapshot is controlled in the libvirt driver via a config option:

[workarounds]/disable_libvirt_livesnapshot

That now defaults to False so we always attempt a live snapshot with the
libvirt driver, at least in CI runs.

Given the guest state of the instance during the snapshot is only a
concern for the underlying virt driver and if it's doing a live snapshot
or not, (which should probably be a capability trait on the compute node
via the driver btw), and the API allows users to snapshot paused and
stopped instances:

https://github.com/openstack/nova/blob/7b96206699ac28f807676bd08c6dee7a89bcb77c/nova/compute/api.py#L2717

We should either downgrade the warning to DEBUG level or remove it
completely from the compute manager since it's really virt-driver
specific.

** Affects: nova
     Importance: Low
         Status: Triaged


** Tags: compute serviceability

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1758354

Title:
  Excessive warnings about snapshotting a paused instance

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  We see this warning from the compute manager in CI runs:

  https://github.com/openstack/nova/blob/7b96206699ac28f807676bd08c6dee7a89bcb77c/nova/compute/manager.py#L3340

  http://logs.openstack.org/20/541420/2/check/tempest-
  full/a416830/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Mar_22_16_45_28_061143

  Mar 22 16:45:28.061143 ubuntu-xenial-ovh-bhs1-0003133028 nova-
  compute[13877]: WARNING nova.compute.manager [None req-ee41217e-
  bf1a-4622-aed5-f872fc772d5f tempest-ImagesTestJSON-641968735 tempest-
  ImagesTestJSON-641968735] [instance:
  6a3e1cb2-63ff-4514-aa32-c5c4c73f84d8] trying to snapshot a non-running
  instance: (state: 3 expected: 1)

  state=3 is PAUSED and 1=RUNNING.

  And this:

  Mar 22 16:45:51.800064 ubuntu-xenial-ovh-bhs1-0003133028 nova-
  compute[13877]: WARNING nova.compute.manager [None req-d9c5694c-899c-
  4a57-8318-043df137d564 tempest-ImagesTestJSON-641968735 tempest-
  ImagesTestJSON-641968735] [instance: 3b7589a5-ce3b-4f82-a43a-
  a48497de9382] trying to snapshot a non-running instance: (state: 4
  expected: 1)

  state=4 is SHUTDOWN.

  Maybe this is related to bug 1741667 which for older versions of
  libvirt, trying to snapshot a PAUSED instance would hang:

  https://review.openstack.org/#/c/532214/

  If you look at that patch, it's specifically about doing *live*
  snapshots with the libvirt driver on a SHUTDOWN or PAUSED instance.
  Live snapshot is controlled in the libvirt driver via a config option:

  [workarounds]/disable_libvirt_livesnapshot

  That now defaults to False so we always attempt a live snapshot with
  the libvirt driver, at least in CI runs.

  Given the guest state of the instance during the snapshot is only a
  concern for the underlying virt driver and if it's doing a live
  snapshot or not, (which should probably be a capability trait on the
  compute node via the driver btw), and the API allows users to snapshot
  paused and stopped instances:

  https://github.com/openstack/nova/blob/7b96206699ac28f807676bd08c6dee7a89bcb77c/nova/compute/api.py#L2717

  We should either downgrade the warning to DEBUG level or remove it
  completely from the compute manager since it's really virt-driver
  specific.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1758354/+subscriptions