yahoo-eng-team team mailing list archive

[OpenStack Infra <1629371@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/380433
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e659a6e7cbb3078b47c0860465530eab05acf73e
Submitter: Jenkins
Branch:    master

commit e659a6e7cbb3078b47c0860465530eab05acf73e
Author: Matt Riedemann <mriedem@xxxxxxxxxx>
Date:   Fri Sep 30 12:16:41 2016 -0400

    libvirt: check if we can quiesce before volume-backed snapshot
    
    The NFS job in the experimental queue is stacktracing when taking
    a volume-backed snapshot because the image used in the job does
    not have the qemu guest agent in it. This is not fatal because the
    code just continues to snapshot without quiesce.
    
    Since we have code that can tell if we can even attempt a quiesce
    we should use that here before attempting the snapshot with quiesce,
    else don't even attempt it so we don't stacktrace.
    
    Note that this also adds a check such that if the image requires
    quiesce as part of a snapshot and we can't honor that, we fail.
    
    The stacktrace is intentionally left in here in case the snapshot
    with quiesce fails because we checked if we could and only get to
    that point if we should be able to quiesce but it failed, which
    is an error that should be investigated.
    
    Also, the unused quiesce parameter is removed from the
    test_volume_snapshot_create_libgfapi test while in this code.
    
    Change-Id: I685c592340dc8a3b7d32834a0114f2925dc4305c
    Closes-Bug: #1629371


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1629371

Title:
  libvirt: volume snapshot stacktraces if the qemu guest agent is not
  available

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The libvirt driver attempts to quiece the filesystem when doing a
  volume snapshot and will stacktrace if the qemu guest agent is not
  available:

  http://logs.openstack.org/86/147186/41/experimental/gate-tempest-dsvm-
  full-devstack-plugin-nfs-
  nv/fe58d89/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-30_10_36_32_039

  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [req-93930c3c-4482-4c30-97ad-4dbcd251a9ba nova service] [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08] Unable to create quiesced VM snapshot, attempting again with quiescing disabled.
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08] Traceback (most recent call last):
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1827, in _volume_snapshot_create
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     reuse_ext=True, quiesce=True)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 508, in snapshot
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     self._domain.snapshotCreateXML(conf.to_xml(), flags=flags)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     rv = execute(f, *args, **kwargs)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     six.reraise(c, e, tb)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     rv = meth(*args, **kwargs)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2592, in snapshotCreateXML
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08]     if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
  2016-09-30 10:36:32.039 10790 ERROR nova.virt.libvirt.driver [instance: 01211c7e-b3ac-4d17-80bb-30fbe8494c08] libvirtError: 

  The volume snapshot method then goes on to snapshot the guest without
  quiescing first.

  We shouldn't stacktrace an error on this, we should dump a warning in
  the logs at most.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1629371/+subscriptions