yahoo-eng-team team mailing list archive

[OpenStack Infra <1709319@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/491822
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=376a902cabd548f1bcc7f7f7f8f98bd2dfa87c89
Submitter: Jenkins
Branch:    master

commit 376a902cabd548f1bcc7f7f7f8f98bd2dfa87c89
Author: Vladyslav Drok <vdrok@xxxxxxxxxxxx>
Date:   Tue Aug 8 17:18:37 2017 +0300

    Add format_dom for PCI device addresses
    
    In case of having a PCI device, its address can not be output
    properly in the instance XML because of the missing format_dom
    method. This change adds this method.
    
    Closes-Bug: #1709319
    Change-Id: I1a8023ee6e8c85eed1c7c55a21f996371a0dd80a


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1709319

Title:
  LibvirtConfigGuestDeviceAddressPCI missing format_dom method

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  In my case, we had a chain of patches from
  https://review.openstack.org/#/q/topic:bug/1686116 backported to ocata
  downstream. Then, when detaching a ceph volume from a node, the
  following happens:

  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] Traceback (most recent call last):
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4835, in _driver_detach_volume
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] encryption=encryption)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1393, in detach_volume
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] live=live)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 413, in detach_device_with_retry
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] _try_detach_device(conf, persistent, live)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 407, in _try_detach_device
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] raise exception.DeviceNotFound(device=device)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in _exit_
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self.force_reraise()
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] six.reraise(self.type_, self.value, self.tb)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 392, in _try_detach_device
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self.detach_device(conf, persistent=persistent, live=live)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 449, in detach_device
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self._domain.detachDeviceFlags(device_xml, flags=flags)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] result = proxy_call(self._autowrap, f, *args, **kwargs)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv = execute(f, *args, **kwargs)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] six.reraise(c, e, tb)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv = meth(*args, **kwargs)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1190, in detachDeviceFlags
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of domain, bus, or slot must be > 0
  nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]

  Here is the instance xml bit for the volume:

      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <auth username='volumes'>
          <secret type='ceph' uuid='ce6d1549-4d63-476b-afb6-88f0b196414f'/>
        </auth>
        <source protocol='rbd' name='volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc'>
          <host name='192.168.90.22' port='6789'/>
        </source>
        <backingStore/>
        <target dev='vdb' bus='virtio'/>
        <serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
        <alias name='virtio-disk1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </disk>

  After inserting some debug logs, the following xml gets passed to the
  libvirt's virDomainDetachDeviceFlags:

      <disk type="network" device="disk">
        <driver name="qemu" type="raw" cache="none"/>
        <source protocol="rbd" name="volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc">
          <host name="192.168.90.22" port="6789"/>
        </source>
        <target bus="virtio" dev="vdb"/>
        <serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
        <address type="pci"/>
      </disk>

  After introducing a proper format_dom method for
  LibvirtConfigGuestDeviceAddressPCI, things seem to be back to normal.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1709319/+subscriptions