yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1245719] Re: RBD backed instance can't shutdown and restart

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Wed, 04 Dec 2013 10:37:02 -0000
Reply-to: Bug 1245719 <1245719@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1245719

Title:
  RBD backed instance can't shutdown and restart

Status in OpenStack Dashboard (Horizon):
  New
Status in OpenStack Compute (Nova):
  Fix Released
Status in Ubuntu:
  Confirmed

Bug description:
  Version: Havana w/ Ubuntu Repos. with Ceph for RBD.

  
  When creating Launching a instance with "Boot from image (Creates a new volume)" this creates the instance fine and all is well however if you shutdown the instance I can't turn it back on again.

  
  I get the following error in the nova-compute.log when trying to power on an shutdown instance.

  #######################################################################################
  2013-10-29 00:48:33.859 2746 WARNING nova.compute.utils [req-89bbd72f-2280-4fac-802a-1211ec774980 27106b78ceac4e389558566857a7875f 464099f86eb94d049ed1f7b0f0144275] [instance: cc370f6d-4be0-4cd3-9f20-bf86f5ad7c09] Can't access image $
  2013-10-29 00:48:34.040 2746 WARNING nova.virt.libvirt.vif [req-89bbd72f-2280-4fac-802a-1211ec774980 27106b78ceac4e389558566857a7875f 464099f86eb94d049ed1f7b0f0144275] Deprecated: The LibvirtHybridOVSBridgeDriver VIF driver is now de$
  2013-10-29 00:48:34.578 2746 ERROR nova.openstack.common.rpc.amqp [req-89bbd72f-2280-4fac-802a-1211ec774980 27106b78ceac4e389558566857a7875f 464099f86eb94d049ed1f7b0f0144275] Exception during message handling
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     **args)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     payload)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     pass
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1832, in start_instance
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     self._power_on(context, instance)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1819, in _power_on
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1948, in power_on
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     self._hard_reboot(context, instance, network_info, block_device_info)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1903, in _hard_reboot
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4318, in get_instance_disk_info
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     dk_size = int(os.path.getsize(path))
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp     return os.stat(filename).st_size
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/cc370f6d-4be0-4cd3-9f20-bf86f5ad7c09/disk'
  2013-10-29 00:48:34.578 2746 TRACE nova.openstack.common.rpc.amqp
  #######################################################################################

  
  On Closer inspection It seems the libvirt.xml file for the instance gets all screwed up.

  This is the libvirt.xml file for this instance before shutdown.

  #######################################################################################
  <domain type="kvm">
    <uuid>dc9749cd-0002-41c9-ac55-5f691637146a</uuid>
    <name>instance-00000004</name>
    <memory>524288</memory>
    <vcpu>1</vcpu>
    <sysinfo type="smbios">
      <system>
        <entry name="manufacturer">OpenStack Foundation</entry>
        <entry name="product">OpenStack Nova</entry>
        <entry name="version">2013.2</entry>
        <entry name="serial">4c4c4544-0053-3210-8032-b6c04f5a5931</entry>
        <entry name="uuid">dc9749cd-0002-41c9-ac55-5f691637146a</entry>
      </system>
    </sysinfo>
    <os>
      <type>hvm</type>
      <boot dev="hd"/>
      <smbios mode="sysinfo"/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <clock offset="utc">
      <timer name="pit" tickpolicy="delay"/>
      <timer name="rtc" tickpolicy="catchup"/>
    </clock>
    <cpu mode="host-model" match="exact"/>
    <devices>
      <disk type="network" device="disk">
        <driver name="qemu" type="raw" cache="none"/>
        <source protocol="rbd" name="volumes/volume-5abadeb8-49e9-4628-a54d-d742d6f2e012">
          <host name="10.100.96.10" port="6789"/>
          <host name="10.100.96.11" port="6789"/>
          <host name="10.100.96.12" port="6789"/>
        </source>
        <auth username="volumes">
          <secret type="ceph" uuid="13a673af-ff80-3036-8310-4c72f566673d"/>
        </auth>
        <target bus="virtio" dev="vda"/>
        <serial>5abadeb8-49e9-4628-a54d-d742d6f2e012</serial>
      </disk>
      <interface type="bridge">
        <mac address="fa:16:3e:45:17:d4"/>
        <model type="virtio"/>
        <source bridge="qbr125aa659-01"/>
        <target dev="tap125aa659-01"/>
      </interface>
      <serial type="file">
        <source path="/var/lib/nova/instances/dc9749cd-0002-41c9-ac55-5f691637146a/console.log"/>
      </serial>
      <serial type="pty"/>
      <input type="tablet" bus="usb"/>
      <graphics type="vnc" autoport="yes" keymap="en-us" listen="10.100.32.10"/>
    </devices>
  </domain>
  #######################################################################################

  All looks fine here i can see the ceph mons etc.

  Next i'll shutdown the instance and as expected the file stays the
  same now this is the file if i try to power the instance back on
  again.

  you'll notice all of the details regarding the disk have changed and
  now it thinks its a qcow2 disk located on the local hard drive... how
  does this happen?

  #######################################################################################

  <domain type="kvm">
    <uuid>dc9749cd-0002-41c9-ac55-5f691637146a</uuid>
    <name>instance-00000004</name>
    <memory>524288</memory>
    <vcpu>1</vcpu>
    <sysinfo type="smbios">
      <system>
        <entry name="manufacturer">OpenStack Foundation</entry>
        <entry name="product">OpenStack Nova</entry>
        <entry name="version">2013.2</entry>
        <entry name="serial">4c4c4544-0053-3210-8032-b6c04f5a5931</entry>
        <entry name="uuid">dc9749cd-0002-41c9-ac55-5f691637146a</entry>
      </system>
    </sysinfo>
    <os>
      <type>hvm</type>
      <boot dev="hd"/>
      <smbios mode="sysinfo"/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <clock offset="utc">
      <timer name="pit" tickpolicy="delay"/>
      <timer name="rtc" tickpolicy="catchup"/>
    </clock>
    <cpu mode="host-model" match="exact"/>
    <devices>
      <disk type="file" device="disk">
        <driver name="qemu" type="qcow2" cache="none"/>
        <source file="/var/lib/nova/instances/dc9749cd-0002-41c9-ac55-5f691637146a/disk"/>
        <target bus="virtio" dev="vda"/>
      </disk>
      <interface type="bridge">
        <mac address="fa:16:3e:45:17:d4"/>
        <model type="virtio"/>
        <source bridge="qbr125aa659-01"/>
        <target dev="tap125aa659-01"/>
      </interface>
      <serial type="file">
        <source path="/var/lib/nova/instances/dc9749cd-0002-41c9-ac55-5f691637146a/console.log"/>
      </serial>
      <serial type="pty"/>
      <input type="tablet" bus="usb"/>
      <graphics type="vnc" autoport="yes" keymap="en-us" listen="10.100.32.10"/>
    </devices>
  </domain>

  #######################################################################################

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1245719/+subscriptions