yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1711603] [NEW] Unable to Launch instance using ceph boot volume

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pranita Desai <1711603@xxxxxxxxxxxxxxxxxx>
Date: Fri, 18 Aug 2017 12:24:22 -0000
Reply-to: Bug 1711603 <1711603@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have deployed Openstack Base bundle cs:~openstack-charmers-next/bundle/openstack-base-xenial-ocata-0 which deploys ceph as storage backend.
https://jujucharms.com/u/openstack-charmers-next/openstack-base-xenial-ocata/

I am unable to launch instance using ceph volume as a boot volume.
Launching instance from glance image is also failing. Getting below
errors in nova-compute.log:


2017-08-18 12:24:02.691 1708784 ERROR nova.virt.libvirt.driver [req-72b3771b-209c-415f-b14c-4e885a72f0b8 ef1fea1a88c744ecb96d08bb56fdc412 5a41e5c16d2c4299b23d26b87ff994aa - - -] [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] Failed to start libvirt guest
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [req-72b3771b-209c-415f-b14c-4e885a72f0b8 ef1fea1a88c744ecb96d08bb56fdc412 5a41e5c16d2c4299b23d26b87ff994aa - - -] [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] Instance failed to spawn
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] Traceback (most recent call last):
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2124, in _build_resources
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     yield resources
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     block_device_info=block_device_info)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2698, in spawn
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     destroy_disks_on_failure=True)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5114, in _create_domain_and_network
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     destroy_disks_on_failure)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     self.force_reraise()
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     six.reraise(self.type_, self.value, self.tb)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5086, in _create_domain_and_network
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     post_xml_callback=post_xml_callback)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5004, in _create_domain
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     guest.launch(pause=pause)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 145, in launch
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     self._encoded_xml, errors='ignore')
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     self.force_reraise()
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     six.reraise(self.type_, self.value, self.tb)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 140, in launch
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     return self._domain.createWithFlags(flags)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     rv = execute(f, *args, **kwargs)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     six.reraise(c, e, tb)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     rv = meth(*args, **kwargs)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1065, in createWithFlags
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] libvirtError: internal error: process exited while connecting to monitor: nfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-16-instance-00000011/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:cinder-ceph/volume-2c04ab2b-9e9e-46c8-8995-a33cf1008331:id=cinder-ceph:key=AQA9P5VZMy4BDxAA1U8ELcourkvQE4MgaJPjbw==:auth_supported=cephx\;none:mon_host=172.16.10.6\:6789\;172.16.10.9\:6789\;172.16.10.11\:6789,format=raw,if=none,id=drive-virtio-disk0,serial=2c04ab2b-9e9e-46c8-8995-a33cf1008331,cache=none' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:f4:e2:46,bus=pci.0,addr=0x2 -add-fd set=2,fd=31 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logap

However I am able to launch instance when volume creation step is skipped.
I tried disabling firewall on all nodes of ceph and openstack. But no luck.

Can someone help?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1711603

Title:
  Unable to Launch instance using ceph boot volume

Status in OpenStack Compute (nova):
  New

Bug description:
  I have deployed Openstack Base bundle cs:~openstack-charmers-next/bundle/openstack-base-xenial-ocata-0 which deploys ceph as storage backend.
  https://jujucharms.com/u/openstack-charmers-next/openstack-base-xenial-ocata/

  I am unable to launch instance using ceph volume as a boot volume.
  Launching instance from glance image is also failing. Getting below
  errors in nova-compute.log:

  
  2017-08-18 12:24:02.691 1708784 ERROR nova.virt.libvirt.driver [req-72b3771b-209c-415f-b14c-4e885a72f0b8 ef1fea1a88c744ecb96d08bb56fdc412 5a41e5c16d2c4299b23d26b87ff994aa - - -] [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] Failed to start libvirt guest
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [req-72b3771b-209c-415f-b14c-4e885a72f0b8 ef1fea1a88c744ecb96d08bb56fdc412 5a41e5c16d2c4299b23d26b87ff994aa - - -] [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] Instance failed to spawn
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] Traceback (most recent call last):
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2124, in _build_resources
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     yield resources
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     block_device_info=block_device_info)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2698, in spawn
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     destroy_disks_on_failure=True)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5114, in _create_domain_and_network
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     destroy_disks_on_failure)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     self.force_reraise()
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     six.reraise(self.type_, self.value, self.tb)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5086, in _create_domain_and_network
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     post_xml_callback=post_xml_callback)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5004, in _create_domain
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     guest.launch(pause=pause)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 145, in launch
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     self._encoded_xml, errors='ignore')
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     self.force_reraise()
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     six.reraise(self.type_, self.value, self.tb)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 140, in launch
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     return self._domain.createWithFlags(flags)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     rv = execute(f, *args, **kwargs)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     six.reraise(c, e, tb)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     rv = meth(*args, **kwargs)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1065, in createWithFlags
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2017-08-18 12:24:03.015 1708784 ERROR nova.compute.manager [instance: 0ff1f63d-e25f-4c75-97a5-05f02f1e6aa0] libvirtError: internal error: process exited while connecting to monitor: nfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-16-instance-00000011/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:cinder-ceph/volume-2c04ab2b-9e9e-46c8-8995-a33cf1008331:id=cinder-ceph:key=AQA9P5VZMy4BDxAA1U8ELcourkvQE4MgaJPjbw==:auth_supported=cephx\;none:mon_host=172.16.10.6\:6789\;172.16.10.9\:6789\;172.16.10.11\:6789,format=raw,if=none,id=drive-virtio-disk0,serial=2c04ab2b-9e9e-46c8-8995-a33cf1008331,cache=none' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:f4:e2:46,bus=pci.0,addr=0x2 -add-fd set=2,fd=31 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logap

  However I am able to launch instance when volume creation step is skipped.
  I tried disabling firewall on all nodes of ceph and openstack. But no luck.

  Can someone help?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1711603/+subscriptions