yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1532657] [NEW] nova-compute failed to start due to exception raised in _destroy_evacuated_instances

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Felix Ma <maqi@xxxxxxxxxxxxxxxxxxxx>
Date: Mon, 11 Jan 2016 03:40:58 -0000
Reply-to: Bug 1532657 <1532657@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

nova-compute can't start up because exceptions are raised in
_destroy_evacuated_instances.

Here is the log:
----------
Jan  8 16:26:04 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
Jan  8 16:26:04 compute-4 nova-compute: return f(*args, **kwargs)
Jan  8 16:26:04 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 666, in disconnect_volume
Jan  8 16:26:04 compute-4 nova-compute: raise exception.VolumeDeviceNotFound(device=host_devices)
Jan  8 16:26:04 compute-4 nova-compute: VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-10.254.4.28:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1', u'/dev/disk/by-path/ip-10.254.4.25:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1'].
Jan  8 16:26:04 compute-4 nova-compute: Removing descriptor: 5
Jan  8 16:26:05 compute-4 systemd: openstack-nova-compute.service holdoff time over, scheduling restart.
Jan  8 16:26:05 compute-4 systemd: Starting OpenStack Nova Compute Server...
Jan  8 16:26:07 compute-4 systemd: Started OpenStack Nova Compute Server.
Jan  8 16:26:08 compute-4 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=120 -- --if-exists del-port br-int qvo21952e65-be
Jan  8 16:26:08 compute-4 nova-compute: Traceback (most recent call last):
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 115, in wait
Jan  8 16:26:08 compute-4 nova-compute: listener.cb(fileno)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Jan  8 16:26:08 compute-4 nova-compute: result = function(*args, **kwargs)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 502, in run_service
Jan  8 16:26:08 compute-4 nova-compute: service.start()
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
Jan  8 16:26:08 compute-4 nova-compute: self.manager.init_host()
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1296, in init_host
Jan  8 16:26:08 compute-4 nova-compute: self._destroy_evacuated_instances(context)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 866, in _destroy_evacuated_instances
Jan  8 16:26:08 compute-4 nova-compute: bdi, destroy_disks)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 790, in destroy
Jan  8 16:26:08 compute-4 nova-compute: destroy_disks, migrate_data)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 924, in cleanup
Jan  8 16:26:08 compute-4 nova-compute: instance=instance)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
Jan  8 16:26:08 compute-4 nova-compute: six.reraise(self.type_, self.value, self.tb)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 913, in cleanup
Jan  8 16:26:08 compute-4 nova-compute: self._disconnect_volume(connection_info, disk_dev)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1098, in _disconnect_volume
Jan  8 16:26:08 compute-4 nova-compute: return driver.disconnect_volume(connection_info, disk_dev)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
Jan  8 16:26:08 compute-4 nova-compute: return f(*args, **kwargs)
Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 666, in disconnect_volume
Jan  8 16:26:08 compute-4 nova-compute: raise exception.VolumeDeviceNotFound(device=host_devices)
Jan  8 16:26:08 compute-4 nova-compute: VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-10.254.4.28:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1', u'/dev/disk/by-path/ip-10.254.4.25:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1'].
Jan  8 16:26:08 compute-4 nova-compute: Removing descriptor: 5
Jan  8 16:26:09 compute-4 systemd: openstack-nova-compute.service holdoff time over, scheduling restart.
Jan  8 16:26:09 compute-4 systemd: Starting OpenStack Nova Compute Server..
----------

** Affects: nova
     Importance: Undecided
     Assignee: Felix Ma (felix23ma)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => Felix Ma (felix23ma)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1532657

Title:
  nova-compute failed to start due to exception raised in
  _destroy_evacuated_instances

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  nova-compute can't start up because exceptions are raised in
  _destroy_evacuated_instances.

  Here is the log:
  ----------
  Jan  8 16:26:04 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
  Jan  8 16:26:04 compute-4 nova-compute: return f(*args, **kwargs)
  Jan  8 16:26:04 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 666, in disconnect_volume
  Jan  8 16:26:04 compute-4 nova-compute: raise exception.VolumeDeviceNotFound(device=host_devices)
  Jan  8 16:26:04 compute-4 nova-compute: VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-10.254.4.28:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1', u'/dev/disk/by-path/ip-10.254.4.25:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1'].
  Jan  8 16:26:04 compute-4 nova-compute: Removing descriptor: 5
  Jan  8 16:26:05 compute-4 systemd: openstack-nova-compute.service holdoff time over, scheduling restart.
  Jan  8 16:26:05 compute-4 systemd: Starting OpenStack Nova Compute Server...
  Jan  8 16:26:07 compute-4 systemd: Started OpenStack Nova Compute Server.
  Jan  8 16:26:08 compute-4 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=120 -- --if-exists del-port br-int qvo21952e65-be
  Jan  8 16:26:08 compute-4 nova-compute: Traceback (most recent call last):
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 115, in wait
  Jan  8 16:26:08 compute-4 nova-compute: listener.cb(fileno)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
  Jan  8 16:26:08 compute-4 nova-compute: result = function(*args, **kwargs)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 502, in run_service
  Jan  8 16:26:08 compute-4 nova-compute: service.start()
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
  Jan  8 16:26:08 compute-4 nova-compute: self.manager.init_host()
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1296, in init_host
  Jan  8 16:26:08 compute-4 nova-compute: self._destroy_evacuated_instances(context)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 866, in _destroy_evacuated_instances
  Jan  8 16:26:08 compute-4 nova-compute: bdi, destroy_disks)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 790, in destroy
  Jan  8 16:26:08 compute-4 nova-compute: destroy_disks, migrate_data)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 924, in cleanup
  Jan  8 16:26:08 compute-4 nova-compute: instance=instance)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  Jan  8 16:26:08 compute-4 nova-compute: six.reraise(self.type_, self.value, self.tb)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 913, in cleanup
  Jan  8 16:26:08 compute-4 nova-compute: self._disconnect_volume(connection_info, disk_dev)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1098, in _disconnect_volume
  Jan  8 16:26:08 compute-4 nova-compute: return driver.disconnect_volume(connection_info, disk_dev)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
  Jan  8 16:26:08 compute-4 nova-compute: return f(*args, **kwargs)
  Jan  8 16:26:08 compute-4 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 666, in disconnect_volume
  Jan  8 16:26:08 compute-4 nova-compute: raise exception.VolumeDeviceNotFound(device=host_devices)
  Jan  8 16:26:08 compute-4 nova-compute: VolumeDeviceNotFound: Volume device not found at [u'/dev/disk/by-path/ip-10.254.4.28:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1', u'/dev/disk/by-path/ip-10.254.4.25:3260-iscsi-iqn.2010-10.org.openstack:volume-e6796d7c-05db-44e6-818f-52f09bfef60f-lun-1'].
  Jan  8 16:26:08 compute-4 nova-compute: Removing descriptor: 5
  Jan  8 16:26:09 compute-4 systemd: openstack-nova-compute.service holdoff time over, scheduling restart.
  Jan  8 16:26:09 compute-4 systemd: Starting OpenStack Nova Compute Server..
  ----------

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1532657/+subscriptions
Follow ups

[Bug 1532657] Re: nova-compute failed to start due to exception raised in _destroy_evacuated_instances
From: OpenStack Infra, 2016-03-02