yahoo-eng-team team mailing list archive

[Ryan Farrell <1849154@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

[Description]
Occasionally when evacuating vms off of nova compute hosts for host reboots, a vms migration will be reported as complete in the migration list, but queries to the openstack api, such as 'openstack show uuid' will report the host & hypervisor-hostname unchanged. The only indication that something is wrong is that power_state will be NOSTATE.  We can see that the instance is in fact migrated and running on the new host with 'sudo virsh list --all | grep $instance_name'. 

In order to resolve this issue we perform a direct database edit such
as:

'update instances
set host="$newhost", node="$newhost.domain", progress="0"
where uuid="" and deleted="0";' 

* In one instance, the 'progress' value was stuck at 99 and I needed to
set that to 0 in the database as well.

[Expected]
Its expected that the live migration completes and that the instance in the openstack database correctly reflects the name of the new host, and its power state.

[Impact]
Instances that are found to be in power state NOSTATE are blocked from performing certain actions; instances in this state do not self recover.


[Environment]
Openstack Queens; Nova 17.0.10
libvirtd/virsh: 4.0.0
ceph: 12.2.8
neutron-openvswitch: 12.0.5

[Logs]
In this particular set of logs (sosreports from the live migration source and destination hosts); the instance that was in error had uuid 67f328d0-cb5e-416a-9af4-c6e47e68a1e0.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1849154

Title:
  Live Migrations complete but occasionally fail to update the Openstack
  Database

Status in OpenStack Compute (nova):
  New

Bug description:
  [Description]
  Occasionally when evacuating vms off of nova compute hosts for host reboots, a vms migration will be reported as complete in the migration list, but queries to the openstack api, such as 'openstack show uuid' will report the host & hypervisor-hostname unchanged. The only indication that something is wrong is that power_state will be NOSTATE.  We can see that the instance is in fact migrated and running on the new host with 'sudo virsh list --all | grep $instance_name'. 

  In order to resolve this issue we perform a direct database edit such
  as:

  'update instances
  set host="$newhost", node="$newhost.domain", progress="0"
  where uuid="" and deleted="0";' 

  * In one instance, the 'progress' value was stuck at 99 and I needed
  to set that to 0 in the database as well.

  [Expected]
  Its expected that the live migration completes and that the instance in the openstack database correctly reflects the name of the new host, and its power state.

  [Impact]
  Instances that are found to be in power state NOSTATE are blocked from performing certain actions; instances in this state do not self recover.

  
  [Environment]
  Openstack Queens; Nova 17.0.10
  libvirtd/virsh: 4.0.0
  ceph: 12.2.8
  neutron-openvswitch: 12.0.5

  [Logs]
  In this particular set of logs (sosreports from the live migration source and destination hosts); the instance that was in error had uuid 67f328d0-cb5e-416a-9af4-c6e47e68a1e0.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1849154/+subscriptions