← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #58310

[Bug 1638582] [NEW] network failure during post live-migrate fails to update hypervisor host

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
If the network fails during post live-migration tasks, the host/hypervisor property of the instance isn't updated to reflect the *new* compute host.

The migrated instance appears to function properly, but issuing 'nova
show <instance_id>' still shows the origin compute node along with a
stack trace.

Per the trace below, a failure occurs inside
post_live_migration_at_destination() during a call to
network_api.migrate_instance_finish() which attempts to update the
neutron port mappings. This code exists outside the try/except/finally
block that will update the database regardless of failure. Should
network connectivity be lost, or the neutron api  doesn't answer, the
instance properties aren't saved.


Steps to reproduce
==================
$ nova live-migration 0386d6cf-edf8-42b8-998f-54103ba675fe node1
$ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | node1                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node1.example.org                                        |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000034                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | migrating                                                |
| OS-EXT-STS:vm_state                  | active                                                  


A Few minutes later, vm_state is in 'error' and a stack trace can be seen
$ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
...
...SNIP
........
 fault  {"message": "Unable to establish connection to http://192.168.1.34:9696/v2.0/ports/dd042ee1-77c0-4c52-b5ea-9b5fbdd02860.json";, "code": 500, "details": "  File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 375, in decorated_function 
            return function(self, context, *args, **kwargs)
          File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 5571, in post_live_migration_at_destination             
            migration)      
          File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1577, in migrate_instance_finish 
            migration['dest_compute'])       
          File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1886, in _update_port_binding_for_instance        
            p['id'], instance=instance)      
          File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 220, in __exit__          
            self.force_reraise()             
          File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 196, in force_reraise     
            six.reraise(self.type_, self.value, self.tb)   
          File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1882, in _update_port_binding_for_instance        
            {'port': {'binding:host_id': host}})           
          File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 97, in with_params  
            ret = self.function(instance, *args, **kwargs) 
          File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 659, in update_port 
            return self.put(self.port_path % (port), body=body)             
          File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 367, in put         
            headers=headers, params=params)  
          File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 335, in retry_request             
            headers=headers, params=params)  
          File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 286, in do_request  
            resp, replybody = self.httpclient.do_request(action, method, body=body)       
          File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\", line 306, in do_request       
            return self.request(url, method, **kwargs)     
          File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\", line 294, in request          
            resp = super(SessionClient, self).request(*args, **kwargs)      
          File \"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py\", line 98, in request          
            return self.session.request(url, method, **kwargs)              
          File \"/usr/lib/python2.7/dist-packages/positional/__init__.py\", line 94, in inner              
            return func(*args, **kwargs)     
          File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\", line 452, in request         
            resp = send(**kwargs)            
          File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\", line 496, in _send_request   
            raise exceptions.ConnectFailure(msg)           
        ", "created": "2016-10-26T22:54:10Z"}                    


Expected result
===============
'host' and 'hypervisor_hostname' should be updated to the destination compute node.


Environment
===========
Ubuntu 16.04
Openstack Mitaka
  Nova 13.1.1
  Neutron 8.1.2
    + Calico driver 1.3.0

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1638582

Title:
  network failure during post live-migrate fails to update hypervisor
  host

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  If the network fails during post live-migration tasks, the host/hypervisor property of the instance isn't updated to reflect the *new* compute host.

  The migrated instance appears to function properly, but issuing 'nova
  show <instance_id>' still shows the origin compute node along with a
  stack trace.

  Per the trace below, a failure occurs inside
  post_live_migration_at_destination() during a call to
  network_api.migrate_instance_finish() which attempts to update the
  neutron port mappings. This code exists outside the try/except/finally
  block that will update the database regardless of failure. Should
  network connectivity be lost, or the neutron api  doesn't answer, the
  instance properties aren't saved.

  
  Steps to reproduce
  ==================
  $ nova live-migration 0386d6cf-edf8-42b8-998f-54103ba675fe node1
  $ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
  +--------------------------------------+----------------------------------------------------------+
  | Property                             | Value                                                    |
  +--------------------------------------+----------------------------------------------------------+
  | OS-DCF:diskConfig                    | AUTO                                                     |
  | OS-EXT-AZ:availability_zone          | nova                                                     |
  | OS-EXT-SRV-ATTR:host                 | node1                                                    |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | node1.example.org                                        |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00000034                                        |
  | OS-EXT-STS:power_state               | 1                                                        |
  | OS-EXT-STS:task_state                | migrating                                                |
  | OS-EXT-STS:vm_state                  | active                                                  

  
  A Few minutes later, vm_state is in 'error' and a stack trace can be seen
  $ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
  ...
  ...SNIP
  ........
   fault  {"message": "Unable to establish connection to http://192.168.1.34:9696/v2.0/ports/dd042ee1-77c0-4c52-b5ea-9b5fbdd02860.json";, "code": 500, "details": "  File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 375, in decorated_function 
              return function(self, context, *args, **kwargs)
            File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 5571, in post_live_migration_at_destination             
              migration)      
            File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1577, in migrate_instance_finish 
              migration['dest_compute'])       
            File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1886, in _update_port_binding_for_instance        
              p['id'], instance=instance)      
            File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 220, in __exit__          
              self.force_reraise()             
            File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 196, in force_reraise     
              six.reraise(self.type_, self.value, self.tb)   
            File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1882, in _update_port_binding_for_instance        
              {'port': {'binding:host_id': host}})           
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 97, in with_params  
              ret = self.function(instance, *args, **kwargs) 
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 659, in update_port 
              return self.put(self.port_path % (port), body=body)             
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 367, in put         
              headers=headers, params=params)  
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 335, in retry_request             
              headers=headers, params=params)  
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 286, in do_request  
              resp, replybody = self.httpclient.do_request(action, method, body=body)       
            File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\", line 306, in do_request       
              return self.request(url, method, **kwargs)     
            File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\", line 294, in request          
              resp = super(SessionClient, self).request(*args, **kwargs)      
            File \"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py\", line 98, in request          
              return self.session.request(url, method, **kwargs)              
            File \"/usr/lib/python2.7/dist-packages/positional/__init__.py\", line 94, in inner              
              return func(*args, **kwargs)     
            File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\", line 452, in request         
              resp = send(**kwargs)            
            File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\", line 496, in _send_request   
              raise exceptions.ConnectFailure(msg)           
          ", "created": "2016-10-26T22:54:10Z"}                    

  
  Expected result
  ===============
  'host' and 'hypervisor_hostname' should be updated to the destination compute node.

  
  Environment
  ===========
  Ubuntu 16.04
  Openstack Mitaka
    Nova 13.1.1
    Neutron 8.1.2
      + Calico driver 1.3.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1638582/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next