yahoo-eng-team team mailing list archive

[Shane Peters <1638582@xxxxxxxxxxxxxxxxxx>]

Marking this bug as invalid in favor of bugs #1685340 and #1662626 with
respective commits https://review.openstack.org/#/c/458958/ and
https://review.openstack.org/#/c/457023/ .

The most we can hope for in a situation such as this is that the true
state is reflected to the user/operator which the referenced
bugs/commits achieve.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1638582

Title:
  network failure during post live-migrate fails to update hypervisor
  host

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========
  If the network fails during post live-migration tasks, the host/hypervisor property of the instance isn't updated to reflect the *new* compute host.

  The migrated instance appears to function properly, but issuing 'nova
  show <instance_id>' still shows the origin compute node along with a
  stack trace.

  Per the trace below, a failure occurs inside
  post_live_migration_at_destination() during a call to
  network_api.migrate_instance_finish() which attempts to update the
  neutron port mappings. This code exists outside the try/except/finally
  block that will update the database regardless of failure. Should
  network connectivity be lost, or the neutron api  doesn't answer, the
  instance properties aren't saved.

  
  Steps to reproduce
  ==================
  $ nova live-migration 0386d6cf-edf8-42b8-998f-54103ba675fe node1
  $ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
  +--------------------------------------+----------------------------------------------------------+
  | Property                             | Value                                                    |
  +--------------------------------------+----------------------------------------------------------+
  | OS-DCF:diskConfig                    | AUTO                                                     |
  | OS-EXT-AZ:availability_zone          | nova                                                     |
  | OS-EXT-SRV-ATTR:host                 | node1                                                    |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | node1.example.org                                        |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00000034                                        |
  | OS-EXT-STS:power_state               | 1                                                        |
  | OS-EXT-STS:task_state                | migrating                                                |
  | OS-EXT-STS:vm_state                  | active                                                  

  
  A Few minutes later, vm_state is in 'error' and a stack trace can be seen
  $ nova show 0386d6cf-edf8-42b8-998f-54103ba675fe
  ...
  ...SNIP
  ........
   fault  {"message": "Unable to establish connection to http://192.168.1.34:9696/v2.0/ports/dd042ee1-77c0-4c52-b5ea-9b5fbdd02860.json";, "code": 500, "details": "  File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 375, in decorated_function 
              return function(self, context, *args, **kwargs)
            File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 5571, in post_live_migration_at_destination             
              migration)      
            File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1577, in migrate_instance_finish 
              migration['dest_compute'])       
            File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1886, in _update_port_binding_for_instance        
              p['id'], instance=instance)      
            File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 220, in __exit__          
              self.force_reraise()             
            File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 196, in force_reraise     
              six.reraise(self.type_, self.value, self.tb)   
            File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 1882, in _update_port_binding_for_instance        
              {'port': {'binding:host_id': host}})           
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 97, in with_params  
              ret = self.function(instance, *args, **kwargs) 
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 659, in update_port 
              return self.put(self.port_path % (port), body=body)             
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 367, in put         
              headers=headers, params=params)  
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 335, in retry_request             
              headers=headers, params=params)  
            File \"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py\", line 286, in do_request  
              resp, replybody = self.httpclient.do_request(action, method, body=body)       
            File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\", line 306, in do_request       
              return self.request(url, method, **kwargs)     
            File \"/usr/lib/python2.7/dist-packages/neutronclient/client.py\", line 294, in request          
              resp = super(SessionClient, self).request(*args, **kwargs)      
            File \"/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py\", line 98, in request          
              return self.session.request(url, method, **kwargs)              
            File \"/usr/lib/python2.7/dist-packages/positional/__init__.py\", line 94, in inner              
              return func(*args, **kwargs)     
            File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\", line 452, in request         
              resp = send(**kwargs)            
            File \"/usr/lib/python2.7/dist-packages/keystoneauth1/session.py\", line 496, in _send_request   
              raise exceptions.ConnectFailure(msg)           
          ", "created": "2016-10-26T22:54:10Z"}                    

  
  Expected result
  ===============
  'host' and 'hypervisor_hostname' should be updated to the destination compute node.

  
  Environment
  ===========
  Ubuntu 16.04
  Openstack Mitaka
    Nova 13.1.1
    Neutron 8.1.2
      + Calico driver 1.3.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1638582/+subscriptions