yahoo-eng-team team mailing list archive

[Artom Lifshitz <notartom@xxxxxxxxx>]

*** This bug is a duplicate of bug 1535918 ***
    https://bugs.launchpad.net/bugs/1535918

I spent some time adding LOG.debug statements all over the place while
trying to reproduce this bug, and then read bug 1535918. They're the
same bug.

To summarise:

a. When spawning a VM, the libvirt driver waits for events from Neutron [1]
b. What events it waits for is decided by _get_neutron_events [2]
c. _get_neutron_events will only return a vif-plugged event if the vif isn't active [3]
d. The network_info that _get_neutron_events gets passed in [2] ultimately comes from [4]
e. If the instance's info cache hasn't been refreshed recently in step d, the vif will be active,
   and _get_neutron_events will return an empty list in c
f. When it comes time to actually wait for events in [5], no waiting is done because the events list
   is empty.
g. The instance boots up just fine even if the vif-plugged event never makes its way to the
   destination compute. Instead, it's received by the source compute service whenever it comes back
   online.

So really, evacuation was always broken, but we got away with it just
enough times for nobody to notice because we sometimes got "lucky" with
a stale cache (which is actually a bug).

[1] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4824
[2] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4817
[3] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4787
[4] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2810
[5] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L475

Snippets of logs follow, but with some LOG.debug statements that I put
in, so not everything will appear the same in a vanilla devstack:

_get_neutron_events:

2016-10-08 11:31:01.824 DEBUG nova.virt.libvirt.driver [req-6a93955f-
cc1e-4918-af95-53a7fa1d81fe admin admin] [{"profile": {},
"ovs_interfaceid": "aaac55e8-92c8-4280-a871-fab899661a30",
"preserve_on_delete": false, "network": {"bridge": "br-int", "subnets":
[{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips":
[], "address": "10.0.0.10"}], "version": 4, "meta": {"dhcp_server":
"10.0.0.2"}, "dns": [], "routes": [], "cidr": "10.0.0.0/24", "gateway":
{"meta": {}, "version": 4, "type": "gateway", "address": "10.0.0.1"}},
{"ips": [{"meta": {}, "version": 6, "type": "fixed", "floating_ips": [],
"address": "2001:db8:8000:0:f816:3eff:fe0b:e6c2"}], "version": 6,
"meta": {"dhcp_server": "2001:db8:8000:0:f816:3eff:fec1:b2e"}, "dns":
[], "routes": [], "cidr": "2001:db8:8000::/64", "gateway": {"meta": {},
"version": 6, "type": "gateway", "address": "2001:db8:8000::1"}}],
"meta": {"injected": false, "tenant_id":
"c145e829d46745bf822fac809223b61e", "mtu": 1450}, "id":
"296e8f10-41f9-4b81-ba40-272aa8603d3b", "label": "private"}, "devname":
"tapaaac55e8-92", "vnic_type": "normal", "qbh_params": null, "meta": {},
"details": {"port_filter": true, "ovs_hybrid_plug": true}, "address":
"fa:16:3e:0b:e6:c2", "active": true, "type": "ovs", "id":
"aaac55e8-92c8-4280-a871-fab899661a30", "qbg_params": null}] from
(pid=17819) _get_neutron_events
/opt/stack/nova/nova/virt/libvirt/driver.py:4786

2016-10-08 11:31:01.824 DEBUG nova.virt.libvirt.driver [req-6a93955f-
cc1e-4918-af95-53a7fa1d81fe admin admin] Returning events=[] from
(pid=17819) _get_neutron_events
/opt/stack/nova/nova/virt/libvirt/driver.py:4790

wait_for_instance_event:

2016-10-08 11:31:03.342 DEBUG nova.compute.manager [req-6a93955f-
cc1e-4918-af95-53a7fa1d81fe admin admin] Iterating over events={} from
(pid=17819) wait_for_instance_event
/opt/stack/nova/nova/compute/manager.py:475

2016-10-08 11:31:03.343 DEBUG nova.compute.manager [req-6a93955f-
cc1e-4918-af95-53a7fa1d81fe admin admin] Done iterating from (pid=17819)
wait_for_instance_event /opt/stack/nova/nova/compute/manager.py:483

** This bug has been marked a duplicate of bug 1535918
   instance.host not updated on evacuation

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1624052

Title:
  Evacuation fails with VirtualInterfaceCreateException

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description:

  With Neutron, evacuating an instance results in a
  VirtualInterfaceCreateException and the evacuation fails.

  Steps to reproduce:

  1. Boot a VM with a Neutron NIC.
  2. Cause the underlying host to be down, for example by stopping the compute service.
  3. Evacuate the VM.
  4. The evacuation will fail because the destination compute host waits for a network-vif-plugged event from Neutron, which it never received because Nova sends the event to the source compute.

  Environment: Neutron, KVM

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1624052/+subscriptions