yahoo-eng-team team mailing list archive

[sean mooney <2073254@xxxxxxxxxxxxxxxxxx>]

the title is incorrrect.

nova does not wait for  vif-plugged at plugtime because ovn does not
replicable send those so we cant safely do that.

** Changed in: nova
       Status: In Progress => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2073254

Title:
  nova doesn't wait for vif-plugged in ovn usecase

Status in OpenStack Compute (nova):
  Opinion

Bug description:
  Description
  ===========
  At environments using OVN as network backend, nova seems to not wait for network-vif-plugged event.
  This leads to issues in bigger environments, where ovn northd needs some time to sync or some ovn component is down. Nova than starts the libvirt migration, but network setup is not done.

  Neutron is using the multi-chassis approach of ovn to send packages to
  both, source and destination host. Nova should wait for Neutron to
  have this configured.

  In nova code I found that get_live_migration_plug_time_events()[1] calls (via has_live_migration_plug_time_event()) is_hybrid_plug_enabled()[2], where it checks if VIF_DETAILS_OVS_HYBRID_PLUG is set in VIF.details.
  But that's not the case for ports in OVN setup. Instead we should check, if the ports driver is ovn.
  Also while debugging at my cluster with OVN setup, I saw nova-compute passing that function.

  Steps to reproduce
  ==================
  1. Run neutron with ovn setup and create a VM that you can ping (via FIP or other VM in same private network)
  2. Stop northd
  3. Start live-migration
  4. Wait till live-migration is done - VM is not reachable anymore

  or patch neutron to not send any network-vif-plugged events and do the
  same steps (besides 2.)

  Expected result
  ===============
  nova waits for network-vif-plugged event from neutron

  Actual result
  =============
  libvirt migration is directly started

  Environment
  ===========
  nova-compute --version: 26.3.0
  neutron-server 21.2.1 zed / unmaintained/zed
  ml2 plugin: ovn
  at neutron: ovsdb-client (Open vSwitch) 3.3.0
  Nova zed / unmaintained/zed
  nova.conf: live_migration_wait_for_vif_plug=true ([3])
  Hypervisor OS: Ubuntu 22.04 with newer kernel (but that shouldn't be relevant here)
  Libvirt + KVM

  Proposed Change
  ===============
  As fix I'm testing an additional function at [1] that checks, if vif/port driver is ovn, than get_live_migration_plug_time_events should return True.

  Related Bugs
  ============
  neutron: https://bugs.launchpad.net/neutron/+bug/2069718
  Comments from neutron patch[4] say, this should be fixed at nova (as setting OVS_HYBRID_PLUG for ovn is wrong)

  Follow up will be to change neutron to send the event at the correct
  time, but first nova needs to wait for it.

  
  [1] https://opendev.org/openstack/nova/src/branch/master/nova/network/model.py#L563
  [2] https://opendev.org/openstack/nova/src/branch/master/nova/network/model.py#L499
  [3] https://docs.openstack.org/nova/latest/configuration/config.html#compute.live_migration_wait_for_vif_plug
  [4] https://review.opendev.org/c/openstack/neutron/+/923962

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2073254/+subscriptions