yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1945646] [NEW] Nova fails to live migrate instance with upper-case port MAC

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dmitriy Rabotyagov <1945646@xxxxxxxxxxxxxxxxxx>
Date: Thu, 30 Sep 2021 13:05:06 -0000
Reply-to: Bug 1945646 <1945646@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Description
===========

When neutron port has MAC address defined in upper case and libvirt stores MAC in XML in lower case, migration is failed with KeyError:
```
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: 2021-09-30 10:31:38.028 3054313 ERROR nova.virt.libvirt.driver [req-911a4b70-5448-48a1-afa4-1bbd0b38737b - - - - -] [instance: 75f7
9d85-6505-486c-bc34-e78fd6350a77] Live Migration failure: '00:50:56:af:e1:73'
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: Traceback (most recent call last):
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     timer()
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     cb(*args, **kw)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/event.py", line 175, in _do_send
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     waiter.switch(result)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     result = function(*args, **kwargs)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/utils.py", line 661, in context_wrapper
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     return func(*args, **kwargs)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9196, in _live_migration_operation
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     LOG.error("Live Migration failure: %s", e, instance=instance)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     self.force_reraise()
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     six.reraise(self.type_, self.value, self.tb)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/six.py", line 703, in reraise
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     raise value
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9152, in _live_migration_operation
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     new_xml_str = libvirt_migrate.get_updated_guest_xml(
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 65, in get_updated_guest_xml
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     xml_doc = _update_vif_xml(xml_doc, migrate_data, get_vif_config)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 355, in _update_vif_xml
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     migrate_vif = migrate_vif_by_mac[mac_addr]
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: KeyError: '00:50:56:af:e1:73'
```

Environment
===========

Ubuntu 20.04
Libvirt 6.0.0-0ubuntu8.14
Nova 22.2.3.dev2 (sha 4ce01d6c49f81b6b2438549b01a89ea1b5956320)
Neutron with OpenVSwitch

** Affects: nova
     Importance: Undecided
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1945646

Title:
  Nova fails to live migrate instance with upper-case port MAC

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========

  When neutron port has MAC address defined in upper case and libvirt stores MAC in XML in lower case, migration is failed with KeyError:
  ```
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: 2021-09-30 10:31:38.028 3054313 ERROR nova.virt.libvirt.driver [req-911a4b70-5448-48a1-afa4-1bbd0b38737b - - - - -] [instance: 75f7
  9d85-6505-486c-bc34-e78fd6350a77] Live Migration failure: '00:50:56:af:e1:73'
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: Traceback (most recent call last):
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     timer()
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     cb(*args, **kw)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/event.py", line 175, in _do_send
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     waiter.switch(result)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     result = function(*args, **kwargs)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/utils.py", line 661, in context_wrapper
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     return func(*args, **kwargs)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9196, in _live_migration_operation
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     LOG.error("Live Migration failure: %s", e, instance=instance)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     self.force_reraise()
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     six.reraise(self.type_, self.value, self.tb)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/six.py", line 703, in reraise
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     raise value
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9152, in _live_migration_operation
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     new_xml_str = libvirt_migrate.get_updated_guest_xml(
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 65, in get_updated_guest_xml
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     xml_doc = _update_vif_xml(xml_doc, migrate_data, get_vif_config)
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:   File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 355, in _update_vif_xml
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:     migrate_vif = migrate_vif_by_mac[mac_addr]
  Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: KeyError: '00:50:56:af:e1:73'
  ```

  Environment
  ===========

  Ubuntu 20.04
  Libvirt 6.0.0-0ubuntu8.14
  Nova 22.2.3.dev2 (sha 4ce01d6c49f81b6b2438549b01a89ea1b5956320)
  Neutron with OpenVSwitch

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1945646/+subscriptions
Follow ups

[Bug 1945646] Re: Nova fails to live migrate instance with upper-case port MAC
From: OpenStack Infra, 2021-10-27
[Bug 1945646] Re: Nova fails to live migrate instance with upper-case port MAC
From: Rodolfo Alonso, 2021-10-08
[Bug 1945646] Re: Nova fails to live migrate instance with upper-case port MAC
From: sean mooney, 2021-10-07