yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2065790] [NEW] Nova fails to live migrate instance with dash separated MAC

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dmitriy Rabotyagov <2065790@xxxxxxxxxxxxxxxxxx>
Date: Wed, 15 May 2024 14:52:54 -0000
Reply-to: Bug 2065790 <2065790@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Description
===========

In case network port was created with manually defined MAC address and
dash ("-") was used as a separator instead of semi-column, attempt of
server migration with that port will lead to failure

Steps to reproduce
==================

1. Manually create a port
openstack port create --mac-address "FA-16-3E-7B-D7-F4" --network f522fb2d-05b2-43e9-bcca-60d06620747d invalid_mac

2. Create a VM from that port:
openstack server create --flavor 1c1gb --image cirros --port 3642e1de-bada-4fc2-a010-ddde1589c154 --boot-from-volume 20 test_mac

3. Try to migrate server:
openstack server migrate --live 9e948ba7-4137-462a-91aa-7e9a64f52a9d


Expected result
===============

Instance is migrated


Actual result
=============

Live migration fails with stack trace


Environment
===========

Nova SHA: cf2abde7ee9437bbd3735d30b9b8e1dcb04a4bda
HV: Libvirt 8.0.0 + QEMU 4.2
Storage: Ceph
Networking: OVS


Logs
====

May 15 14:03:23 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:23.478 855142 INFO nova.compute.manager [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Took 2.75 seconds for pre_live_migration on destination host cc-compute05-dx1.
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:24.803 855142 ERROR nova.virt.libvirt.driver [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Live Migration failure: 'fa:16:3e:7b:d7:f4'
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: Traceback (most recent call last):
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     timer()
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     cb(*args, **kw)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/event.py", line 175, in _do_send
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     waiter.switch(result)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     result = function(*args, **kwargs)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/utils.py", line 654, in context_wrapper
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     return func(*args, **kwargs)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10252, in _live_migration_operation
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     LOG.error("Live Migration failure: %s", e, instance=instance)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     self.force_reraise()
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     raise self.value
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10208, in _live_migration_operation
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     new_xml_str = libvirt_migrate.get_updated_guest_xml(
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 67, in get_updated_guest_xml
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     xml_doc = _update_vif_xml(xml_doc, migrate_data, get_vif_config)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 369, in _update_vif_xml
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     migrate_vif = migrate_vif_by_mac[mac_addr.lower()]
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: KeyError: 'fa:16:3e:7b:d7:f4'
May 15 14:03:25 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:25.288 855142 ERROR nova.virt.libvirt.driver [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Migration operation has aborted
May 15 14:03:25 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:25.352 855142 INFO nova.compute.manager [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Swapping old allocation on dict_keys(['4e2aaa84-0561-4525-820f-0d2c1b3c7f86']) held by migration 75c70d12-41c8-4c67-994c-679e59787690 for instance
May 15 14:03:40 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:40.929 855142 INFO nova.compute.resource_tracker [None req-017bb1d7-a8d5-4a2f-8df6-26280fa4cb89 - - - - - -] Instance 8eee2b1f-01cd-4b9e-b178-5eb9600cbb5f has allocations against this compute host but is not found in the database.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2065790

Title:
  Nova fails to live migrate instance with dash separated MAC

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  In case network port was created with manually defined MAC address and
  dash ("-") was used as a separator instead of semi-column, attempt of
  server migration with that port will lead to failure

  Steps to reproduce
  ==================

  1. Manually create a port
  openstack port create --mac-address "FA-16-3E-7B-D7-F4" --network f522fb2d-05b2-43e9-bcca-60d06620747d invalid_mac

  2. Create a VM from that port:
  openstack server create --flavor 1c1gb --image cirros --port 3642e1de-bada-4fc2-a010-ddde1589c154 --boot-from-volume 20 test_mac

  3. Try to migrate server:
  openstack server migrate --live 9e948ba7-4137-462a-91aa-7e9a64f52a9d


  Expected result
  ===============

  Instance is migrated

  
  Actual result
  =============

  Live migration fails with stack trace

  
  Environment
  ===========

  Nova SHA: cf2abde7ee9437bbd3735d30b9b8e1dcb04a4bda
  HV: Libvirt 8.0.0 + QEMU 4.2
  Storage: Ceph
  Networking: OVS

  
  Logs
  ====

  May 15 14:03:23 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:23.478 855142 INFO nova.compute.manager [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Took 2.75 seconds for pre_live_migration on destination host cc-compute05-dx1.
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:24.803 855142 ERROR nova.virt.libvirt.driver [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Live Migration failure: 'fa:16:3e:7b:d7:f4'
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: Traceback (most recent call last):
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     timer()
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     cb(*args, **kw)
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/event.py", line 175, in _do_send
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     waiter.switch(result)
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     result = function(*args, **kwargs)
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/utils.py", line 654, in context_wrapper
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     return func(*args, **kwargs)
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10252, in _live_migration_operation
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     LOG.error("Live Migration failure: %s", e, instance=instance)
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     self.force_reraise()
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     raise self.value
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10208, in _live_migration_operation
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     new_xml_str = libvirt_migrate.get_updated_guest_xml(
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 67, in get_updated_guest_xml
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     xml_doc = _update_vif_xml(xml_doc, migrate_data, get_vif_config)
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:   File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 369, in _update_vif_xml
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]:     migrate_vif = migrate_vif_by_mac[mac_addr.lower()]
  May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: KeyError: 'fa:16:3e:7b:d7:f4'
  May 15 14:03:25 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:25.288 855142 ERROR nova.virt.libvirt.driver [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Migration operation has aborted
  May 15 14:03:25 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:25.352 855142 INFO nova.compute.manager [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Swapping old allocation on dict_keys(['4e2aaa84-0561-4525-820f-0d2c1b3c7f86']) held by migration 75c70d12-41c8-4c67-994c-679e59787690 for instance
  May 15 14:03:40 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:40.929 855142 INFO nova.compute.resource_tracker [None req-017bb1d7-a8d5-4a2f-8df6-26280fa4cb89 - - - - - -] Instance 8eee2b1f-01cd-4b9e-b178-5eb9600cbb5f has allocations against this compute host but is not found in the database.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2065790/+subscriptions
Follow ups

[Bug 2065790] Re: Nova fails to live migrate instance with dash separated MAC
From: sean mooney, 2024-05-21