yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2059962] Re: Unable to Live Migrate Instance. Horizon doesn't show error, logs do

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dave West <2059962@xxxxxxxxxxxxxxxxxx>
Date: Wed, 03 Apr 2024 18:27:05 -0000
Reply-to: Bug 2059962 <2059962@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
This is not a big, found solution.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2059962

Title:
  Unable to Live Migrate Instance.  Horizon doesn't show error, logs do

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  When trying to live migrate an instance between hosts, I get the error
  below.   I can migrate an instance that is shutdown.

  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.665 27905 DEBUG nova.compute.manager [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] live_migration data is LibvirtLiveMigrateData(bdms=[LibvirtLiveMigrateBDMInfo],block_migration=False,disk_available_mb=63488,disk_over_commit=False,dst_numa_info=<?>,dst_supports_numa_live_migration=<?>,dst_wants_file_backed_memory=False,file_backed_memory_discard=<?>,filename='tmpe_9d4avw',graphics_listen_addr_spice=127.0.0.1,graphics_listen_addr_vnc=10.136.149.5,image_type='default',instance_relative_path='b44705f4-6813-42ad-a4a8-a64255e2a6b7',is_shared_block_storage=True,is_shared_instance_path=False,is_volume_backed=True,migration=Migration(7e5bb92b-8d84-4320-b379-ee7b14fed53d),old_vol_attachment_ids={375dff72-24d4-475c-a172-33e356765410='1432f021-67a6-4b51-9988-fcad5a6e4684'},serial_listen_addr=None,serial_listen_ports=[],src_supports_native_luks=<?>,src_supports_numa_live_migration=<?>,supported_perf_events=[],target_connect_addr='10.136.149.5',vifs=[VIFMigrateData],wait_for_vif_plugged=True) _do_live_migration /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/compute/manager.py:8988
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.668 27905 DEBUG nova.objects.instance [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] Lazy-loading 'migration_context' on Instance uuid b44705f4-6813-42ad-a4a8-a64255e2a6b7 obj_load_attr /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/objects/instance.py:1152
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.668 27905 DEBUG nova.virt.libvirt.driver [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] Starting monitoring of live migration _live_migration /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10641
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.671 27905 DEBUG nova.virt.libvirt.driver [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] Operation thread is still running _live_migration_monitor /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10442
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.671 27905 DEBUG nova.virt.libvirt.driver [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] Migration not running yet _live_migration_monitor /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10451
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.681 27905 DEBUG nova.virt.libvirt.migration [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] Find same serial number: pos=0, serial=375dff72-24d4-475c-a172-33e356765410 _update_volume_xml /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/migration.py:242
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.682 27905 DEBUG nova.virt.libvirt.vif [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,compute_id=15,config_drive='',created_at=2024-04-01T16:44:13Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description=None,display_name='test',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(3),hidden=False,host='osp-compute-c02-01.tsa402.service-now.com',hostname='test',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=3,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2024-04-01T16:44:29Z,launched_on='osp-compute-c02-01.tsa402.service-now.com',locked=False,locked_by=None,memory_mb=1024,metadata={},migration_context=None,new_flavor=None,node='osp-compute-c02-01.tsa402.service-now.com',numa_topology=None,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=InstancePCIRequests,power_state=1,progress=0,project_id='18321e0d96a74a4ab1ec27394166fb66',ramdisk_id='',reservation_id='r-b59ilff4',resources=None,root_device_name='/dev/vda',root_gb=20,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='member,reader,admin,manager',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_hw_cdrom_bus='ide',image_hw_disk_bus='virtio',image_hw_input_bus='usb',image_hw_machine_type='pc',image_hw_pointer_model='usbtablet',image_hw_video_model='virtio',image_hw_vif_model='virtio',image_min_disk='20',image_min_ram='0',image_owner_specified.openstack.md5='',image_owner_specified.openstack.object='images/CirrOS-6.2',image_owner_specified.openstack.sha256='',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='migrating',terminated_at=None,trusted_certs=<?>,updated_at=2024-04-01T18:50:15Z,user_data=None,user_id='dea9b2ecc4644988aec805e735f03de3',uuid=b44705f4-6813-42ad-a4a8-a64255e2a6b7,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='active') vif={"id": "702c30b2-f959-489c-894c-0a17e131d0b4", "address": "fa:16:3e:87:97:92", "network": {"id": "143a8e60-fe8f-462e-ba1e-637ee961c180", "bridge": "br-int", "label": "admin.local", "subnets": [{"cidr": "192.168.0.0/24", "dns": [{"address": "10.255.0.108", "type": "dns", "version": 4, "meta": {}}], "gateway": {"address": "192.168.0.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "192.168.0.34", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"enable_dhcp": true}}], "meta": {"injected": false, "tenant_id": "18321e0d96a74a4ab1ec27394166fb66", "mtu": 1500, "physical_network": null, "tunneled": true}}, "type": "ovs", "details": {"port_filter": true, "connectivity": "l2", "bridge_name": "br-int", "datapath_type": "system"}, "devname": "tap702c30b2-f9", "ovs_interfaceid": "702c30b2-f959-489c-894c-0a17e131d0b4", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {"os_vif_delegation": true}, "preserve_on_delete": false, "delegate_create": true, "meta": {}} virt_type=kvm get_config /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/vif.py:563
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.682 27905 DEBUG nova.network.os_vif_util [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] Converting VIF {"id": "702c30b2-f959-489c-894c-0a17e131d0b4", "address": "fa:16:3e:87:97:92", "network": {"id": "143a8e60-fe8f-462e-ba1e-637ee961c180", "bridge": "br-int", "label": "admin.local", "subnets": [{"cidr": "192.168.0.0/24", "dns": [{"address": "10.255.0.108", "type": "dns", "version": 4, "meta": {}}], "gateway": {"address": "192.168.0.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "192.168.0.34", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"enable_dhcp": true}}], "meta": {"injected": false, "tenant_id": "18321e0d96a74a4ab1ec27394166fb66", "mtu": 1500, "physical_network": null, "tunneled": true}}, "type": "ovs", "details": {"port_filter": true, "connectivity": "l2", "bridge_name": "br-int", "datapath_type": "system"}, "devname": "tap702c30b2-f9", "ovs_interfaceid": "702c30b2-f959-489c-894c-0a17e131d0b4", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {"os_vif_delegation": true}, "preserve_on_delete": false, "delegate_create": true, "meta": {}} nova_to_osvif_vif /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/network/os_vif_util.py:511
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.682 27905 DEBUG nova.network.os_vif_util [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] Converted object VIFOpenVSwitch(active=False,address=fa:16:3e:87:97:92,bridge_name='br-int',has_traffic_filtering=True,id=702c30b2-f959-489c-894c-0a17e131d0b4,network=Network(143a8e60-fe8f-462e-ba1e-637ee961c180),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap702c30b2-f9') nova_to_osvif_vif /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/network/os_vif_util.py:548
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.683 27905 DEBUG nova.virt.libvirt.migration [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] Updating guest XML with vif config: <interface type="ethernet">#012  <mac address="fa:16:3e:87:97:92"/>#012  <model type="virtio"/>#012  <mtu size="1500"/>#012  <target dev="tap702c30b2-f9"/>#012</interface>#012 _update_vif_xml /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/migration.py:388
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.683 27905 DEBUG nova.virt.libvirt.driver [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] About to invoke the migrate API _live_migration_operation /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10274
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.786 27905 ERROR nova.virt.libvirt.driver [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] Live Migration failure: (domain_definition):1: Start tag expected, '<' not found#012* openvswitch.service - Open vSwitch#012^: libvirt.libvirtError: (domain_definition):1: Start tag expected, '<' not found
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01 11:50:56.787 27905 DEBUG nova.virt.libvirt.driver [None req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3 18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance: b44705f4-6813-42ad-a4a8-a64255e2a6b7] Migration operation thread notification thread_finished /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10632
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: Traceback (most recent call last):
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    timer()
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    cb(*args, **kw)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/event.py", line 175, in _do_send
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    waiter.switch(result)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/greenthread.py", line 221, in main
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    result = function(*args, **kwargs)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/utils.py", line 664, in context_wrapper
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    return func(*args, **kwargs)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py", line 10287, in _live_migration_operation
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    with excutils.save_and_reraise_exception():
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    self.force_reraise()
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    raise self.value
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/guest.py", line 642, in migrate
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    self._domain.migrateToURI3(
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    result = proxy_call(self._autowrap, f, *args, **kwargs)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    rv = execute(f, *args, **kwargs)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    six.reraise(c, e, tb)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/six.py", line 719, in reraise
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    raise value
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    rv = meth(*args, **kwargs)
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:  File "/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/libvirt.py", line 2168, in migrateToURI3
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]:    raise libvirtError('virDomainMigrateToURI3() failed')
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: libvirt.libvirtError: (domain_definition):1: Start tag expected, '<' not found
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: * openvswitch.service - Open vSwitch
  Apr  1 11:50:56 osp-compute-c02-01 nova-compute[27905]: ^
  Apr  1 11:50:57 osp-compute-c01-03 nova-compute[28672]: 2024-04-01 11:50:57.045 28672 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 23 __log_wakeup /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/ovs/poller.py:263
  Apr  1 11:50:57 osp-compute-c01-01 nova-compute[27897]: 2024-04-01 11:50:57.202 27897 DEBUG oslo_service.periodic_task [None req-161fba31-483d-4517-ade1-176e066f8c31 - - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/oslo_service/periodic_task.py:210

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2059962/+subscriptions
References

[Bug 2059962] [NEW] Unable to Live Migrate Instance. Horizon doesn't show error, logs do
From: Dave West, 2024-04-01