← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1819735] Re: live migrations doesn't work for VMs with more than 2 network interfaces

 

[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1819735

Title:
  live migrations doesn't work for VMs with more than 2 network
  interfaces

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Description
  ===========
  When trying to migrate VMs with more than 2 network interfaces (ports) migrations fails.
  On source machine I get following errors:

  2019-03-12 15:25:50.809 16855 WARNING nova.compute.resource_tracker [req-b089fd63-ba03-4f0d-ad6d-0585692810ee 5f620a8543154df1930316a7bf15f47e bd582a2363584a868a2966f1c4a1e56c - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:25:52.312 16855 WARNING nova.compute.manager [req-e1435cec-abb5-4490-979c-9240c377b5be 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-unplugged-4078a1e8-0102-4a52-bb88-c3df517a5a1b for instance with vm_state active and task_state migrating.
  2019-03-12 15:25:55.001 16855 WARNING nova.compute.manager [req-6db93de5-aaa3-4838-b622-d4ceb9faed4b 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-plugged-4078a1e8-0102-4a52-bb88-c3df517a5a1b for instance with vm_state active and task_state migrating.
  2019-03-12 15:25:57.020 16855 WARNING nova.compute.manager [req-600c156c-ed9a-4c98-b6f9-91788cd72b78 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-unplugged-ab0b7818-ae3f-4b42-9d52-6fea23dd521c for instance with vm_state active and task_state migrating.
  2019-03-12 15:25:57.070 16855 WARNING nova.compute.manager [req-daa4ff0d-c18e-47bc-958d-4a07c6128ab3 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-plugged-4078a1e8-0102-4a52-bb88-c3df517a5a1b for instance with vm_state active and task_state migrating.
  2019-03-12 15:25:57.124 16855 WARNING nova.compute.manager [req-a5c2e84a-f828-43a4-803c-c8208fe42ab5 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-plugged-ab0b7818-ae3f-4b42-9d52-6fea23dd521c for instance with vm_state active and task_state migrating.
  2019-03-12 15:25:59.008 16855 WARNING nova.compute.manager [req-660a5963-bab7-44f4-9856-4a6ff1268b42 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-unplugged-c16c6463-4ef6-4969-9d7f-30c3a6152933 for instance with vm_state active and task_state migrating.
  2019-03-12 15:25:59.490 16855 ERROR nova.virt.libvirt.driver [req-04b22e72-a9a4-423e-a904-3396dcf7beb2 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Live Migration failure: Cannot get interface MTU on 'brqb010f6ce-84': No such device: libvirtError: Cannot get interface MTU on 'brqb010f6ce-84': No such device
  2019-03-12 15:25:59.790 16855 WARNING nova.compute.manager [req-6bb36e82-7e4e-4d6c-bd83-2e9d8e9a5688 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-plugged-c16c6463-4ef6-4969-9d7f-30c3a6152933 for instance with vm_state active and task_state migrating.
  2019-03-12 15:26:00.424 16855 WARNING nova.compute.manager [req-953b231f-4736-486f-96c1-ee0b99e88e54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-unplugged-f29b05df-0acd-4b7b-9b3a-e985561567fc for instance with vm_state active and task_state migrating.
  2019-03-12 15:26:08.790 16855 WARNING nova.compute.manager [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Received unexpected event network-vif-plugged-f29b05df-0acd-4b7b-9b3a-e985561567fc for instance with vm_state active and task_state migrating.
  2019-03-12 15:26:50.832 16855 WARNING nova.compute.resource_tracker [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:27:50.953 16855 WARNING nova.compute.resource_tracker [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:28:51.788 16855 WARNING nova.compute.resource_tracker [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:29:52.893 16855 WARNING nova.compute.resource_tracker [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:30:52.852 16855 WARNING nova.compute.resource_tracker [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [req-6d54487a-e65b-4d46-8f21-4a3337b1fb54 39cc388544de4fc0b33f421181fc8683 b9e3c4e073bd4c14a1dba80e4f98636a - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Live migration failed.: MigrationError: Migration error: Timeout waiting for VIF plugging events, canceling migration
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Traceback (most recent call last):
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6070, in _do_live_migration
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f]     block_migration, migrate_data)
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6892, in live_migration
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f]     migrate_data)
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7394, in _live_migration
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f]     raise exception.MigrationError(reason=msg)
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] MigrationError: Migration error: Timeout waiting for VIF plugging events, canceling migration
  2019-03-12 15:30:59.034 16855 ERROR nova.compute.manager [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] 

  and on destination compute node:

  2019-03-12 15:26:03.507 19925 WARNING nova.compute.resource_tracker [req-b089fd63-ba03-4f0d-ad6d-0585692810ee 5f620a8543154df1930316a7bf15f47e bd582a2363584a868a2966f1c4a1e56c - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:27:05.511 19925 WARNING nova.compute.resource_tracker [req-b089fd63-ba03-4f0d-ad6d-0585692810ee 5f620a8543154df1930316a7bf15f47e bd582a2363584a868a2966f1c4a1e56c - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:28:05.567 19925 WARNING nova.compute.resource_tracker [req-b089fd63-ba03-4f0d-ad6d-0585692810ee 5f620a8543154df1930316a7bf15f47e bd582a2363584a868a2966f1c4a1e56c - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:29:05.680 19925 WARNING nova.compute.resource_tracker [req-b089fd63-ba03-4f0d-ad6d-0585692810ee 5f620a8543154df1930316a7bf15f47e bd582a2363584a868a2966f1c4a1e56c - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.
  2019-03-12 15:30:07.848 19925 WARNING nova.compute.resource_tracker [req-b089fd63-ba03-4f0d-ad6d-0585692810ee 5f620a8543154df1930316a7bf15f47e bd582a2363584a868a2966f1c4a1e56c - default default] [instance: 981fcdcd-5121-46f8-b36c-35a20c3e651f] Instance not resizing, skipping migration.


  Steps to reproduce
  ==================
  openstack server show 981fcdcd-5121-46f8-b36c-35a20c3e651f -c addresses
  +-----------+--------------------------------------------------------------------------------------------+
  | Field     | Value                                                                                      |
  +-----------+--------------------------------------------------------------------------------------------+
  | addresses | EXTERNAL=x.x.x.x; CROSSCONNECT=y.y.y.y; production=w.w.w.w; test=z.z.z.z                   |
  +-----------+--------------------------------------------------------------------------------------------+

  openstack server migrate --live comp-a01 981fcdcd-5121-46f8-b36c-
  35a20c3e651f

  
  Expected result
  ===============
  VM should be running on new compute comp-a01

  Actual result
  =============

  VM stays on source compute, in state ERROR, need to use:
  nova reset-state --active 981fcdcd-5121-46f8-b36c-35a20c3e651f
  to restore active state

  Environment
  ===========
  Openstack Queens

  nova controller:
  root@nova-01:~# dpkg -l | grep nova
  ii  nova-api                            2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - API frontend
  ii  nova-common                         2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - common files
  ii  nova-conductor                      2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - conductor service
  ii  nova-consoleauth                    2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                     2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - NoVNC proxy
  ii  nova-placement-api                  2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - placement API frontend
  ii  nova-scheduler                      2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                         2:17.0.5-0ubuntu1~cloud0                   all          OpenStack Compute Python libraries
  ii  python-novaclient                   2:9.1.1-0ubuntu1~cloud0                    all          client library for OpenStack Compute API - Python 2.7
  root@nova-01:~# 

  compute nodes:
  source:
  [root@comp-b02 ~]# dpkg -l | grep nova
  ii  nova-common                         2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - common files
  ii  nova-compute                        2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute Python libraries
  ii  python-novaclient                   2:9.1.1-0ubuntu1~cloud0                     all          client library for OpenStack Compute API - Python 2.7

  destination:
  [root@comp-a01 ~]# dpkg -l | grep nova
  ii  nova-common                         2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - common files
  ii  nova-compute                        2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute Python libraries
  ii  python-novaclient                   2:9.1.1-0ubuntu1~cloud0                     all          client library for OpenStack Compute API - Python 2.7

  Hypervisor:
  Libvirt + KVM
  [root@comp-a01 ~]# dpkg -l | grep libvirt
  ii  libvirt-bin                         4.0.0-1ubuntu8.3~cloud0                     amd64        programs for the libvirt library
  ii  libvirt-clients                     4.0.0-1ubuntu8.3~cloud0                     amd64        Programs for the libvirt library
  ii  libvirt-daemon                      4.0.0-1ubuntu8.3~cloud0                     amd64        Virtualization daemon
  ii  libvirt-daemon-driver-storage-rbd   4.0.0-1ubuntu8.3~cloud0                     amd64        Virtualization daemon RBD storage driver
  ii  libvirt-daemon-system               4.0.0-1ubuntu8.3~cloud0                     amd64        Libvirt daemon configuration files
  ii  libvirt0:amd64                      4.0.0-1ubuntu8.3~cloud0                     amd64        library for interfacing with different virtualization systems
  ii  nova-compute-libvirt                2:17.0.5-0ubuntu1~cloud0                    all          OpenStack Compute - compute node libvirt support
  ii  python-libvirt                      4.0.0-1~cloud0                              amd64        libvirt Python bindings

  
  Storage: ceph, luminous 12.2.9

  Networking: neutron + linux bridge

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1819735/+subscriptions


References