yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1540764] [NEW] build instance fail with sriov port on child cell

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: likun <li.kun130@xxxxxxxxxx>
Date: Tue, 02 Feb 2016 06:30:49 -0000
Reply-to: Bug 1540764 <1540764@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Version:
Kilo
Liberty

When I was trying to build instance with sriov port on my child cell, I used the following command:
nova boot --image 6df3b3d4-3c9e-4772-9f18-b6f42c6a9c77 --flavor 3 --nic port-id=61c20d21-43fe-487d-9296-893172cb725f --hint target_cell='api_cell!child_cell' sr-vxlan_vm30

But spawning failed on child_cell compute node. Here are error
information in nova-compute.log:

2016-01-14 11:00:17.246 2908 ERROR nova.compute.manager [req-9d7fafab-ea2a-43c3-a201-4234db29c572 8a9704e00c44452590c6c8d014ed028a 452b30922b3f42b59743158f695264c9 - - -] [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c] Instance failed to spawn
016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c] Traceback (most recent call last):
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2660, in _build_resources
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     yield resources
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2532, in _build_and_run_instance
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     block_device_info=block_device_info)
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2879, in spawn
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     write_to_disk=True)
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4844, in _get_guest_xml
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     memory_backup_file=memory_backup_file)
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4675, in _get_guest_config
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     network_info, virt_type)
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4818, in _get_guest_vif_config
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     instance, vif, image_meta, flavor, virt_type)
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 376, in get_config
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     inst_type, virt_type)
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 292, in get_config_hw_veb
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     conf, net_type, profile["pci_slot"],
2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c] KeyError: 'pci_slot'


In function _create_instances_here() in nova/cells/scheduler.py of Kilo
and Liberty, I found this:

        # FIXME(danms): The instance was brutally serialized before being
        # sent over RPC to us. Thus, the pci_requests value wasn't really
        # sent in a useful form. Since it was getting ignored for cells
        # before it was part of the Instance, skip it now until cells RPC
        # is sending proper instance objects.
        instance_values.pop('pci_requests', None)

The "pci_requests" is discarded by cell scheduler, because of unuseful
instance form.This is the causation of above error.

** Affects: nova
     Importance: Undecided
     Assignee: likun (li-kun130)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => likun (li-kun130)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1540764

Title:
  build instance fail with sriov port on child cell

Status in OpenStack Compute (nova):
  New

Bug description:
  Version:
  Kilo
  Liberty

  When I was trying to build instance with sriov port on my child cell, I used the following command:
  nova boot --image 6df3b3d4-3c9e-4772-9f18-b6f42c6a9c77 --flavor 3 --nic port-id=61c20d21-43fe-487d-9296-893172cb725f --hint target_cell='api_cell!child_cell' sr-vxlan_vm30

  But spawning failed on child_cell compute node. Here are error
  information in nova-compute.log:

  2016-01-14 11:00:17.246 2908 ERROR nova.compute.manager [req-9d7fafab-ea2a-43c3-a201-4234db29c572 8a9704e00c44452590c6c8d014ed028a 452b30922b3f42b59743158f695264c9 - - -] [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c] Instance failed to spawn
  016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c] Traceback (most recent call last):
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2660, in _build_resources
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     yield resources
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2532, in _build_and_run_instance
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     block_device_info=block_device_info)
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2879, in spawn
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     write_to_disk=True)
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4844, in _get_guest_xml
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     memory_backup_file=memory_backup_file)
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4675, in _get_guest_config
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     network_info, virt_type)
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4818, in _get_guest_vif_config
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     instance, vif, image_meta, flavor, virt_type)
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 376, in get_config
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     inst_type, virt_type)
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 292, in get_config_hw_veb
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c]     conf, net_type, profile["pci_slot"],
  2016-01-14 11:00:17.246 2908 TRACE nova.compute.manager [instance: b91305de-5501-4083-9a0e-0f4bfc942f5c] KeyError: 'pci_slot'


  In function _create_instances_here() in nova/cells/scheduler.py of
  Kilo and Liberty, I found this:

          # FIXME(danms): The instance was brutally serialized before being
          # sent over RPC to us. Thus, the pci_requests value wasn't really
          # sent in a useful form. Since it was getting ignored for cells
          # before it was part of the Instance, skip it now until cells RPC
          # is sending proper instance objects.
          instance_values.pop('pci_requests', None)

  The "pci_requests" is discarded by cell scheduler, because of unuseful
  instance form.This is the causation of above error.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1540764/+subscriptions
Follow ups

[Bug 1540764] Re: build instance fail with sriov port on child cell
From: Sean Dague, 2016-02-17