yahoo-eng-team team mailing list archive

[Alexandre arents <1881881@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========

When no more pci slot are available for hot pluggable network interface,
nova api return HTTP 500 internal error which is not very helpfull from
client point of view.

It seems that nova catch all libvirt error and raise: 
nova.exception.InterfaceAttachFailed: Failed to attach network adapter device to 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53

Should it be better to handle "libvirt.libvirtError: internal error: No more available PCI slots"   As a exc.HTTPConflict()  instead of webob.exc.HTTPInternalServerError()  for example?
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/attach_interfaces.py#L182

Steps to reproduce
==================

On a fresh devstack master install

1) spawn one instance

2) create 30 network interfaces
for i in $(seq 2 30) ; do openstack port create --network private port-${i} ; done

3) Attach interface to isntances:
for i in $(seq 2 30) ; do openstack server add port 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53 port-${i} ; done
last attach ones should failed we the folling out put:

openstack server add port 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53 port-26
Failed to attach network adapter device to 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53 (HTTP 500) (Request-ID: req-03476f4a-97ac-483d-bf15-e9f0bda776d4)

in Logs:

Jun 03 09:02:06 alex-devstack nova-compute[28931]: ERROR nova.virt.libvirt.driver [instance: 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53]   File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 593, in attachDeviceFlags
Jun 03 09:02:06 alex-devstack nova-compute[28931]: ERROR nova.virt.libvirt.driver [instance: 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
Jun 03 09:02:06 alex-devstack nova-compute[28931]: ERROR nova.virt.libvirt.driver [instance: 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53] libvirt.libvirtError: internal error: No more available PCI slots
...
Jun 03 09:02:22 alex-devstack nova-compute[28931]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 7574, in attach_interface
Jun 03 09:02:22 alex-devstack nova-compute[28931]: ERROR oslo_messaging.rpc.server     instance_uuid=instance.uuid)
Jun 03 09:02:22 alex-devstack nova-compute[28931]: ERROR oslo_messaging.rpc.server nova.exception.InterfaceAttachFailed: Failed to attach network adapter device to 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53

Expected result
===============
Getting HTTP conflict  or other explicit response indicating that we reach a limit.

Actual result
=============
Getting HTTP 500 internal error.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1881881

Title:
  nova api return HTTP 500 error when interface attach failed due to
  lack of instance PCI slots

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  When no more pci slot are available for hot pluggable network
  interface, nova api return HTTP 500 internal error which is not very
  helpfull from client point of view.

  It seems that nova catch all libvirt error and raise: 
  nova.exception.InterfaceAttachFailed: Failed to attach network adapter device to 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53

  Should it be better to handle "libvirt.libvirtError: internal error: No more available PCI slots"   As a exc.HTTPConflict()  instead of webob.exc.HTTPInternalServerError()  for example?
  https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/attach_interfaces.py#L182

  Steps to reproduce
  ==================

  On a fresh devstack master install

  1) spawn one instance

  2) create 30 network interfaces
  for i in $(seq 2 30) ; do openstack port create --network private port-${i} ; done

  3) Attach interface to isntances:
  for i in $(seq 2 30) ; do openstack server add port 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53 port-${i} ; done
  last attach ones should failed we the folling out put:

  openstack server add port 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53 port-26
  Failed to attach network adapter device to 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53 (HTTP 500) (Request-ID: req-03476f4a-97ac-483d-bf15-e9f0bda776d4)

  in Logs:

  Jun 03 09:02:06 alex-devstack nova-compute[28931]: ERROR nova.virt.libvirt.driver [instance: 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53]   File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 593, in attachDeviceFlags
  Jun 03 09:02:06 alex-devstack nova-compute[28931]: ERROR nova.virt.libvirt.driver [instance: 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  Jun 03 09:02:06 alex-devstack nova-compute[28931]: ERROR nova.virt.libvirt.driver [instance: 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53] libvirt.libvirtError: internal error: No more available PCI slots
  ...
  Jun 03 09:02:22 alex-devstack nova-compute[28931]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 7574, in attach_interface
  Jun 03 09:02:22 alex-devstack nova-compute[28931]: ERROR oslo_messaging.rpc.server     instance_uuid=instance.uuid)
  Jun 03 09:02:22 alex-devstack nova-compute[28931]: ERROR oslo_messaging.rpc.server nova.exception.InterfaceAttachFailed: Failed to attach network adapter device to 0c6d2b7a-07d8-4f64-baa8-a6c05fb6ce53

  Expected result
  ===============
  Getting HTTP conflict  or other explicit response indicating that we reach a limit.

  Actual result
  =============
  Getting HTTP 500 internal error.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1881881/+subscriptions