yahoo-eng-team team mailing list archive

[Liusheng <liusheng@xxxxxxxxxx>]

Public bug reported:

consider a scenario, we have more than 1 compute node in our environment. in some cases, when booting a instance failed because a libvirt error cause spawning failed, 
according the implementation of of _build_instance(), the exceptions will be accepted as "Exception", and then, reschedule the instance or set it to error state. the related code is:

        except Exception:
            exc_info = sys.exc_info()
            # try to re-schedule instance:
            # Make sure the async call finishes
            if network_info is not None:
                network_info.wait(do_raise=False)
            rescheduled = self._reschedule_or_error(context, instance,
                    exc_info, requested_networks, admin_password,
                    injected_files_orig, is_first_time, request_spec,
                    filter_properties, bdms, legacy_bdm_in_spec)
if we have specified a port when booting the instance, the port['device_id'] is the uuid of the this instance. a "PortInUse" exception will be raised after rescheduling successfully and trying to build on another node.

To reproduce, we can add "raise Exception" after spawn in
_build_instance().

more details see:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1043, in _build_instance
    set_access_ip=set_access_ip)
  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1426, in _spawn
    LOG.exception(_('Instance failed to spawn'), instance=instance)
  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1423, in _spawn
    block_device_info)
  File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2083, in spawn
    admin_pass=admin_password)
  File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2480, in _create_image
    net = netutils.get_injected_network_template(network_info)
  File "/usr/lib64/python2.6/site-packages/nova/virt/netutils.py", line 74, in get_injected_network_
    if not (network_info and template):
  File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 379, in __len__
    return self._sync_wrapper(fn, *args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 366, in _sync_wrapper
    self.wait()
  File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 398, in wait
    self[:] = self._gt.wait()
  File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
    return self._exit_event.wait()
  File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 120, in wait
    current.throw(*self._exc)
  File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1244, in _allocate_network
    dhcp_options=dhcp_options)
  File "/usr/lib64/python2.6/site-packages/nova/network/neutronv2/api.py", line 243, in allocate_for
    raise exception.PortInUse(port_id=port_id)
PortInUse: Port faf3aa64-11f8-4fc7-81bc-084098014f4a is still in use.

** Affects: nova
     Importance: Undecided
     Assignee: Liusheng (liusheng)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Liusheng (liusheng)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1308405

Title:
  reschedule failed because port still in use

Status in OpenStack Compute (Nova):
  New

Bug description:
  consider a scenario, we have more than 1 compute node in our environment. in some cases, when booting a instance failed because a libvirt error cause spawning failed, 
  according the implementation of of _build_instance(), the exceptions will be accepted as "Exception", and then, reschedule the instance or set it to error state. the related code is:

          except Exception:
              exc_info = sys.exc_info()
              # try to re-schedule instance:
              # Make sure the async call finishes
              if network_info is not None:
                  network_info.wait(do_raise=False)
              rescheduled = self._reschedule_or_error(context, instance,
                      exc_info, requested_networks, admin_password,
                      injected_files_orig, is_first_time, request_spec,
                      filter_properties, bdms, legacy_bdm_in_spec)
  if we have specified a port when booting the instance, the port['device_id'] is the uuid of the this instance. a "PortInUse" exception will be raised after rescheduling successfully and trying to build on another node.

  To reproduce, we can add "raise Exception" after spawn in
  _build_instance().

  more details see:
  Traceback (most recent call last):
    File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1043, in _build_instance
      set_access_ip=set_access_ip)
    File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1426, in _spawn
      LOG.exception(_('Instance failed to spawn'), instance=instance)
    File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1423, in _spawn
      block_device_info)
    File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2083, in spawn
      admin_pass=admin_password)
    File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2480, in _create_image
      net = netutils.get_injected_network_template(network_info)
    File "/usr/lib64/python2.6/site-packages/nova/virt/netutils.py", line 74, in get_injected_network_
      if not (network_info and template):
    File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 379, in __len__
      return self._sync_wrapper(fn, *args, **kwargs)
    File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 366, in _sync_wrapper
      self.wait()
    File "/usr/lib64/python2.6/site-packages/nova/network/model.py", line 398, in wait
      self[:] = self._gt.wait()
    File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
      return self._exit_event.wait()
    File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 120, in wait
      current.throw(*self._exc)
    File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
      result = function(*args, **kwargs)
    File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 1244, in _allocate_network
      dhcp_options=dhcp_options)
    File "/usr/lib64/python2.6/site-packages/nova/network/neutronv2/api.py", line 243, in allocate_for
      raise exception.PortInUse(port_id=port_id)
  PortInUse: Port faf3aa64-11f8-4fc7-81bc-084098014f4a is still in use.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1308405/+subscriptions