yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Invalid

** Changed in: nova/queens
       Status: New => In Progress

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova/queens
     Assignee: (unassigned) => Stephen Finucane (stephenfinucane)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1821733

Title:
  Failed to compute_task_build_instances: local variable 'sibling_set'
  referenced before assignment

Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) queens series:
  In Progress

Bug description:
  Reproduced from rhbz#1686511
  (https://bugzilla.redhat.com/show_bug.cgi?id=1686511)

  When spawning an Openstack instance, this error is received:

      2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment
      Traceback (most recent call last):

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
          res = self.dispatcher.dispatch(message)

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
          return self._do_dispatch(endpoint, method, ctxt, args)

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
          result = func(ctxt, **new_args)

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
          return func(*args, **kwargs)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
          dests = self.driver.select_destinations(ctxt, spec_obj)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations
          selected_hosts = self._schedule(context, spec_obj)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule
          spec_obj, index=num)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts
          hosts, spec_obj, index)

        File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects
          list_objs = list(objs)

        File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all
          if self._filter_one(obj, spec_obj):

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
          return self.host_passes(obj, spec)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes
          pci_stats=host_state.pci_stats))

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host
          host_cell, instance_cell, limits)

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell
          host_cell, instance_cell)

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning
          max(map(len, host_cell.siblings)))

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores
          itertools.chain(*sibling_set)))

      UnboundLocalError: local variable 'sibling_set' referenced before
  assignment

      2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req-
  e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4
  8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f-
  8b7a-f2f3bc168b29] Setting instance to ERROR state.

  This issues appears to be because of:

  https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875

  This works normally because of loop variables in Python are available
  outside of the scope of the loop:

      >>> for x in range(5):
      ...     pass
      ...
      >>> print(x)
      4

  and because there's usually something in sibling_sets. However, this
  is presumably failing for this user because there are no free cores at
  all on the given host. This is likely the race condition between the
  nova-scheduler and nova-compute services.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1821733/+subscriptions