yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1821733] [NEW] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Stephen Finucane <sfinucan@xxxxxxxxxx>
Date: Tue, 26 Mar 2019 12:35:03 -0000
Reply-to: Bug 1821733 <1821733@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Reproduced from rhbz#1686511
(https://bugzilla.redhat.com/show_bug.cgi?id=1686511)

When spawning an Openstack instance, this error is received:

    2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment
    Traceback (most recent call last):

      File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
        res = self.dispatcher.dispatch(message)

      File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
        return self._do_dispatch(endpoint, method, ctxt, args)

      File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
        result = func(ctxt, **new_args)

      File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
        return func(*args, **kwargs)

      File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
        dests = self.driver.select_destinations(ctxt, spec_obj)

      File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations
        selected_hosts = self._schedule(context, spec_obj)

      File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule
        spec_obj, index=num)

      File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts
        hosts, spec_obj, index)

      File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects
        list_objs = list(objs)

      File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all
        if self._filter_one(obj, spec_obj):

      File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
        return self.host_passes(obj, spec)

      File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes
        pci_stats=host_state.pci_stats))

      File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host
        host_cell, instance_cell, limits)

      File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell
        host_cell, instance_cell)

      File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning
        max(map(len, host_cell.siblings)))

      File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores
        itertools.chain(*sibling_set)))

    UnboundLocalError: local variable 'sibling_set' referenced before
assignment

    2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req-
e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4
8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f-
8b7a-f2f3bc168b29] Setting instance to ERROR state.

This issues appears to be because of:

https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875

This works normally because of loop variables in Python are available
outside of the scope of the loop:

    >>> for x in range(5):
    ...     pass
    ...
    >>> print(x)
    4

and because there's usually something in sibling_sets. However, this is
presumably failing for this user because there are no free cores at all
on the given host. This is likely the race condition between the nova-
scheduler and nova-compute services.

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

- Reproduced from rhbz#1686511.
+ Reproduced from rhbz#1686511
+ (https://bugzilla.redhat.com/show_bug.cgi?id=1686511)
  
  When spawning an Openstack instance, this error is received:
  
+     2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment
+     Traceback (most recent call last):
  
-     2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment
-     Traceback (most recent call last):
+       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
+         res = self.dispatcher.dispatch(message)
  
-       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
-         res = self.dispatcher.dispatch(message)
+       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
+         return self._do_dispatch(endpoint, method, ctxt, args)
  
-       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
-         return self._do_dispatch(endpoint, method, ctxt, args)
+       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
+         result = func(ctxt, **new_args)
  
-       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
-         result = func(ctxt, **new_args)
+       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
+         return func(*args, **kwargs)
  
-       File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
-         return func(*args, **kwargs)
+       File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
+         dests = self.driver.select_destinations(ctxt, spec_obj)
  
-       File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
-         dests = self.driver.select_destinations(ctxt, spec_obj)
+       File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations
+         selected_hosts = self._schedule(context, spec_obj)
  
-       File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations
-         selected_hosts = self._schedule(context, spec_obj)
+       File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule
+         spec_obj, index=num)
  
-       File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule
-         spec_obj, index=num)
+       File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts
+         hosts, spec_obj, index)
  
-       File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts
-         hosts, spec_obj, index)
+       File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects
+         list_objs = list(objs)
  
-       File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects
-         list_objs = list(objs)
+       File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all
+         if self._filter_one(obj, spec_obj):
  
-       File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all
-         if self._filter_one(obj, spec_obj):
+       File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
+         return self.host_passes(obj, spec)
  
-       File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
-         return self.host_passes(obj, spec)
+       File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes
+         pci_stats=host_state.pci_stats))
  
-       File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes
-         pci_stats=host_state.pci_stats))
+       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host
+         host_cell, instance_cell, limits)
  
-       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host
-         host_cell, instance_cell, limits)
+       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell
+         host_cell, instance_cell)
  
-       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell
-         host_cell, instance_cell)
+       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning
+         max(map(len, host_cell.siblings)))
  
-       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning
-         max(map(len, host_cell.siblings)))
+       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores
+         itertools.chain(*sibling_set)))
  
-       File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores
-         itertools.chain(*sibling_set)))
- 
-     UnboundLocalError: local variable 'sibling_set' referenced before
+     UnboundLocalError: local variable 'sibling_set' referenced before
  assignment
  
-     2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req-
+     2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req-
  e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4
  8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f-
  8b7a-f2f3bc168b29] Setting instance to ERROR state.
- 
  
  This issues appears to be because of:
  
  https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875
  
  This works normally because of loop variables in Python are available
  outside of the scope of the loop:
  
-     >>> for x in range(5):
-     ...     pass
-     ...
-     >>> print(x)
-     4
+     >>> for x in range(5):
+     ...     pass
+     ...
+     >>> print(x)
+     4
  
  and because there's usually something in sibling_sets. However, this is
  presumably failing for this user because there are no free cores at all
  on the given host. This is likely the race condition between the nova-
  scheduler and nova-compute services.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1821733

Title:
  Failed to compute_task_build_instances: local variable 'sibling_set'
  referenced before assignment

Status in OpenStack Compute (nova):
  New

Bug description:
  Reproduced from rhbz#1686511
  (https://bugzilla.redhat.com/show_bug.cgi?id=1686511)

  When spawning an Openstack instance, this error is received:

      2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment
      Traceback (most recent call last):

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
          res = self.dispatcher.dispatch(message)

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
          return self._do_dispatch(endpoint, method, ctxt, args)

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
          result = func(ctxt, **new_args)

        File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
          return func(*args, **kwargs)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
          dests = self.driver.select_destinations(ctxt, spec_obj)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations
          selected_hosts = self._schedule(context, spec_obj)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule
          spec_obj, index=num)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts
          hosts, spec_obj, index)

        File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects
          list_objs = list(objs)

        File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all
          if self._filter_one(obj, spec_obj):

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
          return self.host_passes(obj, spec)

        File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes
          pci_stats=host_state.pci_stats))

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host
          host_cell, instance_cell, limits)

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell
          host_cell, instance_cell)

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning
          max(map(len, host_cell.siblings)))

        File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores
          itertools.chain(*sibling_set)))

      UnboundLocalError: local variable 'sibling_set' referenced before
  assignment

      2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req-
  e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4
  8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f-
  8b7a-f2f3bc168b29] Setting instance to ERROR state.

  This issues appears to be because of:

  https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875

  This works normally because of loop variables in Python are available
  outside of the scope of the loop:

      >>> for x in range(5):
      ...     pass
      ...
      >>> print(x)
      4

  and because there's usually something in sibling_sets. However, this
  is presumably failing for this user because there are no free cores at
  all on the given host. This is likely the race condition between the
  nova-scheduler and nova-compute services.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1821733/+subscriptions
Follow ups

[Bug 1821733] Re: Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment
From: Matt Riedemann, 2019-04-01