yahoo-eng-team team mailing list archive

[OpenStack Infra <1852465@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/694179
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=36b110dfc5f8f04234b189a50e96de8dde5f3471
Submitter: Zuul
Branch:    master

commit 36b110dfc5f8f04234b189a50e96de8dde5f3471
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Wed Nov 13 18:27:36 2019 -0500

    Do not reschedule on ExternalNetworkAttachForbidden
    
    When creating a server, if network setup fails with
    ExternalNetworkAttachForbidden then we should abort
    the build rather than reschedule because it's not
    going to work on another host either.
    
    A functional test is added for this since it's hard
    to reason about whether or not it works properly
    due to the async nature of network setup during
    server create and the _build_resources context
    manager. It's probably not a bad idea to have a
    functional test for a scenario that uses external
    networks either (and we can build on this).
    
    Change-Id: I6dd5b3f75e2746f53f9758f1a186794a98dba9bf
    Closes-Bug: #1852465


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852465

Title:
  ExternalNetworkAttachForbidden should result in BuildAbortException,
  not reschedule

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) rocky series:
  Confirmed
Status in OpenStack Compute (nova) stein series:
  Confirmed
Status in OpenStack Compute (nova) train series:
  Confirmed

Bug description:
  I saw this in a CI run where creating a server on an external network
  as a non-admin user failed and was rescheduled. Here are the failures
  from the compute logs:

  https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774

  https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415

  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Traceback (most recent call last):
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     yield resources
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 2433, in _build_and_run_instance
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     block_device_info=block_device_info)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     mdevs=mdevs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     network_info_str = str(network_info)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/model.py", line 601, in __str__
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     return self._sync_wrapper(fn, *args, **kwargs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     self.wait()
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/model.py", line 616, in wait
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     self[:] = self._gt.wait()
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 181, in wait
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     return self._exit_event.wait()
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     current.throw(*self._exc)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     result = function(*args, **kwargs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/utils.py", line 674, in context_wrapper
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     return func(*args, **kwargs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 1704, in _allocate_network_async
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     six.reraise(*exc_info)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     raise value
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 1687, in _allocate_network_async
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     resource_provider_mapping=resource_provider_mapping)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 990, in allocate_for_instance
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     context, instance, neutron, requested_networks, ordered_networks)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 868, in _validate_requested_network_ids
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     self._check_external_network_attach(context, nets)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 542, in _check_external_network_attach
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     network_uuid=net['id'])
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]

  And eventually that results in MaxRetriesExceeded in conductor:

  https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n
  -cond-cell1.txt.gz#878

  Nov 13 15:35:47.968308 ubuntu-bionic-rax-ord-0012778423 nova-
  conductor[25967]: WARNING nova.scheduler.utils [None req-41117798
  -8a4e-469f-bfbb-8bdfdea1a83f demo demo] Failed to
  compute_task_build_instances: Exceeded maximum number of retries.
  Exhausted all hosts available for retrying build failures for instance
  be6eb09a-f0d9-4c04-9c55-29230a253fbd.:
  nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries.
  Exhausted all hosts available for retrying build failures for instance
  be6eb09a-f0d9-4c04-9c55-29230a253fbd.

  ExternalNetworkAttachForbidden is not something that a reschedule is
  going to fix, it's essentially a bad request from the user and needs
  to just abort.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852465/+subscriptions