yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1852465] Re: ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 13 Nov 2019 23:35:00 -0000
Reply-to: Bug 1852465 <1852465@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This goes back a long time but I'll just target to the non-extended-
maintenance branches.

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Also affects: nova/train
   Importance: Undecided
       Status: New

** Changed in: nova/rocky
       Status: New => Confirmed

** Changed in: nova/train
       Status: New => Confirmed

** Changed in: nova/rocky
   Importance: Undecided => Low

** Changed in: nova/stein
       Status: New => Confirmed

** Changed in: nova/train
   Importance: Undecided => Low

** Changed in: nova/stein
   Importance: Undecided => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852465

Title:
  ExternalNetworkAttachForbidden should result in BuildAbortException,
  not reschedule

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) rocky series:
  Confirmed
Status in OpenStack Compute (nova) stein series:
  Confirmed
Status in OpenStack Compute (nova) train series:
  Confirmed

Bug description:
  I saw this in a CI run where creating a server on an external network
  as a non-admin user failed and was rescheduled. Here are the failures
  from the compute logs:

  https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774

  https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415

  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Traceback (most recent call last):
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     yield resources
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 2433, in _build_and_run_instance
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     block_device_info=block_device_info)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     mdevs=mdevs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     network_info_str = str(network_info)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/model.py", line 601, in __str__
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     return self._sync_wrapper(fn, *args, **kwargs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     self.wait()
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/model.py", line 616, in wait
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     self[:] = self._gt.wait()
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 181, in wait
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     return self._exit_event.wait()
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     current.throw(*self._exc)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     result = function(*args, **kwargs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/utils.py", line 674, in context_wrapper
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     return func(*args, **kwargs)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 1704, in _allocate_network_async
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     six.reraise(*exc_info)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     raise value
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/compute/manager.py", line 1687, in _allocate_network_async
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     resource_provider_mapping=resource_provider_mapping)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 990, in allocate_for_instance
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     context, instance, neutron, requested_networks, ordered_networks)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 868, in _validate_requested_network_ids
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     self._check_external_network_attach(context, nets)
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]   File "/opt/stack/nova/nova/network/neutronv2/api.py", line 542, in _check_external_network_attach
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]     network_uuid=net['id'])
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
  Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]

  And eventually that results in MaxRetriesExceeded in conductor:

  https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n
  -cond-cell1.txt.gz#878

  Nov 13 15:35:47.968308 ubuntu-bionic-rax-ord-0012778423 nova-
  conductor[25967]: WARNING nova.scheduler.utils [None req-41117798
  -8a4e-469f-bfbb-8bdfdea1a83f demo demo] Failed to
  compute_task_build_instances: Exceeded maximum number of retries.
  Exhausted all hosts available for retrying build failures for instance
  be6eb09a-f0d9-4c04-9c55-29230a253fbd.:
  nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries.
  Exhausted all hosts available for retrying build failures for instance
  be6eb09a-f0d9-4c04-9c55-29230a253fbd.

  ExternalNetworkAttachForbidden is not something that a reschedule is
  going to fix, it's essentially a bad request from the user and needs
  to just abort.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852465/+subscriptions
References

[Bug 1852465] [NEW] ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule
From: Matt Riedemann, 2019-11-13