yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1527061] Re: Nova should not throw exception when port binding fails for Ironic

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eli Qiao <liyong.qiao@xxxxxxxxx>
Date: Wed, 20 Jan 2016 05:36:45 -0000
Reply-to: Bug 1527061 <1527061@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
hi Johnson,

actually, I don't know well about VMware VCDriver, and have no idea no
whey Neutron return a port  with {'binding:vif_type': binding_failed} ,
in case this happened, nova should though it as a bad port and is not
usable. I personal suggest that we connect Neutron team to take care
about this.

Eli.

** Project changed: nova => neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1527061

Title:
  Nova should not throw exception when port binding fails for Ironic

Status in neutron:
  Confirmed

Bug description:
  Neutron tries to bind port on compute where instance is launched. It
  doesn't make sense when hypervisor_type is ironic, since VM does not
  live on hypervisor in this case. Furthermore it leads to failed
  provisioning of baremetal node, when neutron is not configured on
  ironic compute node.

  When binding fails, nova starts cleanup and then throws exception. As
  part of cleanup ironic port is deleted. This should not be the case
  when virt-driver being used is Ironic.

  Nova versions: liberty on-wards.

  Setup:
  node-1: controller with neutron
  node-2: ironic-compute without neutron/neutron agents

  Register a BM node and perform nova boot.

  From Nova:
  =============================================================
  nova-compute.log:21147:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  nova-compute.log:21148:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager Traceback (most recent call last):
  nova-compute.log:21149:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py", line 1564, in _allocate_network_async
  nova-compute.log:21150:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager     dhcp_options=dhcp_options)
  nova-compute.log:21151:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 727, in allocate_for_instance
  nova-compute.log:21152:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager     self._delete_ports(neutron, instance, created_port_ids)
  nova-compute.log:21153:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
  nova-compute.log:21154:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
  nova-compute.log:21155:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 719, in allocate_for_instance
  nova-compute.log:21156:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager     security_group_ids, available_macs, dhcp_opts)
  nova-compute.log:21157:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 342, in _create_port
  nova-compute.log:21158:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager     raise exception.PortBindingFailed(port_id=port_id)
  nova-compute.log:21159:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 0514b7e8-0408-4d5e-9b48-8292e686494f, please check neutron logs for more information.
  nova-compute.log:21160:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
  2015-12-05 02:57:43.375 7649 DEBUG nova.virt.ironic.driver [req-25ba4c81-4333-43e4-b8d9-e460827435e0 a989fe7fc89e4825b986666d7e6584cc 1308f1a382ef43c79eeed0ebf8a9db3b] unplug: instance_uuid=d9295d4d-1104-47ab-8e75-5b30d0a3838b vif=[] _unp
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [req-25ba4c81-4333-43e4-b8d9-e460827435e0 a989fe7fc89e4825b986666d7e6584cc 1308f1a382ef43c79eeed0ebf8a9db3b] [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b] Instance failed to spawn
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b] Traceback (most recent call last):
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py", line 2157, in _build_resources
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     yield resources
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py", line 2011, in _build_and_run_inst
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     block_device_info=block_device_info)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 791, in spawn
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     flavor=flavor)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     six.reraise(self.type_, self.value, self.tb)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 782, in spawn
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     self._plug_vifs(node, instance, network_info)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1068, in _plug_vifs
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     network_info_str = str(network_info)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py", line 515, in __str__
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     return self._sync_wrapper(fn, *args, **kwargs)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py", line 498, in _sync_wrapper
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     self.wait()
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py", line 530, in wait
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     self[:] = self._gt.wait()
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     return self._exit_event.wait()
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     current.throw(*self._exc)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     result = function(*args, **kwargs)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py", line 1581, in _allocate_network_a
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     six.reraise(*exc_info)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py", line 1564, in _allocate_network_a
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     dhcp_options=dhcp_options)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 727, in allocate_for_i
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     self._delete_ports(neutron, instance, created_port_ids)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     six.reraise(self.type_, self.value, self.tb)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 719, in allocate_for_i
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     security_group_ids, available_macs, dhcp_opts)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]   File "/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 342, in _create_port
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]     raise exception.PortBindingFailed(port_id=port_id)
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b] PortBindingFailed: Binding failed for port 0514b7e8-0408-4d5e-9b48-8292e686494f, please check neutron logs for more information.
  2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance: d9295d4d-1104-47ab-8e75-5b30d0a3838b]
  =============================================================

  From Neutron:
  =============================================================
  f 95bfe8c1d4fe49e38b599e801bc94f1c - - -] Failed to bind port 0514b7e8-0408-4d5e-9b48-8292e686494f on host helion-cp1-ir-compute0001-mgmt 
  2015-12-05 02:57:37.144 845 ERROR neutron.plugins.ml2.managers [req-53993456-ea74-452a-8299-149505d99bb2 0a8461f0be414de584e666aae579cc8f 95bfe8c1d4fe49e38b599e801bc94f1c - - -] Failed to bind port 0514b7e8-0408-4d5e-9b48-8292e686494f on host helion-cp1-ir-compute0001-mgmt 
  2015-12-05 02:57:40.520 845 DEBUG keystoneclient.session [-] Request returned failure status: 404 request /opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py:400
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'tag': u'0514b7e8-0408-4d5e-9b48-8292e686494f', 'name': 'network-vif-deleted', 'server_uuid': u'd9295d4d-1104-47ab-8e75-5b30d0a3838b'}]
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova Traceback (most recent call last):
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in send_events
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     batched_events)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return_raw=True)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     _resp, body = self.api.client.post(url, body=body)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/client.py", line 91, in request
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return func(*args, **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in request
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     auth_headers = self.get_auth_headers(auth)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in get_auth_headers
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return auth.get_headers(self, **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142, in get_headers
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     token = self.get_token(session)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return self.get_access(session).auth_token
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     self.auth_ref = self.get_auth_ref(session)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     authenticated=False, log=False)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     return func(*args, **kwargs)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova   File "/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova     raise exceptions.from_response(resp, method, url)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-1bdb3964-e31f-41bc-b503-116e3628b6a4)
  2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1527061/+subscriptions
References

[Bug 1527061] [NEW] Nova should not throw exception when port binding fails for Ironic
From: Om Kumar, 2015-12-17