yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => havana-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1192131

Title:
  giant race in quantum - quantumclient key errors break creation of
  guests in nova

Status in OpenStack Neutron (virtual network service):
  Invalid
Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  The current #1 issue the in gate is a quantum key error which looks to
  be a race in how quantum responds to requests from nova. This can be
  seen with a representative error here:

  2013-06-17 15:32:13.856 22755 DEBUG quantumclient.client [-] RESP:{'date': 'Mon, 17 Jun 2013 15:32:13 GMT', 'status': '200', 'content-length': '16', 'content-type': 'application/json; charset=UTF-8', 'content-location': 'http://127.0.0.1:9696//v2.0/floatingips.json?fixed_ip_address=10.100.0.2&port_id=fb9461c6-4e82-4fef-a06b-77242289f23d'} {"networks": []}
   http_log_resp /opt/stack/new/python-quantumclient/quantumclient/common/utils.py:179
  2013-06-17 15:32:13.856 22755 ERROR nova.compute.manager [-] [instance: f9327102-6a95-4d14-b262-2992d4ad66f3] Instance failed network setup
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3] Traceback (most recent call last):
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1172, in async_alloc
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     security_groups=security_groups)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/network/api.py", line 48, in wrapper
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     res = f(self, context, *args, **kwargs)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 285, in allocate_for_instance
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     nw_info = self._get_instance_nw_info(context, instance, networks=nets)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 377, in _get_instance_nw_info
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     nw_info = self._build_network_info_model(context, instance, networks)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 856, in _build_network_info_model
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     network_IPs = self._nw_info_get_ips(client, port)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 785, in _nw_info_get_ips
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     client, fixed_ip['ip_address'], port['id'])
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 723, in _get_floating_ips_by_fixed_and_port
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     port_id=port)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/python-quantumclient/quantumclient/v2_0/client.py", line 108, in with_params
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     ret = self.function(instance, *args, **kwargs)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/python-quantumclient/quantumclient/v2_0/client.py", line 441, in list_floatingips
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     **_params)
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]   File "/opt/stack/new/python-quantumclient/quantumclient/v2_0/client.py", line 1022, in list
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3]     res.extend(r[collection])
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3] KeyError: 'floatingips'
  2013-06-17 15:32:13.856 22755 TRACE nova.compute.manager [instance: f9327102-6a95-4d14-b262-2992d4ad66f3] 

  It's very unclear what's going in quantum at this point, as apparently
  only DEBUG logs are running, but not normal log levels? The fact that
  this is fixed with a recheck on unrelated code means there is some
  race in quantum interaction that is not accounted for from nova, and
  this is a huge issue currently in the gate.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1192131/+subscriptions